From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E511C433EF for ; Mon, 27 Sep 2021 16:08:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7468160F24 for ; Mon, 27 Sep 2021 16:08:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235419AbhI0QJi (ORCPT ); Mon, 27 Sep 2021 12:09:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235305AbhI0QJf (ORCPT ); Mon, 27 Sep 2021 12:09:35 -0400 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 40BE3C061604 for ; Mon, 27 Sep 2021 09:07:57 -0700 (PDT) Received: by mail-pj1-x102f.google.com with SMTP id r7so12226836pjo.3 for ; Mon, 27 Sep 2021 09:07:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=CBaIVDF4UN9VVYwJROukTIUHRk1VcAmSIyvI/1GuY6U=; b=T7xGNAbIwgWoQjmHAfjCyFlxZGmZwVgJp13QAcq9bF1OoppgBW2WkXPGLJksJenBai D+5aHtV9bQMmEjbzueuvY4vPf9AE74CBuT3P05dfHp/OCyyMhpnV5CdRPt+he8hrO6mo LnsFPrM4csxIeGPyIxIU0mJ9LzFhT1kne6pBl5she3hl6nVOklpcV1dfO4zW2Th7GRN/ G35PhgJyr3oGGDXkWUgty3HI6mfowgvts5bhUZ0kq+Rxfb+59MpjGhHFODaON7iy0iiM JR5ocMzOQwI/6O/Kqxcof7KhSwGZC3LUFOAW0X3wSiL51FHBFdUd7Azy48rMLavEFeux ORbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=CBaIVDF4UN9VVYwJROukTIUHRk1VcAmSIyvI/1GuY6U=; b=2Ha1LiHfROH3xZE2fJxkhUlmKLCKG02qoiEnRMPv8Zigessj3xA7i9xEwFhlpe8PJw Pg0b5SEstufYOWDZ4xkRaHtECHxXjf3lIovgdi13AJbqQgD6C3rh1nZAprb8YWFBMi/7 RT+GESmoKFZnpZWcJWehkCWptQ5vNcaSGp/4XVX76m4WFpX2A4desbo4KksSuYeSOqbq cSUwiXM5Lm2APdV5JAjZJ6MW6K7fpWcrbIlg+8DQnyMa3YJUthtuyPH9cKC+yKQMBVCN RPNuMQQOtaEexW16mSSvjRVKE3+idIyQ8OB/JK+aGQZWq5kc2j7Kin2fKa6BkMzYfAKi f6oQ== X-Gm-Message-State: AOAM533eyp32ngyXu2hjN8+odFGU35AyeJ3y9/7CEVNPkwHpVo7h+8fE c3fwN/VFs/E7Oc/IP+fky3ut+w== X-Google-Smtp-Source: ABdhPJwDaKS2HT0eEsDrjIENTUe9qQL4tA5WiU/rZM9D+/g6ip8neC4z6hkG15pQ2d/r2fMoMdD1jA== X-Received: by 2002:a17:90a:ca96:: with SMTP id y22mr9642043pjt.115.1632758876549; Mon, 27 Sep 2021 09:07:56 -0700 (PDT) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id i2sm16110859pfa.34.2021.09.27.09.07.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Sep 2021 09:07:55 -0700 (PDT) Date: Mon, 27 Sep 2021 16:07:51 +0000 From: Sean Christopherson To: Dmitry Vyukov Cc: Marco Elver , syzbot , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, syzkaller-bugs@googlegroups.com, viro@zeniv.linux.org.uk, the arch/x86 maintainers , Linux ARM , kasan-dev , Josh Poimboeuf , Peter Zijlstra Subject: Re: [syzbot] upstream test error: KFENCE: use-after-free in kvm_fastop_exception Message-ID: References: <000000000000d6b66705cb2fffd4@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org +Josh and PeterZ On Mon, Sep 27, 2021, Dmitry Vyukov wrote: > On Wed, 22 Sept 2021 at 01:34, 'Sean Christopherson' via > syzkaller-bugs wrote: > > > > On Fri, Sep 17, 2021, Dmitry Vyukov wrote: > > > On Fri, 17 Sept 2021 at 13:04, Marco Elver wrote: > > > > > So it looks like in both cases the top fault frame is just wrong. But > > > > > I would assume it's extracted by arch-dependent code, so it's > > > > > suspicious that it affects both x86 and arm64... > > > > > > > > > > Any ideas what's happening? > > > > > > > > My suspicion for the x86 case is that kvm_fastop_exception is related > > > > to instruction emulation and the fault occurs in an emulated > > > > instruction? > > > > > > Why would the kernel emulate a plain MOV? > > > 2a: 4c 8b 21 mov (%rcx),%r12 > > > > > > And it would also mean a broken unwind because the emulated > > > instruction is in __d_lookup, so it should be in the stack trace. > > > > kvm_fastop_exception is a red herring. It's indeed related to emulation, and > > while MOV emulation is common in KVM, that emulation is for KVM guests not for > > the host kernel where this splat occurs (ignoring the fact that the "host" is > > itself a guest). > > > > kvm_fastop_exception is out-of-line fixup, and certainly shouldn't be reachable > > via d_lookup. It's also two instruction, XOR+RET, neither of which are in the > > code stream. > > > > IIRC, the unwinder gets confused when given an IP that's in out-of-line code, > > e.g. exception fixup like this. If you really want to find out what code blew > > up, you might be able to objdump -D the kernel and search for unique, matching > > disassembly, e.g. find "jmpq 0xf86d288c" and go from there. > > Hi Sean, > > Thanks for the info. > > I don't want to find out what code blew (it's __d_lookup). > I am interested in getting the unwinder fixed to output truthful and > useful frames. I was asking about the exact location to confirm that the explosion is indeed from exception fixup, which is the "unwinder scenario get confused" I was thinking of. Based on the disassembly from syzbot, that does indeed appear to be the case here, i.e. this 2a: 4c 8b 21 mov (%rcx),%r12 is from exception fixup from somewhere in __d_lookup (can't tell exactly what it's from, maybe KASAN?). > Is there more info on this "the unwinder gets confused"? Bug filed > somewhere or an email thread? Is it on anybody's radar? I don't know if there's a bug report or if this is on anyone's radar. The issue I've encountered in the past, and what I'm pretty sure is being hit here, is that the ORC unwinder doesn't play nice with out-of-line fixup code, presumably because there are no tables for the fixup. I believe kvm_fastop_exception() gets blamed because it's the first label that's found when searching back through the tables. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 663DDC433EF for ; Mon, 27 Sep 2021 16:10:00 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 19D9560F9B for ; Mon, 27 Sep 2021 16:10:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 19D9560F9B Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=G/zfj9rPp4tTaXk8iDWg6sYq59KbbKjHr47FLeqe8b4=; b=k41ZMLilO/s2JV 7X2ag4jigW6lIAlRA2r6Elrn6kdrtO1DnCOFdYIFEO2CwtY3s2HpTIbpMTT6Hv68kv3uyRAK8kjZe IN6dH445hr6NV41yuP0P1SWbE7A/+9ux9n65eia+O+2DvNe/rzbmV8KJ2TFP17X5P7eZ72C8sQ1F2 lerLdAFxeqrlAU87QxksjqtNxzs4rdlQQhBThsgGxPw14TVpG3mz9z++dGzk1+s+tWKwy9EBLJQv3 hCUSpKtFRYa1L3D/VlpMAae5FqWmJom/uT8xX0GyxJ0hU1WNCuls64PwatQXx1w5PgVk+VBmR/gVw 9gNztrEyU62XDv/Y7vNw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mUtAf-003Jnd-OY; Mon, 27 Sep 2021 16:08:01 +0000 Received: from mail-pj1-x1036.google.com ([2607:f8b0:4864:20::1036]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mUtAb-003JmO-N5 for linux-arm-kernel@lists.infradead.org; Mon, 27 Sep 2021 16:07:59 +0000 Received: by mail-pj1-x1036.google.com with SMTP id bj3-20020a17090b088300b0019e6603fe89so12169560pjb.4 for ; Mon, 27 Sep 2021 09:07:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=CBaIVDF4UN9VVYwJROukTIUHRk1VcAmSIyvI/1GuY6U=; b=T7xGNAbIwgWoQjmHAfjCyFlxZGmZwVgJp13QAcq9bF1OoppgBW2WkXPGLJksJenBai D+5aHtV9bQMmEjbzueuvY4vPf9AE74CBuT3P05dfHp/OCyyMhpnV5CdRPt+he8hrO6mo LnsFPrM4csxIeGPyIxIU0mJ9LzFhT1kne6pBl5she3hl6nVOklpcV1dfO4zW2Th7GRN/ G35PhgJyr3oGGDXkWUgty3HI6mfowgvts5bhUZ0kq+Rxfb+59MpjGhHFODaON7iy0iiM JR5ocMzOQwI/6O/Kqxcof7KhSwGZC3LUFOAW0X3wSiL51FHBFdUd7Azy48rMLavEFeux ORbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=CBaIVDF4UN9VVYwJROukTIUHRk1VcAmSIyvI/1GuY6U=; b=nbX1j+aZPn3vXBwrMzqKw90MVPpIWDg+OyI3D4tpTgkHw3lX5FlHfc7Gmf+SspgsLX K9mx/OrwSxF9PFoUs87XBqoRNrX/6Bz9WQxi1VUc2oUGI3OsKYmfek51sgQSxEM/oWTp bRc93WI6klDgaBmPvDVd45r5u1S+n//Dp1VHJKIWTtaNK6Z8RJ28gmOruRC/Zo8AAd+R KdI6ayy1WZ+JxPBPbhOmiTS8G3W2YlDAK5v5/Rz2F3ib/F6P6UGXlUuiAR6nqe1YJNdt BswDsT0M5O0r/ilF186nwffZbSy6uzEtu+r2WvUVl7b9oSIFc7tyUopdi1YvwEz1k6HH GFBQ== X-Gm-Message-State: AOAM531rh5LMoh8C2P8ZxSrZwT34TJRXMGHVM58vfeBw47Xfr3sNpbCn iYOFpX6X45IICvorm9NOQT9ivw== X-Google-Smtp-Source: ABdhPJwDaKS2HT0eEsDrjIENTUe9qQL4tA5WiU/rZM9D+/g6ip8neC4z6hkG15pQ2d/r2fMoMdD1jA== X-Received: by 2002:a17:90a:ca96:: with SMTP id y22mr9642043pjt.115.1632758876549; Mon, 27 Sep 2021 09:07:56 -0700 (PDT) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id i2sm16110859pfa.34.2021.09.27.09.07.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Sep 2021 09:07:55 -0700 (PDT) Date: Mon, 27 Sep 2021 16:07:51 +0000 From: Sean Christopherson To: Dmitry Vyukov Cc: Marco Elver , syzbot , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, syzkaller-bugs@googlegroups.com, viro@zeniv.linux.org.uk, the arch/x86 maintainers , Linux ARM , kasan-dev , Josh Poimboeuf , Peter Zijlstra Subject: Re: [syzbot] upstream test error: KFENCE: use-after-free in kvm_fastop_exception Message-ID: References: <000000000000d6b66705cb2fffd4@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210927_090757_784011_0C554064 X-CRM114-Status: GOOD ( 33.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org +Josh and PeterZ On Mon, Sep 27, 2021, Dmitry Vyukov wrote: > On Wed, 22 Sept 2021 at 01:34, 'Sean Christopherson' via > syzkaller-bugs wrote: > > > > On Fri, Sep 17, 2021, Dmitry Vyukov wrote: > > > On Fri, 17 Sept 2021 at 13:04, Marco Elver wrote: > > > > > So it looks like in both cases the top fault frame is just wrong. But > > > > > I would assume it's extracted by arch-dependent code, so it's > > > > > suspicious that it affects both x86 and arm64... > > > > > > > > > > Any ideas what's happening? > > > > > > > > My suspicion for the x86 case is that kvm_fastop_exception is related > > > > to instruction emulation and the fault occurs in an emulated > > > > instruction? > > > > > > Why would the kernel emulate a plain MOV? > > > 2a: 4c 8b 21 mov (%rcx),%r12 > > > > > > And it would also mean a broken unwind because the emulated > > > instruction is in __d_lookup, so it should be in the stack trace. > > > > kvm_fastop_exception is a red herring. It's indeed related to emulation, and > > while MOV emulation is common in KVM, that emulation is for KVM guests not for > > the host kernel where this splat occurs (ignoring the fact that the "host" is > > itself a guest). > > > > kvm_fastop_exception is out-of-line fixup, and certainly shouldn't be reachable > > via d_lookup. It's also two instruction, XOR+RET, neither of which are in the > > code stream. > > > > IIRC, the unwinder gets confused when given an IP that's in out-of-line code, > > e.g. exception fixup like this. If you really want to find out what code blew > > up, you might be able to objdump -D the kernel and search for unique, matching > > disassembly, e.g. find "jmpq 0xf86d288c" and go from there. > > Hi Sean, > > Thanks for the info. > > I don't want to find out what code blew (it's __d_lookup). > I am interested in getting the unwinder fixed to output truthful and > useful frames. I was asking about the exact location to confirm that the explosion is indeed from exception fixup, which is the "unwinder scenario get confused" I was thinking of. Based on the disassembly from syzbot, that does indeed appear to be the case here, i.e. this 2a: 4c 8b 21 mov (%rcx),%r12 is from exception fixup from somewhere in __d_lookup (can't tell exactly what it's from, maybe KASAN?). > Is there more info on this "the unwinder gets confused"? Bug filed > somewhere or an email thread? Is it on anybody's radar? I don't know if there's a bug report or if this is on anyone's radar. The issue I've encountered in the past, and what I'm pretty sure is being hit here, is that the ORC unwinder doesn't play nice with out-of-line fixup code, presumably because there are no tables for the fixup. I believe kvm_fastop_exception() gets blamed because it's the first label that's found when searching back through the tables. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel