From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E57CC433DB for ; Fri, 12 Feb 2021 14:00:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E7DB264E6B for ; Fri, 12 Feb 2021 14:00:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231445AbhBLOAY (ORCPT ); Fri, 12 Feb 2021 09:00:24 -0500 Received: from mail.kernel.org ([198.145.29.99]:40988 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230336AbhBLOAW (ORCPT ); Fri, 12 Feb 2021 09:00:22 -0500 Received: by mail.kernel.org (Postfix) with ESMTPSA id E8DF764E77 for ; Fri, 12 Feb 2021 13:59:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1613138381; bh=EP2CAHVuzyRMG/PUKnUl+jY0zn7rp2M0QTID3TQCH40=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=SShzO+dG8ltMJpUaXev4p59HSQlluU7bRNr94cRICkioaCPO+zzm7n4moRXjSF1ud n/lEVjkw/f0Iawgra3RRLRT+p0wMuKwrZu065Ug9uxnwkyYnD9mfcVO/6zgEzuOJ+3 MVvAW3qTqnEv7e6obiKS1DFfhXbTqG1ko+KoPe2JuT+t4jEd6tRAgy6f8zFl9g8pJ0 7PDSMWBF0QPleCoh5ghng7z4+dvnxzDgJP6QwU5bywOcHCFja123nfiQA6R9peH+1G WN13Ve19f7h+xFCVtIcZm5UE3f2qNmNCaBPTeAbbqG1AU087lRijEkVSFmFGp8nkkm v9urxGLl5B4sA== Received: by mail-oi1-f180.google.com with SMTP id u66so10022360oig.9 for ; Fri, 12 Feb 2021 05:59:40 -0800 (PST) X-Gm-Message-State: AOAM5316nlqt/g+mMYYbO4clrkbojoCGzTfHamHvybdSQdj7dPElmAle /hpQ4h/hDpsi2jeIAjCv15HboCdaQkqI0G2ruoE= X-Google-Smtp-Source: ABdhPJyCXDdL6PXL3t5jXk6NfNgAOpzgcr0LS1EFcA/cMyDFz+G2r8F+sD2aTd1ObyAzLWQUcTySqixHio14ptPsLlE= X-Received: by 2002:aca:2117:: with SMTP id 23mr1849245oiz.4.1613138380166; Fri, 12 Feb 2021 05:59:40 -0800 (PST) MIME-Version: 1.0 References: <20210211202208.31555-1-Sonicadvance1@gmail.com> <58b03e17-3729-99ea-8691-0d735a53b9bc@arm.com> In-Reply-To: <58b03e17-3729-99ea-8691-0d735a53b9bc@arm.com> From: Arnd Bergmann Date: Fri, 12 Feb 2021 14:59:24 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RESEND RFC PATCH v2] arm64: Exposes support for 32-bit syscalls To: Steven Price Cc: Ryan Houdek , "Amanieu d'Antras" , Catalin Marinas , Will Deacon , Mark Rutland , Oleg Nesterov , Al Viro , Dave Martin , Amit Daniel Kachhap , Mark Brown , Marc Zyngier , David Brazdil , Jean-Philippe Brucker , Andrew Morton , Anshuman Khandual , Gavin Shan , Mike Rapoport , Vincenzo Frascino , Kristina Martsenko , Kees Cook , Sami Tolvanen , Frederic Weisbecker , Kevin Hao , Jason Yan , Andrey Ignatov , Peter Collingbourne , Julien Grall , Tian Tao , Qais Yousef , Jens Axboe , Linux ARM , "linux-kernel@vger.kernel.org" Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Feb 12, 2021 at 12:33 PM Steven Price wrote: > On 11/02/2021 20:21, sonicadvance1@gmail.com wrote: > > The problem: > > We need to support 32-bit processes running under a userspace > > compatibility layer. The compatibility layer is a AArch64 process. > > This means exposing the 32bit compatibility syscalls to userspace. > > I'm not sure how you come to this conclusion. Running 32-bit processes > under a compatibility layer is a fine goal, but it's not clear why the > entire 32-bit compat syscall layer is needed for this. > > As a case in point QEMU's user mode emulation already achieves this in > many cases without any changes to the kernel. I think it's a quantitative difference, not a qualitative one: qemu does a nice job at translating the interfaces for many combinations of host and target architectures at a decent speed, and is improving at both the compatibility and the performance over time. What both Tango and FEX promise is to be much faster by focusing on one target architecture each, and to have better compatibility than what qemu can do. > > Who does this matter to? > > Any user that has a specific need to run legacy 32-bit software under a > > compatibility layer. > > Not all software is open source or easy to convert to 64bit, it's > > something we need to live with. > > Professional software and the gaming ecosystem is rife with this. > > > > What applications have tried to work around this problem? > > FEX emulator (1) - Userspace x86 to AArch64 compatibility layer > > Tango binary translator (2) - AArch32 to AArch64 compatibility layer > > QEmu (3) - Not really but they do some userspace ioctl emulation > > Can you expand on "not really"? Clearly there are limitations, but in > general I can happily "chroot" into a distro filesystem using an > otherwise incompatible architecture using a qemu-xxx-static binary. The ioctl emulation in qemu is limited in multiple ways: - it needs to duplicate the kernel's compat emulation for every single command it wants to handle, and will always lag behind what gets merged into the kernel and what drivers a particular distro ships. - some ioctl commands cannot be emulated in user space because the compat code relies on tracking device state in the kernel. - In some cases, emulation can be expensive, both for runtime overhead and for code complexity > > What problems did they hit? > > FEX and Tango hit problems with emulating memory related syscalls. > > - Emulating 32-bit mmap, mremap, shmat in userspace changes behaviour > > All three hit issues with ioctl emulation > > - ioctls are free to do what they want including allocating memory and > > returning opaque structures with pointers. > > Now I think we're getting to what the actual problems are: > > * mmap and friends have no (easy) way of forcing a mapping into a 32 > bit region. > * ioctls are a mess > > The first seems like a reasonable goal - I've seen examples of MAP_32BIT > being (ab)used to do this, but it actually restricts to 31 bits and it's > not even available on arm64. Here I think you'd be better off focusing > on coming up with a new (generic) way of restricting the addresses that > the kernel will pick. I think that would be useful for other projects as well. > ioctls are going to be a problem whatever you do, and I don't think > there is much option other than having a list of known ioctls and > translating them in user space - see below. In particular for the arm32-on-arm64 ioctl case, we have a known-working implementation in the kernel, I don't see why we wouldn't want to use it. the x86-32-on-anything case for FEX is trickier because it does require handling the ia32 alignment case, and the proposed patch does not handle that correctly for all commands. I think this would be fixable in the kernel, but it requires a little more work. > > This is now exposing the compat syscalls to userspace, but for the sake > > of userspace compatibility it is a necessary evil. > > You've yet to convince me that it's "necessary" - I agree on the "evil" > part ;) I think it's much easier to argue in favor of exposing the kernel's ioctl() emulation and a get_unmapped_area() limit to a process specific address than for doing the entire syscalls emulation. The emulation for any of the other syscalls should be manageable once ioctl can be called directly, though there are a couple that could fall into the same category (setsockopt, sendmsg/recvmsg, fcntl). Arnd From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17447C433DB for ; Fri, 12 Feb 2021 14:01:00 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A612A64E57 for ; Fri, 12 Feb 2021 14:00:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A612A64E57 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=xOyqt2k3Fdkni9WvyDUSJzyFTdsOFfc/slnpte3sJw8=; b=pVdo/jDgnTtZdM4TBqebpBznw 7cFXmrjeUcxbp0z1zaj5DaxNyzn8XaUwyuBV7BzBR4SzN54pC1AEvcVufis25+FACGfFW3PEMvLXt TG3naF6/MSosC9iwU8JLx5EyNn6BwhsgxrvIO7O4frU7DwyOq3qsoIo6KXJJeOJ9D6XgmURjBe0DF Izyi/9vfbGJtbenZ7TtkhXeNRdI+cy0793dnSdaDmZvt8uAnGIN3zm0klFOSjxBMI0KQOt0+njWnd S4AF9zQqPagv6I/iWPkg4XQLe/nyKBHDK9N9ukRd4eu9b1B6Jw+NIiAGfvyunwgH3vXCFEkkf8ltH b1DOufKCw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1lAYz3-0007Cp-U4; Fri, 12 Feb 2021 13:59:45 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1lAYz1-0007BW-6E for linux-arm-kernel@lists.infradead.org; Fri, 12 Feb 2021 13:59:44 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 2126364E92 for ; Fri, 12 Feb 2021 13:59:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1613138381; bh=EP2CAHVuzyRMG/PUKnUl+jY0zn7rp2M0QTID3TQCH40=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=SShzO+dG8ltMJpUaXev4p59HSQlluU7bRNr94cRICkioaCPO+zzm7n4moRXjSF1ud n/lEVjkw/f0Iawgra3RRLRT+p0wMuKwrZu065Ug9uxnwkyYnD9mfcVO/6zgEzuOJ+3 MVvAW3qTqnEv7e6obiKS1DFfhXbTqG1ko+KoPe2JuT+t4jEd6tRAgy6f8zFl9g8pJ0 7PDSMWBF0QPleCoh5ghng7z4+dvnxzDgJP6QwU5bywOcHCFja123nfiQA6R9peH+1G WN13Ve19f7h+xFCVtIcZm5UE3f2qNmNCaBPTeAbbqG1AU087lRijEkVSFmFGp8nkkm v9urxGLl5B4sA== Received: by mail-oi1-f171.google.com with SMTP id r75so10015645oie.11 for ; Fri, 12 Feb 2021 05:59:41 -0800 (PST) X-Gm-Message-State: AOAM532nvPvoin1hTehUz3rn+FurFrZlftcrG2t9uWtaQMVQ71xNVLEZ vWF5yderOg2NA6K0P8pLdntNvlGtf5vov1jvHKU= X-Google-Smtp-Source: ABdhPJyCXDdL6PXL3t5jXk6NfNgAOpzgcr0LS1EFcA/cMyDFz+G2r8F+sD2aTd1ObyAzLWQUcTySqixHio14ptPsLlE= X-Received: by 2002:aca:2117:: with SMTP id 23mr1849245oiz.4.1613138380166; Fri, 12 Feb 2021 05:59:40 -0800 (PST) MIME-Version: 1.0 References: <20210211202208.31555-1-Sonicadvance1@gmail.com> <58b03e17-3729-99ea-8691-0d735a53b9bc@arm.com> In-Reply-To: <58b03e17-3729-99ea-8691-0d735a53b9bc@arm.com> From: Arnd Bergmann Date: Fri, 12 Feb 2021 14:59:24 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RESEND RFC PATCH v2] arm64: Exposes support for 32-bit syscalls To: Steven Price X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210212_085943_475530_6B8887B7 X-CRM114-Status: GOOD ( 38.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Gavin Shan , Catalin Marinas , "linux-kernel@vger.kernel.org" , Julien Grall , Amit Daniel Kachhap , Vincenzo Frascino , Will Deacon , Qais Yousef , Jean-Philippe Brucker , Marc Zyngier , Anshuman Khandual , Andrey Ignatov , Sami Tolvanen , David Brazdil , Dave Martin , Kees Cook , Ryan Houdek , Frederic Weisbecker , Kristina Martsenko , Mark Brown , Al Viro , Kevin Hao , Peter Collingbourne , Linux ARM , Jens Axboe , Amanieu d'Antras , Jason Yan , Oleg Nesterov , Tian Tao , Andrew Morton , Mike Rapoport Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Feb 12, 2021 at 12:33 PM Steven Price wrote: > On 11/02/2021 20:21, sonicadvance1@gmail.com wrote: > > The problem: > > We need to support 32-bit processes running under a userspace > > compatibility layer. The compatibility layer is a AArch64 process. > > This means exposing the 32bit compatibility syscalls to userspace. > > I'm not sure how you come to this conclusion. Running 32-bit processes > under a compatibility layer is a fine goal, but it's not clear why the > entire 32-bit compat syscall layer is needed for this. > > As a case in point QEMU's user mode emulation already achieves this in > many cases without any changes to the kernel. I think it's a quantitative difference, not a qualitative one: qemu does a nice job at translating the interfaces for many combinations of host and target architectures at a decent speed, and is improving at both the compatibility and the performance over time. What both Tango and FEX promise is to be much faster by focusing on one target architecture each, and to have better compatibility than what qemu can do. > > Who does this matter to? > > Any user that has a specific need to run legacy 32-bit software under a > > compatibility layer. > > Not all software is open source or easy to convert to 64bit, it's > > something we need to live with. > > Professional software and the gaming ecosystem is rife with this. > > > > What applications have tried to work around this problem? > > FEX emulator (1) - Userspace x86 to AArch64 compatibility layer > > Tango binary translator (2) - AArch32 to AArch64 compatibility layer > > QEmu (3) - Not really but they do some userspace ioctl emulation > > Can you expand on "not really"? Clearly there are limitations, but in > general I can happily "chroot" into a distro filesystem using an > otherwise incompatible architecture using a qemu-xxx-static binary. The ioctl emulation in qemu is limited in multiple ways: - it needs to duplicate the kernel's compat emulation for every single command it wants to handle, and will always lag behind what gets merged into the kernel and what drivers a particular distro ships. - some ioctl commands cannot be emulated in user space because the compat code relies on tracking device state in the kernel. - In some cases, emulation can be expensive, both for runtime overhead and for code complexity > > What problems did they hit? > > FEX and Tango hit problems with emulating memory related syscalls. > > - Emulating 32-bit mmap, mremap, shmat in userspace changes behaviour > > All three hit issues with ioctl emulation > > - ioctls are free to do what they want including allocating memory and > > returning opaque structures with pointers. > > Now I think we're getting to what the actual problems are: > > * mmap and friends have no (easy) way of forcing a mapping into a 32 > bit region. > * ioctls are a mess > > The first seems like a reasonable goal - I've seen examples of MAP_32BIT > being (ab)used to do this, but it actually restricts to 31 bits and it's > not even available on arm64. Here I think you'd be better off focusing > on coming up with a new (generic) way of restricting the addresses that > the kernel will pick. I think that would be useful for other projects as well. > ioctls are going to be a problem whatever you do, and I don't think > there is much option other than having a list of known ioctls and > translating them in user space - see below. In particular for the arm32-on-arm64 ioctl case, we have a known-working implementation in the kernel, I don't see why we wouldn't want to use it. the x86-32-on-anything case for FEX is trickier because it does require handling the ia32 alignment case, and the proposed patch does not handle that correctly for all commands. I think this would be fixable in the kernel, but it requires a little more work. > > This is now exposing the compat syscalls to userspace, but for the sake > > of userspace compatibility it is a necessary evil. > > You've yet to convince me that it's "necessary" - I agree on the "evil" > part ;) I think it's much easier to argue in favor of exposing the kernel's ioctl() emulation and a get_unmapped_area() limit to a process specific address than for doing the entire syscalls emulation. The emulation for any of the other syscalls should be manageable once ioctl can be called directly, though there are a couple that could fall into the same category (setsockopt, sendmsg/recvmsg, fcntl). Arnd _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel