From mboxrd@z Thu Jan 1 00:00:00 1970 Received: with ECARTIS (v1.0.0; list linux-mips); Wed, 15 Mar 2017 22:30:58 +0100 (CET) Received: from mail-wr0-x241.google.com ([IPv6:2a00:1450:400c:c0c::241]:33121 "EHLO mail-wr0-x241.google.com" rhost-flags-OK-OK-OK-OK) by eddie.linux-mips.org with ESMTP id S23992111AbdCOVawFkzb4 (ORCPT ); Wed, 15 Mar 2017 22:30:52 +0100 Received: by mail-wr0-x241.google.com with SMTP id g10so3602132wrg.0; Wed, 15 Mar 2017 14:30:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20161025; h=from:date:to:cc:subject:message-id:mail-followup-to:mime-version :content-disposition:in-reply-to:user-agent; bh=1+M/lApnImxxd201Kj3J/Twbmk6raMaTIn7GKmQAQ/I=; b=cvUJBdexXS4lUo8eGmr/vJO0QBXQBjia8M5cRstba8gBLzglcXb14w3ZL5BrVZkrol Gsxt6qtDKUHZ2dV22CE7jkYHHXIgxwPFLDAUV8/N+KFF7k1ScNA1o2fb+/cDt2RBvj08 y/OgwCOMYTaFn7LiqITptz4NbT3+F4sdD2iFxJgufR7pRg0JGCKyIZrvo1C4Vvx3Eh4V gO7S7bqHyBmgRCLG6wuXva3NKZgPaj/LJSh7NC0TxOWI8kKiXnOnW6uQ9HJfuJ2/dP20 74PWFAmFSLfbA9ZDitBfRqdA03o+0y9VlUqOQifeSdPNk2bS+GVQXr7WPyReoDzw5uw6 a3pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:date:to:cc:subject:message-id :mail-followup-to:mime-version:content-disposition:in-reply-to :user-agent; bh=1+M/lApnImxxd201Kj3J/Twbmk6raMaTIn7GKmQAQ/I=; b=osKRJnCH2PBH6aE6rBJRuP8FCs1VEgvVgXYN9dqGUoIC1qy9lOo2zs9rEeAP5H1T7J EsdUk3/uYUBiazn982ZVL6CodalDtxMLNV2RSPg6ZA2kFYvcMd01p6LBA2QEVVUGg6VY HOTo/UmHex99+eu3yGzmnyiHbWG09sylyXPv4i8xtPwAnfJMH8iVGI50gOQAhoilF4kz ItG/vAzWv1RF2szfrqaK/6+2Z3/WAC7dAgdQ4tV5Unpqvxz5A56roZRNb5wFt51rT7Qi 0kAbJN4H2MDisgSdXK7fmwwLJiE7RyR394hWq8poHi2JVwdVGmUjohr6CjM50A4/Wk2F MJXw== X-Gm-Message-State: AFeK/H07ID9UZRG1U33VddZghSlnawtkAw6eV1pwLmKX6iaR0Xt9c80YGtxuFSaM+9kKNw== X-Received: by 10.223.130.144 with SMTP id 16mr5385217wrc.32.1489613446772; Wed, 15 Mar 2017 14:30:46 -0700 (PDT) Received: from localhost (login1.zih.tu-dresden.de. [141.76.16.140]) by smtp.googlemail.com with ESMTPSA id k43sm3740776wrk.42.2017.03.15.14.30.44 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 15 Mar 2017 14:30:45 -0700 (PDT) From: Till Smejkal X-Google-Original-From: Till Smejkal Date: Wed, 15 Mar 2017 14:30:43 -0700 To: Rich Felker Cc: Andy Lutomirski , Andy Lutomirski , Till Smejkal , Richard Henderson , Ivan Kokshaysky , Matt Turner , Vineet Gupta , Russell King , Catalin Marinas , Will Deacon , Steven Miao , Richard Kuo , Tony Luck , Fenghua Yu , James Hogan , Ralf Baechle , "James E.J. Bottomley" , Helge Deller , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Martin Schwidefsky , Heiko Carstens , Yoshinori Sato , "David S. Miller" , Chris Metcalf , Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , X86 ML , Chris Zankel , Max Filippov , Arnd Bergmann , Greg Kroah-Hartman , Laurent Pinchart , Mauro Carvalho Chehab , Pawel Osciak , Marek Szyprowski , Kyungmin Park , David Woodhouse , Brian Norris , Boris Brezillon , Marek Vasut , Richard Weinberger , Cyrille Pitchen , Felipe Balbi , Alexander Viro , Benjamin LaHaise , Nadia Yvette Chambers , Jeff Layton , "J. Bruce Fields" , Peter Zijlstra , Hugh Dickins , Arnaldo Carvalho de Melo , Alexander Shishkin , Jaroslav Kysela , Takashi Iwai , "linux-kernel@vger.kernel.org" , linux-alpha@vger.kernel.org, arcml , "linux-arm-kernel@lists.infradead.org" , adi-buildroot-devel@lists.sourceforge.net, linux-hexagon@vger.kernel.org, "linux-ia64@vger.kernel.org" , linux-metag@vger.kernel.org, Linux MIPS Mailing List , linux-parisc@vger.kernel.org, linuxppc-dev , "linux-s390@vger.kernel.org" , "linux-sh@vger.kernel.org" , sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, Linux Media Mailing List , linux-mtd@lists.infradead.org, USB list , Linux FS Devel , linux-aio@kvack.org, "linux-mm@kvack.org" , Linux API , linux-arch , ALSA development Subject: Re: [RFC PATCH 00/13] Introduce first class virtual address spaces Message-ID: <20170315213042.e5q6daqkqoam2iun@arch-dev> Mail-Followup-To: Rich Felker , Andy Lutomirski , Andy Lutomirski , Till Smejkal , Richard Henderson , Ivan Kokshaysky , Matt Turner , Vineet Gupta , Russell King , Catalin Marinas , Will Deacon , Steven Miao , Richard Kuo , Tony Luck , Fenghua Yu , James Hogan , Ralf Baechle , "James E.J. Bottomley" , Helge Deller , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Martin Schwidefsky , Heiko Carstens , Yoshinori Sato , "David S. Miller" , Chris Metcalf , Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , X86 ML , Chris Zankel , Max Filippov , Arnd Bergmann , Greg Kroah-Hartman , Laurent Pinchart , Mauro Carvalho Chehab , Pawel Osciak , Marek Szyprowski , Kyungmin Park , David Woodhouse , Brian Norris , Boris Brezillon , Marek Vasut , Richard Weinberger , Cyrille Pitchen , Felipe Balbi , Alexander Viro , Benjamin LaHaise , Nadia Yvette Chambers , Jeff Layton , "J. Bruce Fields" , Peter Zijlstra , Hugh Dickins , Arnaldo Carvalho de Melo , Alexander Shishkin , Jaroslav Kysela , Takashi Iwai , "linux-kernel@vger.kernel.org" , linux-alpha@vger.kernel.org, arcml , "linux-arm-kernel@lists.infradead.org" , adi-buildroot-devel@lists.sourceforge.net, linux-hexagon@vger.kernel.org, "linux-ia64@vger.kernel.org" , linux-metag@vger.kernel.org, Linux MIPS Mailing List , linux-parisc@vger.kernel.org, linuxppc-dev , "linux-s390@vger.kernel.org" , "linux-sh@vger.kernel.org" , sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, Linux Media Mailing List , linux-mtd@lists.infradead.org, USB list , Linux FS Devel , linux-aio@kvack.org, "linux-mm@kvack.org" , Linux API , linux-arch , ALSA development MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170315194723.GJ1693@brightrain.aerifal.cx> User-Agent: NeoMutt/20170306 (1.8.0) Return-Path: X-Envelope-To: <"|/home/ecartis/ecartis -s linux-mips"> (uid 0) X-Orcpt: rfc822;linux-mips@linux-mips.org Original-Recipient: rfc822;linux-mips@linux-mips.org X-archive-position: 57310 X-ecartis-version: Ecartis v1.0.0 Sender: linux-mips-bounce@linux-mips.org Errors-to: linux-mips-bounce@linux-mips.org X-original-sender: till.smejkal@googlemail.com Precedence: bulk List-help: List-unsubscribe: List-software: Ecartis version 1.0.0 List-Id: linux-mips X-List-ID: linux-mips List-subscribe: List-owner: List-post: List-archive: X-list: linux-mips On Wed, 15 Mar 2017, Rich Felker wrote: > On Wed, Mar 15, 2017 at 12:44:47PM -0700, Till Smejkal wrote: > > On Wed, 15 Mar 2017, Andy Lutomirski wrote: > > > > One advantage of VAS segments is that they can be globally queried by user programs > > > > which means that VAS segments can be shared by applications that not necessarily have > > > > to be related. If I am not mistaken, MAP_SHARED of pure in memory data will only work > > > > if the tasks that share the memory region are related (aka. have a common parent that > > > > initialized the shared mapping). Otherwise, the shared mapping have to be backed by a > > > > file. > > > > > > What's wrong with memfd_create()? > > > > > > > VAS segments on the other side allow sharing of pure in memory data by > > > > arbitrary related tasks without the need of a file. This becomes especially > > > > interesting if one combines VAS segments with non-volatile memory since one can keep > > > > data structures in the NVM and still be able to share them between multiple tasks. > > > > > > What's wrong with regular mmap? > > > > I never wanted to say that there is something wrong with regular mmap. We just > > figured that with VAS segments you could remove the need to mmap your shared data but > > instead can keep everything purely in memory. > > > > Unfortunately, I am not at full speed with memfds. Is my understanding correct that > > if the last user of such a file descriptor closes it, the corresponding memory is > > freed? Accordingly, memfd cannot be used to keep data in memory while no program is > > currently using it, can it? To be able to do this you need again some representation > > I have a name for application-allocated kernel resources that persist > without a process holding a reference to them or a node in the > filesystem: a bug. See: sysvipc. VAS segments are first class citizens of the OS similar to processes. Accordingly, I would not see this behavior as a bug. VAS segments are a kernel handle to "persistent" memory (in the sense that they are independent of the lifetime of the application that created them). That means the memory that is described by VAS segments can be reused by other applications even if the VAS segment was not used by any application in between. It is very much like a pure in-memory file. An application creates a VAS segment, fills it with content and if it does not delete it again, can reuse/open it again later. This also means, that if you know that you never want to use this memory again you have to remove it explicitly, like you have to remove a file, if you don't want to use it anymore. I think it really might be better to implement VAS segments (if I should keep this feature at all) with a special purpose filesystem. The way I've designed it seams to be very misleading. Till From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr0-x241.google.com ([IPv6:2a00:1450:400c:c0c::241]:33121 "EHLO mail-wr0-x241.google.com" rhost-flags-OK-OK-OK-OK) by eddie.linux-mips.org with ESMTP id S23992111AbdCOVawFkzb4 (ORCPT ); Wed, 15 Mar 2017 22:30:52 +0100 From: Till Smejkal Date: Wed, 15 Mar 2017 14:30:43 -0700 Subject: Re: [RFC PATCH 00/13] Introduce first class virtual address spaces Message-ID: <20170315213042.e5q6daqkqoam2iun@arch-dev> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170315194723.GJ1693@brightrain.aerifal.cx> Return-Path: Sender: linux-mips-bounce@linux-mips.org Errors-to: linux-mips-bounce@linux-mips.org List-help: List-unsubscribe: List-software: Ecartis version 1.0.0 List-subscribe: List-owner: List-post: List-archive: To: Rich Felker Cc: Andy Lutomirski , Andy Lutomirski , Till Smejkal , Richard Henderson , Ivan Kokshaysky , Matt Turner , Vineet Gupta , Russell King , Catalin Marinas , Will Deacon , Steven Miao , Richard Kuo , Tony Luck , Fenghua Yu , James Hogan , Ralf Baechle , "James E.J. Bottomley" , Helge Deller , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Martin Schwidefsky , Heiko Carstens , Yoshinori Sato , "David S. Miller" , Chris Metcalf , Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , X86 ML , Chris Zankel , Max Filippov , Arnd Bergmann , Greg Kroah-Hartman , Laurent Pinchart , Mauro Carvalho Chehab , Pawel Osciak , Marek Szyprowski , Kyungmin Park , David Woodhouse , Brian Norris , Boris Brezillon , Marek Vasut , Richard Weinberger , Cyrille Pitchen , Felipe Balbi , Alexander Viro , Benjamin LaHaise , Nadia Yvette Chambers , Jeff Layton , "J. Bruce Fields" , Peter Zijlstra , Hugh Dickins , Arnaldo Carvalho de Melo , Alexander Shishkin , Jaroslav Kysela , Takashi Iwai , "linux-kernel@vger.kernel.org" , linux-alpha@vger.kernel.org, arcml , "linux-arm-kernel@lists.infradead.org" , adi-buildroot-devel@lists.sourceforge.net, linux-hexagon@vger.kernel.org, "linux-ia64@vger.kernel.org" , linux-metag@vger.kernel.org, Linux MIPS Mailing List , linux-parisc@vger.kernel.org, linuxppc-dev , "linux-s390@vger.kernel.org" , "linux-sh@vger.kernel.org" , sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, Linux Media Mailing List , linux-mtd@lists.infradead.org, USB list , Linux FS Devel , linux-aio@kvack.org, "linux-mm@kvack.org" , Linux API , linux-arch , ALSA development Message-ID: <20170315213043.3xaZA8D2gCDmt2qdNGg8fJGZc8aIrSuwfCnN989AonA@z> On Wed, 15 Mar 2017, Rich Felker wrote: > On Wed, Mar 15, 2017 at 12:44:47PM -0700, Till Smejkal wrote: > > On Wed, 15 Mar 2017, Andy Lutomirski wrote: > > > > One advantage of VAS segments is that they can be globally queried by user programs > > > > which means that VAS segments can be shared by applications that not necessarily have > > > > to be related. If I am not mistaken, MAP_SHARED of pure in memory data will only work > > > > if the tasks that share the memory region are related (aka. have a common parent that > > > > initialized the shared mapping). Otherwise, the shared mapping have to be backed by a > > > > file. > > > > > > What's wrong with memfd_create()? > > > > > > > VAS segments on the other side allow sharing of pure in memory data by > > > > arbitrary related tasks without the need of a file. This becomes especially > > > > interesting if one combines VAS segments with non-volatile memory since one can keep > > > > data structures in the NVM and still be able to share them between multiple tasks. > > > > > > What's wrong with regular mmap? > > > > I never wanted to say that there is something wrong with regular mmap. We just > > figured that with VAS segments you could remove the need to mmap your shared data but > > instead can keep everything purely in memory. > > > > Unfortunately, I am not at full speed with memfds. Is my understanding correct that > > if the last user of such a file descriptor closes it, the corresponding memory is > > freed? Accordingly, memfd cannot be used to keep data in memory while no program is > > currently using it, can it? To be able to do this you need again some representation > > I have a name for application-allocated kernel resources that persist > without a process holding a reference to them or a node in the > filesystem: a bug. See: sysvipc. VAS segments are first class citizens of the OS similar to processes. Accordingly, I would not see this behavior as a bug. VAS segments are a kernel handle to "persistent" memory (in the sense that they are independent of the lifetime of the application that created them). That means the memory that is described by VAS segments can be reused by other applications even if the VAS segment was not used by any application in between. It is very much like a pure in-memory file. An application creates a VAS segment, fills it with content and if it does not delete it again, can reuse/open it again later. This also means, that if you know that you never want to use this memory again you have to remove it explicitly, like you have to remove a file, if you don't want to use it anymore. I think it really might be better to implement VAS segments (if I should keep this feature at all) with a special purpose filesystem. The way I've designed it seams to be very misleading. Till