* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation [not found] ` <3VNYt-4M4-15@gated-at.bofh.it> @ 2005-04-22 13:10 ` Bodo Eggert <harvested.in.lkml@posting.7eggert.dyndns.org> 2005-04-22 17:01 ` [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbsimplementation Fab Tillier 0 siblings, 1 reply; 59+ messages in thread From: Bodo Eggert <harvested.in.lkml@posting.7eggert.dyndns.org> @ 2005-04-22 13:10 UTC (permalink / raw) To: Andy Isaacson, Timur Tabi, Troy Benjegerdes, Bernhard Fischer, Arjan van de Ven, linux-kernel, openib-general Andy Isaacson <adi@hexapodia.org> wrote: > On Wed, Apr 20, 2005 at 10:07:45PM -0500, Timur Tabi wrote: >> I don't know if VM_REGISTERED is a good idea or not, but it should be >> absolutely impossible for the kernel to reclaim "registered" (aka pinned) >> memory, no matter what. For RDMA services (such as Infiniband, iWARP, etc), >> it's normal for non-root processes to pin hundreds of megabytes of memory, >> and that memory better be locked to those physical pages until the >> application deregisters them. > > If you take the hardline position that "the app is the only thing that > matters", your code is unlikely to get merged. Linux is a > general-purpose OS. All userspace hardware drivers with DMA will require pinned pages (and some of them will require continuous memory). Since this memory may be scheduled to be accessed by DMA, reclaiming those pages may (aka. will) result in "random" memory corruption unless done by the driver itself. You can't even set a time limit, the driver may have allocated all DMA memory to queued transfers, and some media needs to get plugged in by the lazy robot. As soon as the robot arrives - boom. (For the same reason, this memory MUST NOT be freed if the application terminates abnormally, e.g. killed by OOM). In other words, you need to make this memory as unaccessible as the framebuffer on a graphic card. If that causes a lockup, you better had prevented that while allocating. > In a Linux context, I doubt that fullblown SA is necessary or > appropriate. Rather, I'd suggest two new signals, SIGMEMLOW and > SIGMEMCRIT. The userland comms library registers handlers for both. > When the kernel decides that it needs to reclaim some memory from the > app, it sends SIGMEMLOW. The comms library then has the responsibility > to un-reserve some memory in an orderly fashion. If a reasonable [1] > time has expired since SIGMEMLOW and the kernel is still hungry, the > kernel sends SIGMEMCRIT. At this point, the comms lib *must* unregister > some memory [2] even if it has to drop state to do so; if it returns > from the signal handler without having unregistered the memory, the > kernel will SIGKILL. Choosing Data loss vs. finitely stalled system may sometimes be a bad decision. If I designes an application that might get a "gimme memory or die", I'd reserve an extra bunch of memory with the only purpose of being released in this situation. If the kernel had done that instead, this part of memory could have been used e.g. as a read-only disk cache in the meantime (off cause provided somebody cared to implement that). > [2] Is there a way for the kernel to pass down to userspace how many > pages it wants, maybe in the sigcontext? Then you'd need only one signal. I think this interface is usefull, it would e.g. allow a picture viewer to cache as many decoded and scaled pictures as the RAM permits, freeing them if the RAM gets full and the swap would have to be used. -- "When the pin is pulled, Mr. Grenade is not our friend. -U.S. Marine Corps ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbsimplementation 2005-04-22 13:10 ` [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation Bodo Eggert <harvested.in.lkml@posting.7eggert.dyndns.org> @ 2005-04-22 17:01 ` Fab Tillier 2005-04-22 22:01 ` Bodo Eggert 0 siblings, 1 reply; 59+ messages in thread From: Fab Tillier @ 2005-04-22 17:01 UTC (permalink / raw) To: 'Bodo Eggert <harvested.in.lkml@posting.7eggert.dyndns.org>', Andy Isaacson, Timur Tabi, Troy Benjegerdes, Bernhard Fischer, Arjan van de Ven, linux-kernel, openib-general > From: Bodo Eggert <harvested.in.lkml@posting.7eggert.dyndns.org> > Sent: Friday, April 22, 2005 6:10 AM > > All userspace hardware drivers with DMA will require pinned pages (and > some of them will require continuous memory). Since this memory may be > scheduled to be accessed by DMA, reclaiming those pages may (aka. will) > result in "random" memory corruption unless done by the driver itself. Any reclaim must involve the driver. That doesn't mean that it must involve the application. That said this isn't trivial to implement. > > You can't even set a time limit, the driver may have allocated all DMA > memory to queued transfers, and some media needs to get plugged in by > the lazy robot. As soon as the robot arrives - boom. (For the same reason, > this memory MUST NOT be freed if the application terminates abnormally, > e.g. killed by OOM). InfiniBand provides support for deregistering memory that might be referenced at some future time by an RDMA operation. The only side effect this has is that the QP on both sides of the connection transition to an error state. Upon abnormal termination, all registrations must be undone and the memory unpinned. This must be synchronized with the hardware so that there are no races. The IB deregistration semantics provide such synchronization. I'd venture that any HW design that does not do this is broken. Requiring the memory to never be freed upon abnormal termination equates to a serious memory leak, in that physical memory is leaked, not virtual. - Fab ^ permalink raw reply [flat|nested] 59+ messages in thread
* RE: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbsimplementation 2005-04-22 17:01 ` [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbsimplementation Fab Tillier @ 2005-04-22 22:01 ` Bodo Eggert 0 siblings, 0 replies; 59+ messages in thread From: Bodo Eggert @ 2005-04-22 22:01 UTC (permalink / raw) To: Fab Tillier Cc: 'Bodo Eggert <harvested.in.lkml@posting.7eggert.dyndns.org>', Andy Isaacson, Timur Tabi, Troy Benjegerdes, Bernhard Fischer, Arjan van de Ven, linux-kernel, openib-general On Fri, 22 Apr 2005, Fab Tillier wrote: > > From: Bodo Eggert <harvested.in.lkml@posting.7eggert.dyndns.org> > > Sent: Friday, April 22, 2005 6:10 AM > > You can't even set a time limit, the driver may have allocated all DMA > > memory to queued transfers, and some media needs to get plugged in by > > the lazy robot. As soon as the robot arrives - boom. (For the same reason, > > this memory MUST NOT be freed if the application terminates abnormally, > > e.g. killed by OOM). > > InfiniBand provides support for deregistering memory that might be > referenced at some future time by an RDMA operation. The only side effect > this has is that the QP on both sides of the connection transition to an > error state. > > Upon abnormal termination, all registrations must be undone and the memory > unpinned. This must be synchronized with the hardware so that there are no > races. If you know the hardware. If you have userspace drivers, this will be impossible, and even if you have kernel drivers, you'll need to know which of them is responsible for each part of the pinned memory. This doesn't imply the affected memory to be lost. The same application that created the pinned memory can reset the hardware (provided nobody changed the configuration), then reconnect to the shared memory segment you'll use for that purpose and use or free it. -- To iterate is human; to recurse, divine. ^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH][RFC][0/4] InfiniBand userspace verbs implementation @ 2005-04-04 22:09 Roland Dreier 2005-04-11 14:22 ` Troy Benjegerdes 0 siblings, 1 reply; 59+ messages in thread From: Roland Dreier @ 2005-04-04 22:09 UTC (permalink / raw) To: linux-kernel, openib-general Here is an initial implementation of InfiniBand userspace verbs. I plan to commit this code to the OpenIB repository shortly, and submit it for inclusion during the 2.6.13 cycle, so I am posting it early for comments. This code, in conjunction with the libibverbs and libmthca userspace libraries available from the subversion trees at https://openib.org/svn/gen2/branches/roland-uverbs/src/userspace/libibverbs https://openib.org/svn/gen2/branches/roland-uverbs/src/userspace/libmthca enables userspace processes to access InfiniBand HCAs directly. For those not familiar with the InfiniBand architecture, this so-called "userspace verbs" support allows userspace to post data path commands directly to the HCA. Resource allocation and other control path operations still go through the kernel driver. Please take a look at this code if you have a chance. I would appreciate high-level criticism of the design and implementation as well as nitpicky complaints about coding style and typos. In particular, the memory pinning code in in uverbs_mem.c could stand a looking over. In addition, a sanity check of the write()-based scheme for passing commands into the kernel in uverbs_main.c and uverbs_cmd.c is probably worthwhile. Thanks, Roland ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-04 22:09 [PATCH][RFC][0/4] InfiniBand userspace verbs implementation Roland Dreier @ 2005-04-11 14:22 ` Troy Benjegerdes 2005-04-11 15:34 ` Roland Dreier 0 siblings, 1 reply; 59+ messages in thread From: Troy Benjegerdes @ 2005-04-11 14:22 UTC (permalink / raw) To: Roland Dreier; +Cc: linux-kernel, openib-general > In particular, the memory pinning code in in uverbs_mem.c could stand > a looking over. In addition, a sanity check of the write()-based > scheme for passing commands into the kernel in uverbs_main.c and > uverbs_cmd.c is probably worthwhile. How is memory pinning handled? (I haven't had time to read all the code, so please excuse my ignorance of something obvious). The old mellanox drivers used to have a hack to call 'sys_mlock', and promiscuously lock memory any old userspace application asked for. What is the API for the new uverbs memory registration, and how will things like memory hotplug and NUMA page migration be able to unpin pages locked by a user program? I have applications that would benefit from being able to register 15GB of memory on a machine with 16GB. Right now, MPI and other possible users of infiniband in userspace have to play cacheing games and limit what they can register. But locking all that memory without providing the kernel a way to unlock it under memory pressure or for page migration seems like a bad idea. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-11 14:22 ` Troy Benjegerdes @ 2005-04-11 15:34 ` Roland Dreier 2005-04-11 16:33 ` Troy Benjegerdes 2005-04-18 16:09 ` Timur Tabi 0 siblings, 2 replies; 59+ messages in thread From: Roland Dreier @ 2005-04-11 15:34 UTC (permalink / raw) To: Troy Benjegerdes; +Cc: linux-kernel, openib-general Troy> How is memory pinning handled? (I haven't had time to read Troy> all the code, so please excuse my ignorance of something Troy> obvious). The userspace library calls mlock() and then the kernel does get_user_pages(). Troy> The old mellanox drivers used to have a hack to call Troy> 'sys_mlock', and promiscuously lock memory any old userspace Troy> application asked for. What is the API for the new uverbs Troy> memory registration, and how will things like memory hotplug Troy> and NUMA page migration be able to unpin pages locked by a Troy> user program? The API for uverbs memory registration is ibv_reg_mr(), and right now the memory is pinned and that's it. - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-11 15:34 ` Roland Dreier @ 2005-04-11 16:33 ` Troy Benjegerdes 2005-04-11 16:56 ` Roland Dreier 2005-04-18 16:09 ` Timur Tabi 1 sibling, 1 reply; 59+ messages in thread From: Troy Benjegerdes @ 2005-04-11 16:33 UTC (permalink / raw) To: Roland Dreier; +Cc: linux-kernel, openib-general On Mon, Apr 11, 2005 at 08:34:19AM -0700, Roland Dreier wrote: > Troy> How is memory pinning handled? (I haven't had time to read > Troy> all the code, so please excuse my ignorance of something > Troy> obvious). > > The userspace library calls mlock() and then the kernel does > get_user_pages(). Is there a check in the kernel that the memory is actually mlock()ed? What if a malicious (or broken) application does ibv_reg_mr() but doesn't lock the memory? Does the IB card get a physical address for a page that might get swapped out? > Troy> The old mellanox drivers used to have a hack to call > Troy> 'sys_mlock', and promiscuously lock memory any old userspace > Troy> application asked for. What is the API for the new uverbs > Troy> memory registration, and how will things like memory hotplug > Troy> and NUMA page migration be able to unpin pages locked by a > Troy> user program? > > The API for uverbs memory registration is ibv_reg_mr(), and right now > the memory is pinned and that's it. > > - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-11 16:33 ` Troy Benjegerdes @ 2005-04-11 16:56 ` Roland Dreier 2005-04-11 18:01 ` Troy Benjegerdes 0 siblings, 1 reply; 59+ messages in thread From: Roland Dreier @ 2005-04-11 16:56 UTC (permalink / raw) To: Troy Benjegerdes; +Cc: linux-kernel, openib-general Troy> Is there a check in the kernel that the memory is actually Troy> mlock()ed? No. Troy> What if a malicious (or broken) application does Troy> ibv_reg_mr() but doesn't lock the memory? Does the IB card Troy> get a physical address for a page that might get swapped Troy> out? No, the kernel does get_user_pages(). So the pages that the HCA gets will not be swapped or used for anything else. The only thing a malicious userspace app can do is screw itself up. - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-11 16:56 ` Roland Dreier @ 2005-04-11 18:01 ` Troy Benjegerdes 2005-04-11 18:03 ` Roland Dreier 0 siblings, 1 reply; 59+ messages in thread From: Troy Benjegerdes @ 2005-04-11 18:01 UTC (permalink / raw) To: Roland Dreier; +Cc: linux-kernel, openib-general On Mon, Apr 11, 2005 at 09:56:53AM -0700, Roland Dreier wrote: > Troy> Is there a check in the kernel that the memory is actually > Troy> mlock()ed? > > No. > > Troy> What if a malicious (or broken) application does > Troy> ibv_reg_mr() but doesn't lock the memory? Does the IB card > Troy> get a physical address for a page that might get swapped > Troy> out? > > No, the kernel does get_user_pages(). So the pages that the HCA gets > will not be swapped or used for anything else. The only thing a > malicious userspace app can do is screw itself up. > > - R. Do we even need the mlock in userspace then? ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-11 18:01 ` Troy Benjegerdes @ 2005-04-11 18:03 ` Roland Dreier 2005-04-12 0:13 ` Andrew Morton 0 siblings, 1 reply; 59+ messages in thread From: Roland Dreier @ 2005-04-11 18:03 UTC (permalink / raw) To: Troy Benjegerdes; +Cc: linux-kernel, openib-general Troy> Do we even need the mlock in userspace then? Yes, because the kernel may go through and unmap pages from userspace while trying to swap. Since we have the page locked in the kernel, the physical page won't go anywhere, but userspace might end up with a different page mapped at the same virtual address. - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-11 18:03 ` Roland Dreier @ 2005-04-12 0:13 ` Andrew Morton 2005-04-13 1:04 ` [openib-general] " Libor Michalek 2005-04-18 16:22 ` Timur Tabi 0 siblings, 2 replies; 59+ messages in thread From: Andrew Morton @ 2005-04-12 0:13 UTC (permalink / raw) To: Roland Dreier; +Cc: hozer, linux-kernel, openib-general Roland Dreier <roland@topspin.com> wrote: > > Troy> Do we even need the mlock in userspace then? > > Yes, because the kernel may go through and unmap pages from userspace > while trying to swap. Since we have the page locked in the kernel, > the physical page won't go anywhere, but userspace might end up with a > different page mapped at the same virtual address. That shouldn't happen. If get_user_pages() has elevated the refcount on a page then the following can happen: - The VM may decide to add the page to swapcache (if it's not mmapped from a file). - Once the page is backed by either swapcache of a (mmapped) file, the VM may decide the unmap the application's pte's. A later minor fault by the app will cause the same physical page to be remapped. - The VM may decide to try to write the page to its backing file or swap. If it does, the page is still in core, but is now clean. - Once all pte's are unmapped and the page is clean, the VM may decide to try to reclaim the page. The VM will then see the elevated refcount and will bale out, leaving the page in core. - If your code was doing a read-from-disk (modifying memory), then your code should run set_page_dirty() or set_page_dirty_lock() against the page before dropping the refcount which get_user_pages() added. Once the page is dirty, the VM can't reclaim it until it has been been written to swap or mmapped backing file. IOW: while the page has an elevated refcount from get_user_pages(), that physical page is 100% pinned. Once you've done the set_page_dirty+put_page(), the page is again under control of the VM. There should be no need to run mlock() from userspace. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-12 0:13 ` Andrew Morton @ 2005-04-13 1:04 ` Libor Michalek 2005-04-18 17:15 ` Timur Tabi 2005-04-26 3:31 ` Libor Michalek 2005-04-18 16:22 ` Timur Tabi 1 sibling, 2 replies; 59+ messages in thread From: Libor Michalek @ 2005-04-13 1:04 UTC (permalink / raw) To: Andrew Morton; +Cc: Roland Dreier, linux-kernel, openib-general On Mon, Apr 11, 2005 at 05:13:47PM -0700, Andrew Morton wrote: > Roland Dreier <roland@topspin.com> wrote: > > > > Troy> Do we even need the mlock in userspace then? > > > > Yes, because the kernel may go through and unmap pages from userspace > > while trying to swap. Since we have the page locked in the kernel, > > the physical page won't go anywhere, but userspace might end up with a > > different page mapped at the same virtual address. With the last few kernels I haven't had a chance to retest the problem that pushed us in the direction of using mlock. I will go back and do so with the latest kernel. Below I've given a quick description of the issue. > That shouldn't happen. If get_user_pages() has elevated the refcount on a > page then the following can happen: > > - The VM may decide to add the page to swapcache (if it's not mmapped > from a file). > > - Once the page is backed by either swapcache of a (mmapped) file, the VM > may decide the unmap the application's pte's. A later minor fault by the > app will cause the same physical page to be remapped. The driver did use get_user_pages() to elevated the refcount on all the pages it was going to use for IO, as well as call set_page_dirty() since the pages were going to have data written to them from the device. The problem we were seeing is that the minor fault by the app resulted in a new physical page getting mapped for the application. The page that had the elevated refcount was still waiting for the data to be written to by the driver at the time that the app accessed the page causing the minor fault. Obviously since the app had a new mapping the data written by the driver was lost. It looks like code was added to try_to_unmap_one() to address this, so hopefully it's no longer an issue... -Libor ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-13 1:04 ` [openib-general] " Libor Michalek @ 2005-04-18 17:15 ` Timur Tabi 2005-04-26 3:31 ` Libor Michalek 1 sibling, 0 replies; 59+ messages in thread From: Timur Tabi @ 2005-04-18 17:15 UTC (permalink / raw) To: Libor Michalek; +Cc: Andrew Morton, Roland Dreier, linux-kernel, openib-general Libor Michalek wrote: > The problem we were seeing is that the minor fault by the app resulted > in a new physical page getting mapped for the application. The page that > had the elevated refcount was still waiting for the data to be written > to by the driver at the time that the app accessed the page causing the > minor fault. Obviously since the app had a new mapping the data written > by the driver was lost. Thanks Libor, this is much better explanation of the problem than what I posted. > It looks like code was added to try_to_unmap_one() to address this, so > hopefully it's no longer an issue... I doubt it. I tried this with an earlier 2.6 kernel, and get_user_pages() was still not enough to really pin the memory down. Maybe it works in 2.6.12, but that doesn't help me any, because our driver needs to support all 2.4 and 2.6 kernels. Currently, mlock() alone seems to be good enough, but I'm going to add calls to get_user_pages() just to be sure. -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-13 1:04 ` [openib-general] " Libor Michalek 2005-04-18 17:15 ` Timur Tabi @ 2005-04-26 3:31 ` Libor Michalek 2005-05-04 18:27 ` Timur Tabi 1 sibling, 1 reply; 59+ messages in thread From: Libor Michalek @ 2005-04-26 3:31 UTC (permalink / raw) To: Andrew Morton; +Cc: linux-kernel, openib-general [-- Attachment #1: Type: text/plain, Size: 2775 bytes --] On Tue, Apr 12, 2005 at 06:04:47PM -0700, Libor Michalek wrote: > On Mon, Apr 11, 2005 at 05:13:47PM -0700, Andrew Morton wrote: > > Roland Dreier <roland@topspin.com> wrote: > > > > > > Troy> Do we even need the mlock in userspace then? > > > > > > Yes, because the kernel may go through and unmap pages from userspace > > > while trying to swap. Since we have the page locked in the kernel, > > > the physical page won't go anywhere, but userspace might end up with a > > > different page mapped at the same virtual address. > > With the last few kernels I haven't had a chance to retest the problem > that pushed us in the direction of using mlock. I will go back and do > so with the latest kernel. Below I've given a quick description of the > issue. > > > That shouldn't happen. If get_user_pages() has elevated the refcount on a > > page then the following can happen: > > > > - The VM may decide to add the page to swapcache (if it's not mmapped > > from a file). > > > > - Once the page is backed by either swapcache of a (mmapped) file, the VM > > may decide the unmap the application's pte's. A later minor fault by the > > app will cause the same physical page to be remapped. > > The driver did use get_user_pages() to elevated the refcount on all the > pages it was going to use for IO, as well as call set_page_dirty() since > the pages were going to have data written to them from the device. > > The problem we were seeing is that the minor fault by the app resulted > in a new physical page getting mapped for the application. The page that > had the elevated refcount was still waiting for the data to be written > to by the driver at the time that the app accessed the page causing the > minor fault. Obviously since the app had a new mapping the data written > by the driver was lost. > > It looks like code was added to try_to_unmap_one() to address this, so > hopefully it's no longer an issue... I wrote a quick test module and program to confirm that the problem we saw in older kernels with get_user_pages() no longer exists. The module creates a character device with three different ioctl commands: - Pin the pages of a buffer using get_user_pages() - Check the pages by calling get_user_pages() a second time and comparing the new and original page list. - Relase the pages using put_page() The program opens the charcter device file descriptor, pins the pages and waits for a signal, before checking the pages, which is sent to the process after running some other program which exercises the VM. On older kernels the check fails, on my 2.6.11 kernel the check succeeds. So mlock is not needed on top of get_user_pages() as it was before. Thanks for the heads up. Module and program attached. -Libor [-- Attachment #2: mltest.c --] [-- Type: text/plain, Size: 7203 bytes --] /* * Copyright (c) 2005 Topspin Communications. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU * General Public License (GPL) Version 2, available from the file * COPYING in the main directory of this source tree, or the * OpenIB.org BSD license below: * * Redistribution and use in source and binary forms, with or * without modification, are permitted provided that the following * conditions are met: * * - Redistributions of source code must retain the above * copyright notice, this list of conditions and the following * disclaimer. * * - Redistributions in binary form must reproduce the above * copyright notice, this list of conditions and the following * disclaimer in the documentation and/or other materials * provided with the distribution. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE * SOFTWARE. * * $Id: $ */ #include <linux/init.h> #include <linux/fs.h> #include <linux/module.h> #include <linux/device.h> #include <linux/err.h> #include <linux/poll.h> #include <linux/file.h> #include <linux/mount.h> #include <linux/cdev.h> #include <linux/devfs_fs_kernel.h> #include <asm/uaccess.h> #include <asm/highmem.h> MODULE_AUTHOR("Libor Michalek"); MODULE_DESCRIPTION("Get pages test"); MODULE_LICENSE("GPL"); enum { TEST_MAJOR = 232, TEST_MINOR = 255 }; #define TEST_DEV MKDEV(TEST_MAJOR, TEST_MINOR) enum { TEST_CMD_REGISTER = 1, TEST_CMD_UNREGISTER = 2, TEST_CMD_CHECK = 3 }; struct ioctl_arg { __u64 addr; __u64 size; }; struct region_root { struct semaphore mutex; struct list_head regions; /* list of pending events. */ struct file *filp; int nr_region; }; struct test_region { unsigned long user; unsigned long addr; unsigned long size; int nr_pages; struct page **pages; struct region_root *root; struct list_head region_list; /* member in root region list */ }; static void test_unlock(struct test_region *region) { long i; list_del(®ion->region_list); for (i = 0; i < region->nr_pages; i++) put_page(region->pages[i]); printk(KERN_ERR "TEST: Unlocked address <%016lx>\n", region->user); kfree(region->pages); kfree(region); } static struct test_region *test_lookup(struct region_root *root, unsigned long addr) { struct test_region *region; list_for_each_entry(region, &root->regions, region_list) if (region->user == addr) return region; return NULL; } static int test_lock(struct region_root *root, unsigned long uaddr, unsigned long size) { struct test_region *region; int nr_pages; int result; region = kmalloc(sizeof(*region), GFP_KERNEL); if (!region) return -ENOMEM; region->user = uaddr; region->addr = uaddr & PAGE_MASK; region->size = PAGE_ALIGN(size + (uaddr & ~PAGE_MASK)); region->root = root; nr_pages = (region->size + PAGE_SIZE-1) >> PAGE_SHIFT; region->pages = kmalloc(sizeof(struct page *) * nr_pages, GFP_KERNEL); if (!region->pages) { result = -ENOMEM; goto page_err; } region->nr_pages = get_user_pages(current, current->mm, region->addr, nr_pages, 1, 0, region->pages, NULL); if (region->nr_pages != nr_pages) { result = -EFAULT; goto get_err; } list_add_tail(®ion->region_list, &root->regions); printk(KERN_ERR "TEST: Locked address <%016lx>\n", region->user); return 0; get_err: kfree(region->pages); page_err: kfree(region); return result; } static int test_check(struct test_region *region) { struct page **pages; int nr_pages; int result = 0; int i; pages = kmalloc(sizeof(struct page *) * region->nr_pages, GFP_KERNEL); if (!pages) return -ENOMEM; nr_pages = get_user_pages(current, current->mm, region->addr, region->nr_pages, 1, 0, pages, NULL); if (region->nr_pages != nr_pages) { result = -EFAULT; goto get_err; } for (i = 0; i < nr_pages; i++) { if (region->pages[i] != pages[i]) printk(KERN_ERR "TEST: Check error <%p:%p> " "page <%u> of <%u>\n", pages[i], region->pages[i], i, nr_pages); put_page(pages[i]); } get_err: kfree(pages); return result; } static long test_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) { struct region_root *root = filp->private_data; struct test_region *region; struct ioctl_arg ureq; int result = 0; if (!root) return -EINVAL; if (copy_from_user(&ureq, (void __user *)arg, sizeof(ureq))) return -EFAULT; down(&root->mutex); switch (cmd) { case TEST_CMD_REGISTER: result = test_lock(root, ureq.addr, ureq.size); break; case TEST_CMD_UNREGISTER: region = test_lookup(root, ureq.addr); if (!region) result = -ENOENT; else test_unlock(region); break; case TEST_CMD_CHECK: region = test_lookup(root, ureq.addr); if (!region) result = -ENOENT; else result = test_check(region); break; default: result = -ERANGE; break; } up(&root->mutex); return result; } static int test_open(struct inode *inode, struct file *filp) { struct region_root *root; root = kmalloc(sizeof(*root), GFP_KERNEL); if (!root) return -ENOMEM; memset(root, 0, sizeof(*root)); INIT_LIST_HEAD(&root->regions); init_MUTEX(&root->mutex); filp->private_data = root; root->filp = filp; printk(KERN_ERR "TEST: Created root struct\n"); return 0; } static int test_close(struct inode *inode, struct file *filp) { struct region_root *root = filp->private_data; struct test_region *region; down(&root->mutex); while (!list_empty(&root->regions)) { region = list_entry(root->regions.next, struct test_region, region_list); test_unlock(region); } up(&root->mutex); kfree(root); filp->private_data = NULL; printk(KERN_ERR "TEST: Deleted root struct\n"); return 0; } static struct file_operations test_fops = { .owner = THIS_MODULE, .open = test_open, .release = test_close, .compat_ioctl = test_ioctl, .unlocked_ioctl = test_ioctl, }; static struct cdev test_cdev; static int __init test_init(void) { int result; result = register_chrdev_region(TEST_DEV, 1, "mltest"); if (result) { printk(KERN_ERR "TEST: Error <%d> registering dev\n", result); goto err_chr; } cdev_init(&test_cdev, &test_fops); result = cdev_add(&test_cdev, TEST_DEV, 1); if (result) { printk(KERN_ERR "TEST: Error <%d> adding cdev\n", result); goto err_cdev; } return 0; err_cdev: unregister_chrdev_region(TEST_DEV, 1); err_chr: return result; } static void __exit test_cleanup(void) { cdev_del(&test_cdev); unregister_chrdev_region(TEST_DEV, 1); } module_init(test_init); module_exit(test_cleanup); [-- Attachment #3: umlt.c --] [-- Type: text/plain, Size: 3356 bytes --] /* * Copyright (c) 2005 Topspin Communications. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU * General Public License (GPL) Version 2, available from the file * COPYING in the main directory of this source tree, or the * OpenIB.org BSD license below: * * Redistribution and use in source and binary forms, with or * without modification, are permitted provided that the following * conditions are met: * * - Redistributions of source code must retain the above * copyright notice, this list of conditions and the following * disclaimer. * * - Redistributions in binary form must reproduce the above * copyright notice, this list of conditions and the following * disclaimer in the documentation and/or other materials * provided with the distribution. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE * SOFTWARE. * * $Id: $ */ #include <stdlib.h> #include <string.h> #include <glob.h> #include <stdio.h> #include <fcntl.h> #include <errno.h> #include <stdint.h> #include <poll.h> #include <unistd.h> #include <signal.h> #include <linux/types.h> #define TEST_DEV_PATH "/dev/mltest" #define TEST_SLEEP_UTIME 50000 struct ioctl_arg { __u64 addr; __u64 size; }; enum { TEST_CMD_REGISTER = 1, TEST_CMD_UNREGISTER = 2, TEST_CMD_CHECK = 3 }; static int hold = 1; void sig_usr(int value) { hold = 0; } int main(int argc, char **argv) { struct ioctl_arg req; void *data; int param_c = 0; int size; int fd; int result; if (2 != argc || 0 > (size = atoi(argv[++param_c]))) { fprintf(stderr, "usage: %s <size>\n", argv[0]); fprintf(stderr, " size - allocated region size in bytes.\n"); exit(1); } signal(SIGUSR1, sig_usr); data = malloc(size); if (!data) { fprintf(stderr, "Failed to allocated region of size <%d>\n", size); exit(1); } fd = open(TEST_DEV_PATH, O_RDWR); if (fd < 0) { fprintf(stderr, "Error <%d:%d> opening device <%s>\n", fd, errno, TEST_DEV_PATH); goto open_err; } req.addr = (unsigned long)data; req.size = size; result = ioctl(fd, TEST_CMD_REGISTER, &req); if (result) { fprintf(stderr, "Error <%d:%d> registering region\n", result, errno); goto ioctl_err; } fprintf(stdout, "Address <%016lx> registered, process <%d> waiting...\n", data, getpid()); while (hold) { usleep(TEST_SLEEP_UTIME); } fprintf(stdout, "Process continuing, checking address <%016lx>\n", data); result = ioctl(fd, TEST_CMD_CHECK, &req); if (result) { fprintf(stderr, "Error <%d:%d> checking region\n", result, errno); goto ioctl_err; } result = ioctl(fd, TEST_CMD_UNREGISTER, &req); if (result) { fprintf(stderr, "Error <%d:%d> unregistering region\n", result, errno); goto ioctl_err; } ioctl_err: close(fd); open_err: free(data); return 0; } ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 3:31 ` Libor Michalek @ 2005-05-04 18:27 ` Timur Tabi 2005-05-05 18:48 ` Timur Tabi 2005-05-05 23:34 ` Libor Michalek 0 siblings, 2 replies; 59+ messages in thread From: Timur Tabi @ 2005-05-04 18:27 UTC (permalink / raw) To: Libor Michalek; +Cc: Andrew Morton, linux-kernel, openib-general Libor Michalek wrote: > The program opens the charcter device file descriptor, pins the pages > and waits for a signal, before checking the pages, which is sent to the > process after running some other program which exercises the VM. On older > kernels the check fails, on my 2.6.11 kernel the check succeeds. So > mlock is not needed on top of get_user_pages() as it was before. Libor, When you say "older", what exactly do you mean? I have different test that normally fails with just get_user_pages(), but it works with 2.6.9 and above. I haven't been able to get any kernel earlier than 2.6.9 to compile or boot properly, so I'm having a hard time narrowing down the actual point when get_user_pages() started working. -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-04 18:27 ` Timur Tabi @ 2005-05-05 18:48 ` Timur Tabi 2005-05-06 23:08 ` Timur Tabi 2005-05-05 23:34 ` Libor Michalek 1 sibling, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-05-05 18:48 UTC (permalink / raw) To: Timur Tabi; +Cc: Libor Michalek, Andrew Morton, linux-kernel, openib-general Timur Tabi wrote: > When you say "older", what exactly do you mean? I have different test > that normally fails with just get_user_pages(), but it works with 2.6.9 > and above. I haven't been able to get any kernel earlier than 2.6.9 to > compile or boot properly, so I'm having a hard time narrowing down the > actual point when get_user_pages() started working. I haven't gotten a reply to this question, but I've done my own research, and I think I found the answer. Using my own test of get_user_pages(), it appears that the fix was placed in 2.6.7. However, I would like to know specifically what the fix is. Unfortunately, tracking this stuff down is beyond my understanding of the Linux VM. Assuming that the fix is in try_to_unmap_one(), the only significant change I see between 2.6.6 and 2.6.7 is the addition of this code: pgd = pgd_offset(mm, address); if (!pgd_present(*pgd)) goto out_unlock; pmd = pmd_offset(pgd, address); if (!pmd_present(*pmd)) goto out_unlock; pte = pte_offset_map(pmd, address); if (!pte_present(*pte)) goto out_unmap; if (page_to_pfn(page) != pte_pfn(*pte)) goto out_unmap; Can anyone tell me if this is the actual fix, or at least a major part of the actual fix? -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-05 18:48 ` Timur Tabi @ 2005-05-06 23:08 ` Timur Tabi 2005-05-07 13:18 ` Hugh Dickins 0 siblings, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-05-06 23:08 UTC (permalink / raw) To: Timur Tabi; +Cc: Libor Michalek, Andrew Morton, linux-kernel, openib-general Timur Tabi wrote: > I haven't gotten a reply to this question, but I've done my own > research, and I think I found the answer. Using my own test of > get_user_pages(), it appears that the fix was placed in 2.6.7. However, > I would like to know specifically what the fix is. Unfortunately, > tracking this stuff down is beyond my understanding of the Linux VM. I'm also still waiting for a reply to this question. Anyone???? Upon doing some more research, I think the fix might be those code instead: /* * Don't pull an anonymous page out from under get_user_pages. * GUP carefully breaks COW and raises page count (while holding * page_table_lock, as we have here) to make sure that the page * cannot be freed. If we unmap that page here, a user write * access to the virtual address will bring back the page, but * its raised count will (ironically) be taken to mean it's not * an exclusive swap page, do_wp_page will replace it by a copy * page, and the user never get to see the data GUP was holding * the original page for. */ if (PageSwapCache(page) && page_count(page) != page->mapcount + 2) { ret = SWAP_FAIL; goto out_unmap; } Both this change and the other one I mentioned are new to 2.6.7. I suppose I could try applying these patches to the 2.6.6 kernel and see if anything improves, but that won't help me understand what's really going on. The above comment makes sounds almost like it's a fix, but it talks about copy-on-write, which is has nothing to do with the real problem. -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-06 23:08 ` Timur Tabi @ 2005-05-07 13:18 ` Hugh Dickins 2005-05-07 14:45 ` Timur Tabi 0 siblings, 1 reply; 59+ messages in thread From: Hugh Dickins @ 2005-05-07 13:18 UTC (permalink / raw) To: Timur Tabi Cc: Libor Michalek, Andrew Morton, Andrea Arcangeli, linux-kernel, openib-general Sorry for not replying earlier (indeed, sorry for not joining in the wider RDMA pinning discussion), concentrating on other stuff at present. On Fri, 6 May 2005, Timur Tabi wrote: > Timur Tabi wrote: > > > I haven't gotten a reply to this question, but I've done my own research, > > and I think I found the answer. Using my own test of get_user_pages(), > > it appears that the fix was placed in 2.6.7. However, I would like to > > know specifically what the fix is. Unfortunately, tracking this stuff > > down is beyond my understanding of the Linux VM. > > I'm also still waiting for a reply to this question. Anyone???? > > Upon doing some more research, I think the fix might be those code instead: I believe you're right this time - I was rather puzzled by your earlier choice, then unhelpfully forgot to reply and point you a few lines further down to this comment, which does shout "get_user_pages fix" quite loudly. > /* > * Don't pull an anonymous page out from under get_user_pages. > * GUP carefully breaks COW and raises page count (while holding > * page_table_lock, as we have here) to make sure that the page > * cannot be freed. If we unmap that page here, a user write > * access to the virtual address will bring back the page, but > * its raised count will (ironically) be taken to mean it's not > * an exclusive swap page, do_wp_page will replace it by a copy > * page, and the user never get to see the data GUP was holding > * the original page for. > */ > if (PageSwapCache(page) && > page_count(page) != page->mapcount + 2) { > ret = SWAP_FAIL; > goto out_unmap; > } > > Both this change and the other one I mentioned are new to 2.6.7. I suppose I > could try applying these patches to the 2.6.6 kernel and see if anything > improves, but that won't help me understand what's really going on. There's a lot of change in the rmap area between 2.6.6 and 2.6.7, but you're right that this is an isolated fix, which could in principle be applied to earlier releases. Though I don't see it's worth doing now. > The above comment makes sounds almost like it's a fix, Almost? Sorry if my comment doesn't make it obvious it's a fix for a get_user_pages issue - I rewrote Andrea Arcangeli's original commment. The analysis and fix are his. > but it talks about copy-on-write, > which is has nothing to do with the real problem. Oh, well, maybe, but what is the real problem? Are you sure that copy-on-write doesn't come into it? I haven't reread through the whole thread, but my recollection is that you never quite said what the real problem is: you'd found some time ago that get_user_pages sometimes failed to pin the pages for your complex app, so were forced to mlock too; but couldn't provide any simple test case for the failure (which can indeed be a lot of work to devise), so we were all in the dark as to what went wrong. But you've now found that 2.6.7 and later kernels allow your app to work correctly without mlock, good. get_user_pages is certainly the right tool to use for such pinning. (On the question of whether mlock guarantees that user virtual addresses map to the same physical addresses, I prefer Arjan's view that it does not; but accept that there might prove to be difficulties in holding that position.) So, it works now, you've exonerated today's get_user_pages, and you've identified at least one get_user_pages fix which went in at that time: do we really need to chase this further? Oh, in writing of copy-on-write, I've just remembered another fix for get_user_pages which I made in 2.6.7 (though I've not heard of anyone seeing the problem fixed): call to do_wp_page in do_swap_page. get_user_pages assumes that the write fault it generates will break copy-on-write i.e. will make a private copy page when necessary, before returning to the caller; but that wasn't happening in the do_swap_page case. By the way, please don't be worried when soon the try_to_unmap_one comment and code that you identified above disappear. When I'm back in patch submission mode, I'll be sending Andrew a patch which removes it, instead reworking can_share_swap_page to rely on the page_mapcount instead of page_count, which avoids the ironical behaviour my comment refers to, and allows an awkward page migration case to proceed (once unpinned). Andrea and I now both prefer this page_mapcount approach. Hugh ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-07 13:18 ` Hugh Dickins @ 2005-05-07 14:45 ` Timur Tabi 2005-05-07 16:30 ` Hugh Dickins 0 siblings, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-05-07 14:45 UTC (permalink / raw) To: Hugh Dickins Cc: Libor Michalek, Andrew Morton, Andrea Arcangeli, linux-kernel, openib-general Hugh Dickins wrote: > Oh, well, maybe, but what is the real problem? > Are you sure that copy-on-write doesn't come into it? No, but I do know that my test case doesn't call fork(), so it's reproducible without involving COW. Of course, I'm sure someone's going to tell me now that COW comes into effect even without fork(). If so, please explain. > I haven't reread through the whole thread, but my recollection is > that you never quite said what the real problem is: you'd found some > time ago that get_user_pages sometimes failed to pin the pages for > your complex app, so were forced to mlock too; but couldn't provide > any simple test case for the failure (which can indeed be a lot of > work to devise), so we were all in the dark as to what went wrong. The short answer: under "extreme" memory pressure, the data inside a page pinned by get_user_pages() is swapped out, moved, or deleted (I'm not sure which). Some other data is placed into that physical location. By extreme memory pressure, I mean having the process allocate and touch as much memory as possible. Something like this: num_bytes = get_amount_of_physical_ram(); char *p = malloc(num_bytes); for (i=0; i<num_bytes; i+=PAGE_SIZE) p[i] = 0; The above over-simplified code fails on earlier 2.6 kernels (or earlier versions of glibc that accompany most distros the use the earlier 2.6 kernels). Either malloc() returns NULL, or the p[i]=0 part causes a segfault. I haven't bothered to trace down why. But when it does work, the page pinned by get_user_pages() changes. > But you've now found that 2.6.7 and later kernels allow your app to > work correctly without mlock, good. get_user_pages is certainly the > right tool to use for such pinning. (On the question of whether > mlock guarantees that user virtual addresses map to the same physical > addresses, I prefer Arjan's view that it does not; but accept that > there might prove to be difficulties in holding that position.) My understanding is that mlock() could in theory allow the page to be moved, but that currently nothing in the kernel would actually move it. However, that could change in the future to allow hot-swapping of RAM. > So, it works now, you've exonerated today's get_user_pages, and you've > identified at least one get_user_pages fix which went in at that time: > do we really need to chase this further? My driver needs to support all 2.4 and 2.6 kernel versions. My makefile scans the kernel source tree with 'grep' to identify various characterists, and I use #ifdefs to conditionally compile code depending on what features are present in the kernel. I can't use the kernel version number, because that's not reliable - distros will incorporate patches from future kernels without changing the version ID. So I need to take into account distro vendors that use an earlier kernel, like 2.6.5, and back-port the patch from 2.6.7. The distro vendor will keep the 2.6.5 version number, which is why I can't rely on it. I need to know exactly what the fix is, so that when I scan mm/rmap.c, I know what to look for. Currently, I look for this regex: try_to_unmap_one.*vm_area_struct which seems to work. However, now I think it's just a coincidence. > By the way, please don't be worried when soon the try_to_unmap_one > comment and code that you identified above disappear. When I'm > back in patch submission mode, I'll be sending Andrew a patch which > removes it, instead reworking can_share_swap_page to rely on the > page_mapcount instead of page_count, which avoids the ironical > behaviour my comment refers to, and allows an awkward page migration > case to proceed (once unpinned). Andrea and I now both prefer this > page_mapcount approach. Ugh, that means my regex is probably going to break. Not only that, but I don't understand what you're saying either. Trying to understand the VM is really hard. I guess in this specific case, it doesn't really matter, because calling mlock() when I should be calling get_user_pages() is not a bad thing. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-07 14:45 ` Timur Tabi @ 2005-05-07 16:30 ` Hugh Dickins 2005-05-11 20:12 ` William Jordan 0 siblings, 1 reply; 59+ messages in thread From: Hugh Dickins @ 2005-05-07 16:30 UTC (permalink / raw) To: Timur Tabi Cc: Libor Michalek, Andrew Morton, Andrea Arcangeli, linux-kernel, openib-general On Sat, 7 May 2005, Timur Tabi wrote: > > > Oh, well, maybe, but what is the real problem? > > Are you sure that copy-on-write doesn't come into it? > > No, but I do know that my test case doesn't call fork(), so it's reproducible > without involving COW. Of course, I'm sure someone's going to tell me now > that COW comes into effect even without fork(). If so, please explain. I'll try. COW comes into effect whenever you're sharing a page and then need to make private changes to it. Fork is one way of sharing (with ancestor and descendant processes). Using the empty zero page is another way of sharing (with all other processes and parts of your own address space with a readonly page full of zeroes). Using a file page from the page cache is another way of sharing. None of those is actually your case, but our test for whether a page is shared has been inadequate: oversimplifying, if page_count is more than 1 then we have to assume it is shared and do the copy-on-write (if the modifications are to be private). But there are various places where the page_count is temporarily raised (e.g. while paging out), which we cannot distinguish, so occasionally we'll copy on write even when it's not necessary, but we lack the information to tell us so. In particular, of course, get_user_pages raises page_count to pin the page: so making a page appear shared when it's not shared at all. > The short answer: under "extreme" memory pressure, the data inside a page > pinned by get_user_pages() is swapped out, moved, or deleted (I'm not sure > which). Some other data is placed into that physical location. > > By extreme memory pressure, I mean having the process allocate and touch as > much memory as possible. Something like this: > > num_bytes = get_amount_of_physical_ram(); > char *p = malloc(num_bytes); > for (i=0; i<num_bytes; i+=PAGE_SIZE) > p[i] = 0; > > The above over-simplified code fails on earlier 2.6 kernels (or earlier > versions of glibc that accompany most distros the use the earlier 2.6 > kernels). Either malloc() returns NULL, or the p[i]=0 part causes a segfault. > I haven't bothered to trace down why. But when it does work, the page pinned > by get_user_pages() changes. Which has to be a bug with get_user_pages, which has no other purpose than to pin the pages. I cannot criticize you for working around it to get your app working on lots of releases, but what _we_ have to do is fix get_user_pages - and it appears that Andrea did so a year ago. I'm surprised if it's as simple as you describe (you do say over- simplified, maybe the critical points have fallen out), since GUP users would have complained long ago if it wasn't doing the job in normal cases of memory pressure. Andrea's case does involve the process independently trying to touch a page it has pinned for I/O with get_user_pages. Or (and I've only just thought of this, suspect it might be exactly your case) not touch, but apply get_user_pages again to a page already so pinned (while memory pressure has caused try_to_unmap_one temporarily to detach it from the user address space - the aspect of the problem that Andrea's fix attacks). > My understanding is that mlock() could in theory allow the page to be moved, > but that currently nothing in the kernel would actually move it. However, > that could change in the future to allow hot-swapping of RAM. That's my understanding too, that nothing currently does so. Aside from hot-swapping RAM, there's also a need to be able to migrate pages around RAM, either to unfragment memory allowing higher-order allocations to succeed more often, or to get around extreme dmamem/normal-mem/highmem imbalances without dedicating huge reserves. Those would more often succeed if uninhibited by mlock. > So I need to take into account distro vendors that use an earlier kernel, like > 2.6.5, and back-port the patch from 2.6.7. The distro vendor will keep the > 2.6.5 version number, which is why I can't rely on it. > > I need to know exactly what the fix is, so that when I scan mm/rmap.c, I know > what to look for. Currently, I look for this regex: > > try_to_unmap_one.*vm_area_struct > > which seems to work. However, now I think it's just a coincidence. Perhaps any release based on 2.6.7 or above, or any release which mentions "get_user_pages" in its mm/rmap.c or mm/objrmap.c? > > By the way, please don't be worried when soon the try_to_unmap_one > > comment and code that you identified above disappear. When I'm > > back in patch submission mode, I'll be sending Andrew a patch which > > removes it, instead reworking can_share_swap_page to rely on the > > page_mapcount instead of page_count, which avoids the ironical > > behaviour my comment refers to, and allows an awkward page migration > > case to proceed (once unpinned). Andrea and I now both prefer this > > page_mapcount approach. > > Ugh, that means my regex is probably going to break. Not only that, but I > don't understand what you're saying either. Trying to understand the VM is > really hard. Sorry about that, but suiting your regex is low in our priorities for VM design! I was tempted to offer to keep a comment on get_user_pages in mm/rmap.c after the change, but that's really rather babyish: just assume 2.6.7 and upwards are fixed (or complain if you find not). Perhaps I'll manage a clearer explanation when I come to write the change description for the patch, we'll have to see. > I guess in this specific case, it doesn't really matter, because calling > mlock() when I should be calling get_user_pages() is not a bad thing. If you can afford to keep that amount of memory mlocked, and have to capability to do so, yes, it should do no harm. We were just upset to think that mlock was still needed to get around a get_user_pages bug which was fixed a year ago. Hugh ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-07 16:30 ` Hugh Dickins @ 2005-05-11 20:12 ` William Jordan 2005-05-11 20:42 ` Hugh Dickins 2005-05-11 22:49 ` Andrea Arcangeli 0 siblings, 2 replies; 59+ messages in thread From: William Jordan @ 2005-05-11 20:12 UTC (permalink / raw) To: Hugh Dickins Cc: Timur Tabi, Andrew Morton, Andrea Arcangeli, linux-kernel, openib-general On 5/7/05, Hugh Dickins <hugh@veritas.com> wrote: > > My understanding is that mlock() could in theory allow the page to be moved, > > but that currently nothing in the kernel would actually move it. However, > > that could change in the future to allow hot-swapping of RAM. > > That's my understanding too, that nothing currently does so. Aside from > hot-swapping RAM, there's also a need to be able to migrate pages around > RAM, either to unfragment memory allowing higher-order allocations to > succeed more often, or to get around extreme dmamem/normal-mem/highmem > imbalances without dedicating huge reserves. Those would more often > succeed if uninhibited by mlock. Hugh, If I am reading you correctly, you are saying that mlock currently prevents pages from migrating around to unfragment memory, but get_user_pages does not prevent this? If this is the case, this could very easily be the problem Timur was experiencing. He is using get_user_pages to lock pages long term (for the life of the process, beyond the bounds of a single system call). If it is possible for a page to be migrated in physical memory during extreme virtual memory pressure while the reference count is held with get_user_pages, that would cause the problem where the hardware is no longer mapped to the same page as the application. BTW: In earlier kernels, I experienced the same issues in our IB drivers when trying to pin pages using only get_user_pages. -- Bill Jordan InfiniCon Systems ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-11 20:12 ` William Jordan @ 2005-05-11 20:42 ` Hugh Dickins 2005-05-11 22:52 ` Andrea Arcangeli 2005-05-11 22:49 ` Andrea Arcangeli 1 sibling, 1 reply; 59+ messages in thread From: Hugh Dickins @ 2005-05-11 20:42 UTC (permalink / raw) To: William Jordan Cc: Timur Tabi, Andrew Morton, Andrea Arcangeli, linux-kernel, openib-general On Wed, 11 May 2005, William Jordan wrote: > On 5/7/05, Hugh Dickins <hugh@veritas.com> wrote: > > > My understanding is that mlock() could in theory allow the page to be moved, > > > but that currently nothing in the kernel would actually move it. However, > > > that could change in the future to allow hot-swapping of RAM. > > > > That's my understanding too, that nothing currently does so. Aside from > > hot-swapping RAM, there's also a need to be able to migrate pages around > > RAM, either to unfragment memory allowing higher-order allocations to > > succeed more often, or to get around extreme dmamem/normal-mem/highmem > > imbalances without dedicating huge reserves. Those would more often > > succeed if uninhibited by mlock. > > If I am reading you correctly, you are saying that mlock currently > prevents pages from migrating around to unfragment memory, but > get_user_pages does not prevent this? No, not what I meant at all. I'm saying that currently (aside from proposed patches) there is no such migration of pages; that we'd prefer to implement migration in such a way that mlock does not inhibit it (though there might prove to be strong arguments defeating that); and that get_user_pages _must_ prevent migration (and if there were already such migration, I'd be saying it _does_ prevent it). Hugh ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-11 20:42 ` Hugh Dickins @ 2005-05-11 22:52 ` Andrea Arcangeli 0 siblings, 0 replies; 59+ messages in thread From: Andrea Arcangeli @ 2005-05-11 22:52 UTC (permalink / raw) To: Hugh Dickins Cc: William Jordan, Timur Tabi, Andrew Morton, linux-kernel, openib-general On Wed, May 11, 2005 at 09:42:24PM +0100, Hugh Dickins wrote: > proposed patches) there is no such migration of pages; that we'd prefer > to implement migration in such a way that mlock does not inhibit it > (though there might prove to be strong arguments defeating that); > and that get_user_pages _must_ prevent migration (and if there > were already such migration, I'd be saying it _does_ prevent it). Indeed, mlock is a virtual pin and as such it won't be guaranteed to always prevent migration. While get_user_pages is a physical pin on the physical page so it has to prevent migration. I think for him the physical pin is better since I guess IB would break (at least unless you've some method to call to stop IB, adjust the IB dma tracking, and restart IB, that hotplug can call). For the short term using only get_user_pages sounds simpler IMHO. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-11 20:12 ` William Jordan 2005-05-11 20:42 ` Hugh Dickins @ 2005-05-11 22:49 ` Andrea Arcangeli 2005-05-11 22:53 ` Timur Tabi 1 sibling, 1 reply; 59+ messages in thread From: Andrea Arcangeli @ 2005-05-11 22:49 UTC (permalink / raw) To: William Jordan Cc: Hugh Dickins, Timur Tabi, Andrew Morton, linux-kernel, openib-general On Wed, May 11, 2005 at 04:12:41PM -0400, William Jordan wrote: > If I am reading you correctly, you are saying that mlock currently > prevents pages from migrating around to unfragment memory, but > get_user_pages does not prevent this? If this is the case, this could This is not the case. Infact get_user_pages is a stronger pin than mlock. But if you call it by hand and you plan to write to the page, you have to use the "write=1" flag, this is fundamental if you want to write to the physical page from userland while it's being tracked by IB dma. In short you should not use mlock and you should use only get_user_pages(write=1). If the problem appears again even after the last fix for the COW I did last year, than it means we've another yet another bug to fix. Using mlock for this is unnecessary. mlock is a "virtual" pin and it provides weaker guarantees than what you need. You need _physical_ pin and get_user_pages(write=1) is the only one that will give it to you. write=0 is ok too if you're never ever going to write to the page with the cpu from userland. In the old days there was the concept that get_user_pages wasn't a "pte-pin", but that was infact broken in the way COW was working with threads, but this is fixed now that is really a "pte-pin" again (like in 2.2 which never had the corruption cow bug!) even though the pte may temporarily be set to swapcache or null. In current 2.6 you're guaranteed that despite the pte may be temporarly be set to not-present, the next minor fault will bring into memory the very same physical page that was there before. At least unless you map the thing writeprotect (i.e. write=0) and you write to it from userland.. ;). ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-11 22:49 ` Andrea Arcangeli @ 2005-05-11 22:53 ` Timur Tabi 2005-05-11 23:05 ` Andrea Arcangeli 0 siblings, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-05-11 22:53 UTC (permalink / raw) To: Andrea Arcangeli Cc: William Jordan, Hugh Dickins, Andrew Morton, linux-kernel, openib-general Andrea Arcangeli wrote: > If the problem appears again even after the last fix for the COW I did > last year, than it means we've another yet another bug to fix. All of my memory pinning test cases pass when I use get_user_pages() with kernels 2.6.7 and later. -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-11 22:53 ` Timur Tabi @ 2005-05-11 23:05 ` Andrea Arcangeli 0 siblings, 0 replies; 59+ messages in thread From: Andrea Arcangeli @ 2005-05-11 23:05 UTC (permalink / raw) To: Timur Tabi Cc: William Jordan, Hugh Dickins, Andrew Morton, linux-kernel, openib-general On Wed, May 11, 2005 at 05:53:36PM -0500, Timur Tabi wrote: > Andrea Arcangeli wrote: > > >If the problem appears again even after the last fix for the COW I did > >last year, than it means we've another yet another bug to fix. > > All of my memory pinning test cases pass when I use get_user_pages() with > kernels 2.6.7 and later. Well then your problem was the cow bug, that was corrupting userland with O_DIRECT too... ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-04 18:27 ` Timur Tabi 2005-05-05 18:48 ` Timur Tabi @ 2005-05-05 23:34 ` Libor Michalek 1 sibling, 0 replies; 59+ messages in thread From: Libor Michalek @ 2005-05-05 23:34 UTC (permalink / raw) To: Timur Tabi; +Cc: Andrew Morton, linux-kernel, openib-general On Wed, May 04, 2005 at 01:27:54PM -0500, Timur Tabi wrote: > Libor Michalek wrote: > > > The program opens the charcter device file descriptor, pins the pages > > and waits for a signal, before checking the pages, which is sent to the > > process after running some other program which exercises the VM. On older > > kernels the check fails, on my 2.6.11 kernel the check succeeds. So > > mlock is not needed on top of get_user_pages() as it was before. > > When you say "older", what exactly do you mean? I have different test > that normally fails with just get_user_pages(), but it works with 2.6.9 > and above. I haven't been able to get any kernel earlier than 2.6.9 to > compile or boot properly, so I'm having a hard time narrowing down the > actual point when get_user_pages() started working. The older kernel I tried was one of the 2.4.21 RHEL 3 kernels. I hadn't spent much time investigating the issue since this was a new kernel, so it was a natural one for me to try. -Libor ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-12 0:13 ` Andrew Morton 2005-04-13 1:04 ` [openib-general] " Libor Michalek @ 2005-04-18 16:22 ` Timur Tabi 2005-04-18 16:43 ` Christoph Hellwig 1 sibling, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-04-18 16:22 UTC (permalink / raw) To: Andrew Morton; +Cc: Roland Dreier, hozer, linux-kernel, openib-general Andrew Morton wrote: > Roland Dreier <roland@topspin.com> wrote: > >> Troy> Do we even need the mlock in userspace then? >> >>Yes, because the kernel may go through and unmap pages from userspace >>while trying to swap. Since we have the page locked in the kernel, >>the physical page won't go anywhere, but userspace might end up with a >>different page mapped at the same virtual address. > > > That shouldn't happen. If get_user_pages() has elevated the refcount on a > page then the following can happen: > > - The VM may decide to add the page to swapcache (if it's not mmapped > from a file). > > - Once the page is backed by either swapcache of a (mmapped) file, the VM > may decide the unmap the application's pte's. A later minor fault by the > app will cause the same physical page to be remapped. That's not what we're seeing. We have hardware that does DMA over the network (much like the Infiniband stuff), and we have a testcase that fails if get_user_pages() is used, but not if mlock() is used. Consider two computers on a network, X and Y. Both have our hardware, which can transfer a page of memory from a given physical address on X to a physical address on Y. 1) Application on X allocates a block of memory, and passes the virtual address to the driver. 2) Driver on X calls get_user_pages() and then obtains a physical address for the memory. 3) Application and driver on Y do the same thing. 4) App X fills memory with some data D. 5) App X then allocates as much memory as it possibly can. It touches every page in this memory, and then frees the memory. This will force other pages to be swapped out, including the supposedly pinned memory. 6) App X then tells Driver X to transfer data D to computer Y. 7) App Y compares data D and finds that it doesn't match with it's supposed to. Conclusion: during step 5, the data in pinned memory is swapped out or something. I'm not sure where it goes. We can only demonstrate this problem using our hardware, because you need the ability to transfer memory without using the CPU. We were going to prepare a test case and ship same hardware to a few kernel developers to prove our point, but now that we're able to call mlock() in non-user processes, we decided it wasn't worth our time. Actually, I discovered that I can call cap_raise() and set the ulimit structure, which gives me the ability to call mlock() on any amount of memory from any process in 2.4 and 2.6 kernels, which we need to support. If I had thought of that earlier, I wouldn't have needed all those hacks to call sys_mlock() from the driver. -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-18 16:22 ` Timur Tabi @ 2005-04-18 16:43 ` Christoph Hellwig 2005-04-18 16:45 ` Timur Tabi 0 siblings, 1 reply; 59+ messages in thread From: Christoph Hellwig @ 2005-04-18 16:43 UTC (permalink / raw) To: Timur Tabi Cc: Andrew Morton, Roland Dreier, hozer, linux-kernel, openib-general On Mon, Apr 18, 2005 at 11:22:29AM -0500, Timur Tabi wrote: > That's not what we're seeing. We have hardware that does DMA over the > network (much like the Infiniband stuff), and we have a testcase that fails > if get_user_pages() is used, but not if mlock() is used. If you don't share your testcase it's unlikely to be fixed. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-18 16:43 ` Christoph Hellwig @ 2005-04-18 16:45 ` Timur Tabi 2005-04-24 2:44 ` Andrew Morton 0 siblings, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-04-18 16:45 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Roland Dreier, hozer, linux-kernel, openib-general Christoph Hellwig wrote: > On Mon, Apr 18, 2005 at 11:22:29AM -0500, Timur Tabi wrote: > >>That's not what we're seeing. We have hardware that does DMA over the >>network (much like the Infiniband stuff), and we have a testcase that fails >>if get_user_pages() is used, but not if mlock() is used. > > > If you don't share your testcase it's unlikely to be fixed. As I said, the testcase only works with our hardware, and it's also very large. It's one small test that's part of a huge test suite. It takes a couple hours just to install the damn thing. We want to produce a simpler test case that demonstrates the problem in an easy-to-understand manner, but we don't have time to do that now. -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-18 16:45 ` Timur Tabi @ 2005-04-24 2:44 ` Andrew Morton 2005-04-24 14:23 ` Timur Tabi 0 siblings, 1 reply; 59+ messages in thread From: Andrew Morton @ 2005-04-24 2:44 UTC (permalink / raw) To: Timur Tabi; +Cc: hch, roland, hozer, linux-kernel, openib-general Timur Tabi <timur.tabi@ammasso.com> wrote: > > Christoph Hellwig wrote: > > On Mon, Apr 18, 2005 at 11:22:29AM -0500, Timur Tabi wrote: > > > >>That's not what we're seeing. We have hardware that does DMA over the > >>network (much like the Infiniband stuff), and we have a testcase that fails > >>if get_user_pages() is used, but not if mlock() is used. > > > > > > If you don't share your testcase it's unlikely to be fixed. > > As I said, the testcase only works with our hardware, and it's also very large. It's one > small test that's part of a huge test suite. It takes a couple hours just to install the > damn thing. > > We want to produce a simpler test case that demonstrates the problem in an > easy-to-understand manner, but we don't have time to do that now. If your theory is correct then it should be able to demonstrate this problem without any special hardware at all: pin some user memory, then generate memory pressure then check the contents of those pinned pages. But if, for the DMA transfer, you're using the array of page*'s which were originally obtained from get_user_pages() then it's rather hard to see how the kernel could alter the page's contents. Then again, if mlock() fixes it then something's up. Very odd. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-24 2:44 ` Andrew Morton @ 2005-04-24 14:23 ` Timur Tabi 2005-04-25 13:15 ` Roland Dreier 0 siblings, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-04-24 14:23 UTC (permalink / raw) To: Andrew Morton; +Cc: hch, roland, hozer, linux-kernel, openib-general Andrew Morton wrote: > If your theory is correct then it should be able to demonstrate this > problem without any special hardware at all: pin some user memory, then > generate memory pressure then check the contents of those pinned pages. I tried that, but I couldn't get it to fail. But that was a while ago, and I've learned a few things since then, so I'll try again. > But if, for the DMA transfer, you're using the array of page*'s which were > originally obtained from get_user_pages() then it's rather hard to see how > the kernel could alter the page's contents. > > Then again, if mlock() fixes it then something's up. Very odd. With mlock(), we don't need to use get_user_pages() at all. Arjan tells me the only time an mlocked page can move is with hot (un)plug of memory, but that isn't supported on the systems that we support. We actually prefer mlock() over get_user_pages(), because if the process dies, the locks automatically go away too. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-24 14:23 ` Timur Tabi @ 2005-04-25 13:15 ` Roland Dreier 2005-04-25 20:54 ` Andrew Morton 0 siblings, 1 reply; 59+ messages in thread From: Roland Dreier @ 2005-04-25 13:15 UTC (permalink / raw) To: Timur Tabi; +Cc: Andrew Morton, hch, hozer, linux-kernel, openib-general Timur> With mlock(), we don't need to use get_user_pages() at all. Timur> Arjan tells me the only time an mlocked page can move is Timur> with hot (un)plug of memory, but that isn't supported on Timur> the systems that we support. We actually prefer mlock() Timur> over get_user_pages(), because if the process dies, the Timur> locks automatically go away too. There actually is another way pages can move, with both get_user_pages() and mlock(): copy-on-write after a fork(). If userspace does a fork(), then all PTEs are marked read-only, and if the original process touches the page after the fork(), a new page will be allocated and mapped at the original virtual address. This is actually a pretty big pain, because the only good solution seems to be for the kernel to mark these registered regions as VM_DONTCOPY. Right now this means that driver code ends up monkeying with vm_flags for user vmas. Does it seem reasonable to add a new system call to let userspace mark memory it doesn't want copied into forked processes? Something like long sys_mark_nocopy(unsigned long addr, size_t len, int mark) which would set VM_DONTCOPY if mark != 0, and clear it if mark == 0. A better name would be gratefully accepted... Then to register memory for RDMA, userspace would call sys_mark_nocopy() (with appropriate accounting to handle possibly overlapping regions) and the kernel would call get_user_pages(). The get_user_pages() is of course required because the kernel can't trust userspace to keep the pages locked. mlock() would no longer be necessary. We can trust userspace to call sys_mark_nocopy() as needed, because a process can only hurt itself and its children by misusing the sys_mark_nocopy() call. If this seems reasonable then I can code a patch. - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 13:15 ` Roland Dreier @ 2005-04-25 20:54 ` Andrew Morton 2005-04-25 21:12 ` Roland Dreier 0 siblings, 1 reply; 59+ messages in thread From: Andrew Morton @ 2005-04-25 20:54 UTC (permalink / raw) To: Roland Dreier; +Cc: timur.tabi, hch, hozer, linux-kernel, openib-general Roland Dreier <roland@topspin.com> wrote: > > Timur> With mlock(), we don't need to use get_user_pages() at all. > Timur> Arjan tells me the only time an mlocked page can move is > Timur> with hot (un)plug of memory, but that isn't supported on > Timur> the systems that we support. We actually prefer mlock() > Timur> over get_user_pages(), because if the process dies, the > Timur> locks automatically go away too. > > There actually is another way pages can move, with both > get_user_pages() and mlock(): copy-on-write after a fork(). If > userspace does a fork(), then all PTEs are marked read-only, and if > the original process touches the page after the fork(), a new page > will be allocated and mapped at the original virtual address. Do we care about that? A straightforward scenario under which this can happen is: a) app starts some read I/O in an asynchronous manner b) app forks c) child writes to one of the pages which is still under read I/O d) the read I/O completes e) the child is left with the old data plus the child's modification instead of the new data which is a very silly application which is giving itself unpredictable memory contents anyway. I assume there's a more sensible scenario? ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 20:54 ` Andrew Morton @ 2005-04-25 21:12 ` Roland Dreier 2005-04-25 22:14 ` Andrew Morton 0 siblings, 1 reply; 59+ messages in thread From: Roland Dreier @ 2005-04-25 21:12 UTC (permalink / raw) To: Andrew Morton; +Cc: timur.tabi, hch, hozer, linux-kernel, openib-general Andrew> Do we care about that? A straightforward scenario under Andrew> which this can happen is: Andrew> a) app starts some read I/O in an asynchronous manner Andrew> b) app forks Andrew> c) child writes to one of the pages which is still under read I/O Andrew> d) the read I/O completes Andrew> e) the child is left with the old data plus the child's modification instead Andrew> of the new data Andrew> which is a very silly application which is giving itself Andrew> unpredictable memory contents anyway. Andrew> I assume there's a more sensible scenario? You're right, that is a silly scenario ;) In fact if we mark vmas with VM_DONTCOPY, then the child just crashes with a seg fault. The type of thing I'm worried about is something like, for example: a) app registers memory region with RDMA hardware -- in other words, loads the device's translation table for future I/O b) app forks c) app writes to the registered memory region, and the kernel breaks the COW for the (now read-only) page by mapping a new page d) app starts an I/O that will do a DMA read from the region e) device reads using the wrong, old mapping This can be pretty insiduous because for example fork() + immediate exec() or just using system() still leaves the parent with PTEs marked read-only. If an application does overlapping memory registrations so get_user_pages() is called a lot, then as far as I can see can_share_swap_page() will always return 0 and the COW will happen even if the child process has thrown out its original vmas. Or if the counts are in the correct range, then there's a small window between fork() and exec() where the parent process can screw itself up, so most of the time the app works, until it doesn't. - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 21:12 ` Roland Dreier @ 2005-04-25 22:14 ` Andrew Morton 2005-04-25 22:21 ` Timur Tabi ` (2 more replies) 0 siblings, 3 replies; 59+ messages in thread From: Andrew Morton @ 2005-04-25 22:14 UTC (permalink / raw) To: Roland Dreier; +Cc: timur.tabi, hch, hozer, linux-kernel, openib-general Roland Dreier <roland@topspin.com> wrote: > > Andrew> Do we care about that? A straightforward scenario under > Andrew> which this can happen is: > > Andrew> a) app starts some read I/O in an asynchronous manner > Andrew> b) app forks > Andrew> c) child writes to one of the pages which is still under read I/O > Andrew> d) the read I/O completes > Andrew> e) the child is left with the old data plus the child's modification instead > Andrew> of the new data > > Andrew> which is a very silly application which is giving itself > Andrew> unpredictable memory contents anyway. > > Andrew> I assume there's a more sensible scenario? > > You're right, that is a silly scenario ;) In fact if we mark vmas > with VM_DONTCOPY, then the child just crashes with a seg fault. > > The type of thing I'm worried about is something like, for example: > > a) app registers memory region with RDMA hardware -- in other words, > loads the device's translation table for future I/O Whoa, hang on. The way we expect get_user_pages() to be used is that the kernel will use get_user_pages() once per application I/O request. Are you saying that RDMA clients will semi-permanently own pages which were pinned by get_user_pages()? That those pages will be used for multiple separate I/O operations? If so, then that's a significant design departure and it would be good to hear why it is necessary. > b) app forks > c) app writes to the registered memory region, and the kernel breaks > the COW for the (now read-only) page by mapping a new page > d) app starts an I/O that will do a DMA read from the region > e) device reads using the wrong, old mapping Sure. But such an app could be declared to be buggy... > This can be pretty insiduous because for example fork() + immediate > exec() or just using system() still leaves the parent with PTEs marked > read-only. If an application does overlapping memory registrations so > get_user_pages() is called a lot, then as far as I can see > can_share_swap_page() will always return 0 and the COW will happen > even if the child process has thrown out its original vmas. > > Or if the counts are in the correct range, then there's a small window > between fork() and exec() where the parent process can screw itself > up, so most of the time the app works, until it doesn't. > > - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 22:14 ` Andrew Morton @ 2005-04-25 22:21 ` Timur Tabi 2005-04-25 22:32 ` Andrew Morton 2005-04-25 22:23 ` Timur Tabi 2005-04-26 0:02 ` Roland Dreier 2 siblings, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-04-25 22:21 UTC (permalink / raw) To: Andrew Morton; +Cc: Roland Dreier, hch, hozer, linux-kernel, openib-general Andrew Morton wrote: > The way we expect get_user_pages() to be used is that the kernel will use > get_user_pages() once per application I/O request. > > Are you saying that RDMA clients will semi-permanently own pages which were > pinned by get_user_pages()? That those pages will be used for multiple > separate I/O operations? Yes, absolutely! The memory buffer is allocated by the process (usually just via malloc) and registed/pinned by the driver. It then stays pinned for the life of the process (typically). > If so, then that's a significant design departure and it would be good to > hear why it is necessary. That's just how RMDA works. Once the memory is pinned, if the app wants to send data to another node, it does two things: 1) Puts the data into its buffer 2) Sends a "work request" to the driver with (among other things) the offset and length of the data. This is a time-critical operation. It must occurs as fast as possible, which means the memory must have already been pinned. -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 22:21 ` Timur Tabi @ 2005-04-25 22:32 ` Andrew Morton 2005-04-25 23:58 ` Roland Dreier 0 siblings, 1 reply; 59+ messages in thread From: Andrew Morton @ 2005-04-25 22:32 UTC (permalink / raw) To: Timur Tabi; +Cc: roland, hch, hozer, linux-kernel, openib-general Timur Tabi <timur.tabi@ammasso.com> wrote: > > Andrew Morton wrote: > > > The way we expect get_user_pages() to be used is that the kernel will use > > get_user_pages() once per application I/O request. > > > > Are you saying that RDMA clients will semi-permanently own pages which were > > pinned by get_user_pages()? That those pages will be used for multiple > > separate I/O operations? > > Yes, absolutely! > > The memory buffer is allocated by the process (usually just via malloc) and > registed/pinned by the driver. It then stays pinned for the life of the process (typically). ug. What stops the memory from leaking if the process exits? I hope this is a privileged operation? > > If so, then that's a significant design departure and it would be good to > > hear why it is necessary. > > That's just how RMDA works. Once the memory is pinned, if the app wants to send data to > another node, it does two things: > > 1) Puts the data into its buffer > 2) Sends a "work request" to the driver with (among other things) the offset and length of > the data. > > This is a time-critical operation. It must occurs as fast as possible, which means the > memory must have already been pinned. It would be better to obtain this memory via a mmap() of some special device node, so we can perform appropriate permission checking and clean everything up on unclean application exit. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 22:32 ` Andrew Morton @ 2005-04-25 23:58 ` Roland Dreier 2005-04-26 0:11 ` Andrew Morton 2005-04-26 2:03 ` IWAMOTO Toshihiro 0 siblings, 2 replies; 59+ messages in thread From: Roland Dreier @ 2005-04-25 23:58 UTC (permalink / raw) To: Andrew Morton; +Cc: Timur Tabi, hch, hozer, linux-kernel, openib-general Andrew> ug. What stops the memory from leaking if the process Andrew> exits? Andrew> I hope this is a privileged operation? I don't think it has to be privileged. In my implementation, the driver keeps a per-process list of registered memory regions and unpins/cleans up on process exit. Andrew> It would be better to obtain this memory via a mmap() of Andrew> some special device node, so we can perform appropriate Andrew> permission checking and clean everything up on unclean Andrew> application exit. This seems to interact poorly with how applications want to use RDMA, ie typically through a library interface such as MPI. People doing HPC don't want to recode their apps to use a new allocator, they just want to link to a new MPI library and have the app go fast. - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 23:58 ` Roland Dreier @ 2005-04-26 0:11 ` Andrew Morton 2005-04-26 0:23 ` Roland Dreier 2005-04-26 2:03 ` IWAMOTO Toshihiro 1 sibling, 1 reply; 59+ messages in thread From: Andrew Morton @ 2005-04-26 0:11 UTC (permalink / raw) To: Roland Dreier; +Cc: timur.tabi, hch, hozer, linux-kernel, openib-general Roland Dreier <roland@topspin.com> wrote: > > Andrew> ug. What stops the memory from leaking if the process > Andrew> exits? > > Andrew> I hope this is a privileged operation? > > I don't think it has to be privileged. In my implementation, the > driver keeps a per-process list of registered memory regions and > unpins/cleans up on process exit. How does the driver detect process exit? > Andrew> It would be better to obtain this memory via a mmap() of > Andrew> some special device node, so we can perform appropriate > Andrew> permission checking and clean everything up on unclean > Andrew> application exit. > > This seems to interact poorly with how applications want to use RDMA, > ie typically through a library interface such as MPI. People doing > HPC don't want to recode their apps to use a new allocator, they just > want to link to a new MPI library and have the app go fast. Fair enough. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 0:11 ` Andrew Morton @ 2005-04-26 0:23 ` Roland Dreier 2005-04-26 0:37 ` Andrew Morton 0 siblings, 1 reply; 59+ messages in thread From: Roland Dreier @ 2005-04-26 0:23 UTC (permalink / raw) To: Andrew Morton; +Cc: timur.tabi, hch, hozer, linux-kernel, openib-general Andrew> How does the driver detect process exit? I already answered earlier but just to be clear: registration goes through a character device, and all regions are cleaned up in the ->release() of that device. I don't currently have any code accounting against RLIMIT_MEMLOCK or testing CAP_FOO, but I have no problem adding whatever is thought appropriate. Userspace also has control over the permissions and owner/group of the /dev node. - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 0:23 ` Roland Dreier @ 2005-04-26 0:37 ` Andrew Morton 2005-04-26 15:31 ` Roland Dreier 0 siblings, 1 reply; 59+ messages in thread From: Andrew Morton @ 2005-04-26 0:37 UTC (permalink / raw) To: Roland Dreier; +Cc: timur.tabi, hch, hozer, linux-kernel, openib-general Roland Dreier <roland@topspin.com> wrote: > > Andrew> How does the driver detect process exit? > > I already answered earlier but just to be clear: registration goes > through a character device, and all regions are cleaned up in the > ->release() of that device. yup. > I don't currently have any code accounting against RLIMIT_MEMLOCK or > testing CAP_FOO, but I have no problem adding whatever is thought > appropriate. Userspace also has control over the permissions and > owner/group of the /dev node. I guess device node permissions won't be appropriate here, if only because it sounds like everyone will go and set them to 0666. RLIMIT_MEMLOCK sounds like the appropriate mechanism. We cannot rely upon userspace running mlock(), so perhaps it is appropriate to run sys_mlock() in-kernel because that gives us the appropriate RLIMIT_MEMLOCK checking. However an hostile app can just go and run munlock() and then allocate some more pinned-by-get_user_pages() memory. umm, how about we - force the special pages into a separate vma - run get_user_pages() against it all - use RLIMIT_MEMLOCK accounting to check whether the user is allowed to do this thing - undo the RMLIMIT_MEMLOCK accounting in ->release This will all interact with user-initiated mlock/munlock in messy ways. Maybe a new kernel-internal vma->vm_flag which works like VM_LOCKED but is unaffected by mlock/munlock activity is needed. A bit of generalisation in do_mlock() should suit? ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 0:37 ` Andrew Morton @ 2005-04-26 15:31 ` Roland Dreier 2005-04-26 15:42 ` [openib-general] " Libor Michalek 0 siblings, 1 reply; 59+ messages in thread From: Roland Dreier @ 2005-04-26 15:31 UTC (permalink / raw) To: Andrew Morton; +Cc: timur.tabi, hch, hozer, linux-kernel, openib-general Andrew> umm, how about we Andrew> - force the special pages into a separate vma Andrew> - run get_user_pages() against it all Andrew> - use RLIMIT_MEMLOCK accounting to check whether the user Andrew> is allowed to do this thing Andrew> - undo the RMLIMIT_MEMLOCK accounting in ->release Andrew> This will all interact with user-initiated mlock/munlock Andrew> in messy ways. Maybe a new kernel-internal vma->vm_flag Andrew> which works like VM_LOCKED but is unaffected by Andrew> mlock/munlock activity is needed. Andrew> A bit of generalisation in do_mlock() should suit? Yes, it seems that modifying do_mlock() to something like int do_mlock(unsigned long start, size_t len, unsigned int set, unsigned int clear) and then exporting a function along the lines of int do_mem_pin(unsigned long start, size_t len, int on) that sets/clears (VM_LOCKED_KERNEL | VM_DONTCOPY) according to the on flag. Seem reasonable? If so I can code this up. - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 15:31 ` Roland Dreier @ 2005-04-26 15:42 ` Libor Michalek 2005-04-26 15:49 ` Roland Dreier 0 siblings, 1 reply; 59+ messages in thread From: Libor Michalek @ 2005-04-26 15:42 UTC (permalink / raw) To: Roland Dreier Cc: Andrew Morton, hch, linux-kernel, openib-general, timur.tabi On Tue, Apr 26, 2005 at 08:31:32AM -0700, Roland Dreier wrote: > Andrew> umm, how about we > > Andrew> - force the special pages into a separate vma > > Andrew> - run get_user_pages() against it all > > Andrew> - use RLIMIT_MEMLOCK accounting to check whether the user > Andrew> is allowed to do this thing > > Andrew> - undo the RMLIMIT_MEMLOCK accounting in ->release > > Andrew> This will all interact with user-initiated mlock/munlock > Andrew> in messy ways. Maybe a new kernel-internal vma->vm_flag > Andrew> which works like VM_LOCKED but is unaffected by > Andrew> mlock/munlock activity is needed. > > Andrew> A bit of generalisation in do_mlock() should suit? > > Yes, it seems that modifying do_mlock() to something like > > int do_mlock(unsigned long start, size_t len, > unsigned int set, unsigned int clear) > > and then exporting a function along the lines of > > int do_mem_pin(unsigned long start, size_t len, int on) > > that sets/clears (VM_LOCKED_KERNEL | VM_DONTCOPY) according to the on > flag. Do you mean that the set/clear parameters to do_mlock() are the actual flags which are set/cleared by the caller? Also, the issue remains that the flags are not reference counted which is a problem if you are dealing with overlapping memory region, or even if one region ends and another begins on the same page. Since the desire is to be able to pin any memory that a user can malloc this is a likely scenario. -Libor ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 15:42 ` [openib-general] " Libor Michalek @ 2005-04-26 15:49 ` Roland Dreier 2005-04-26 19:28 ` Andrew Morton 0 siblings, 1 reply; 59+ messages in thread From: Roland Dreier @ 2005-04-26 15:49 UTC (permalink / raw) To: Libor Michalek Cc: Andrew Morton, hch, linux-kernel, openib-general, timur.tabi Libor> Do you mean that the set/clear parameters to do_mlock() Libor> are the actual flags which are set/cleared by the caller? Libor> Also, the issue remains that the flags are not reference Libor> counted which is a problem if you are dealing with Libor> overlapping memory region, or even if one region ends and Libor> another begins on the same page. Since the desire is to be Libor> able to pin any memory that a user can malloc this is a Libor> likely scenario. Good point... we need to figure out how to handle: a) app registers 0x0000 through 0x17ff b) app registers 0x1800 through 0x2fff c) app unregisters 0x0000 through 0x17ff d) the page at 0x1000 must stay pinned hmm... - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 15:49 ` Roland Dreier @ 2005-04-26 19:28 ` Andrew Morton 2005-04-26 20:14 ` Roland Dreier 2005-04-27 3:15 ` Caitlin Bestler 0 siblings, 2 replies; 59+ messages in thread From: Andrew Morton @ 2005-04-26 19:28 UTC (permalink / raw) To: Roland Dreier; +Cc: libor, hch, linux-kernel, openib-general, timur.tabi Roland Dreier <roland@topspin.com> wrote: > > Libor> Do you mean that the set/clear parameters to do_mlock() > Libor> are the actual flags which are set/cleared by the caller? > Libor> Also, the issue remains that the flags are not reference > Libor> counted which is a problem if you are dealing with > Libor> overlapping memory region, or even if one region ends and > Libor> another begins on the same page. Since the desire is to be > Libor> able to pin any memory that a user can malloc this is a > Libor> likely scenario. > > Good point... we need to figure out how to handle: > > a) app registers 0x0000 through 0x17ff > b) app registers 0x1800 through 0x2fff > c) app unregisters 0x0000 through 0x17ff > d) the page at 0x1000 must stay pinned The userspace library should be able to track the tree and the overlaps, etc. Things might become interesting when the memory is MAP_SHARED pagecache and multiple independent processes are involved, although I guess that'd work OK. But afaict the problem wherein part of a page needs VM_DONTCOPY and the other part does not cannot be solved. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 19:28 ` Andrew Morton @ 2005-04-26 20:14 ` Roland Dreier 2005-04-26 20:18 ` Timur Tabi 2005-04-26 20:32 ` Andrew Morton 2005-04-27 3:15 ` Caitlin Bestler 1 sibling, 2 replies; 59+ messages in thread From: Roland Dreier @ 2005-04-26 20:14 UTC (permalink / raw) To: Andrew Morton; +Cc: libor, hch, linux-kernel, openib-general, timur.tabi Roland> a) app registers 0x0000 through 0x17ff Roland> b) app registers 0x1800 through 0x2fff Roland> c) app unregisters 0x0000 through 0x17ff Roland> d) the page at 0x1000 must stay pinned Andrew> The userspace library should be able to track the tree and Andrew> the overlaps, etc. Things might become interesting when Andrew> the memory is MAP_SHARED pagecache and multiple Andrew> independent processes are involved, although I guess Andrew> that'd work OK. I used to think I knew how to handle this, but in your scheme where the kernel is doing accounting for pinned memory by marking vmas with VM_KERNEL_LOCKED, at step c), I don't see why the kernel won't unlock vmas covering 0x0000 through 0x1fff and credit 8K back to the process's pinning count. Sorry to be so dense but can you spell out what you think should happen at steps a), b) and c) above? Andrew> But afaict the problem wherein part of a page needs Andrew> VM_DONTCOPY and the other part does not cannot be solved. Yes, I agree. If an app wants to register half a page and pass the other half to a child process, I think the only answer is "don't do that then." - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 20:14 ` Roland Dreier @ 2005-04-26 20:18 ` Timur Tabi 2005-04-26 20:37 ` Andrew Morton 2005-04-26 20:32 ` Andrew Morton 1 sibling, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-04-26 20:18 UTC (permalink / raw) To: Roland Dreier; +Cc: Andrew Morton, libor, hch, linux-kernel, openib-general Roland Dreier wrote: > Yes, I agree. If an app wants to register half a page and pass the > other half to a child process, I think the only answer is "don't do > that then." How can the app know that, though? It would have to allocate I/O buffers with knowledge of page boundaries. Today, the apps just malloc() a bunch of memory and pay no attention to whether the beginning or the end of the buffer shares a page with some other, unrelated object. We may as well tell the app that it needs to page-align all I/O buffers. My point is that we can't just simply say, "Don't do that". Some entity (the kernel, libraries, whatever) should be able to tell the app that its usage of memory is going to break in some unpredictable way. -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 20:18 ` Timur Tabi @ 2005-04-26 20:37 ` Andrew Morton 2005-04-29 14:26 ` Bill Jordan 0 siblings, 1 reply; 59+ messages in thread From: Andrew Morton @ 2005-04-26 20:37 UTC (permalink / raw) To: Timur Tabi; +Cc: roland, libor, hch, linux-kernel, openib-general Timur Tabi <timur.tabi@ammasso.com> wrote: > > Roland Dreier wrote: > > > Yes, I agree. If an app wants to register half a page and pass the > > other half to a child process, I think the only answer is "don't do > > that then." > > How can the app know that, though? It would have to allocate I/O buffers with knowledge > of page boundaries. Today, the apps just malloc() a bunch of memory and pay no attention > to whether the beginning or the end of the buffer shares a page with some other, unrelated > object. We may as well tell the app that it needs to page-align all I/O buffers. > > My point is that we can't just simply say, "Don't do that". Some entity (the kernel, > libraries, whatever) should be able to tell the app that its usage of memory is going to > break in some unpredictable way. Our point is that contemporary microprocessors cannot electrically do what you want them to do! Now, conceeeeeeiveably the kernel could keep track of the state of the pages down to the byte level, and could keep track of all COWed pages and could look at faulting addresses at the byte level and could copy sub-page ranges by hand from one process's address space into another process's after I/O completion. I don't think we want to do that. Methinks your specification is busted. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 20:37 ` Andrew Morton @ 2005-04-29 14:26 ` Bill Jordan 2005-04-29 15:56 ` Caitlin Bestler 0 siblings, 1 reply; 59+ messages in thread From: Bill Jordan @ 2005-04-29 14:26 UTC (permalink / raw) To: Andrew Morton; +Cc: Timur Tabi, hch, linux-kernel, openib-general On 4/26/05, Andrew Morton <akpm@osdl.org> wrote: > Our point is that contemporary microprocessors cannot electrically do what > you want them to do! > > Now, conceeeeeeiveably the kernel could keep track of the state of the > pages down to the byte level, and could keep track of all COWed pages and > could look at faulting addresses at the byte level and could copy sub-page > ranges by hand from one process's address space into another process's > after I/O completion. I don't think we want to do that. > > Methinks your specification is busted. I agree in principal. However, I expect this issue will come up with more and more new specifications, and if it isn't addressed once in the linux kernel, it will be kludged and broken many times in many drivers. I believe we need an kernel level interface that will pin user pages, and lock the user vma in a single step. The interface should be used by drivers when the hardware mappings are done. If the process is split into a user operation to lock the memory, and a driver operation to map the hardware, there will always be opportunity for abuse. Reference counting needs to be done by this interface to allow different hardware to interoperate. The interface can't overload the VM_LOCKED flag, or rely on any other attributes that the user can tinker with via any other interface. And as much as I hate to admit it, I think on a fork, we will need to copy parts of pages at the beginning or end of user I/O buffers. -- Bill Jordan InfiniCon Systems ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-29 14:26 ` Bill Jordan @ 2005-04-29 15:56 ` Caitlin Bestler 2005-04-29 17:04 ` Libor Michalek 0 siblings, 1 reply; 59+ messages in thread From: Caitlin Bestler @ 2005-04-29 15:56 UTC (permalink / raw) To: Bill Jordan; +Cc: Andrew Morton, hch, linux-kernel, openib-general, Timur Tabi On 4/29/05, Bill Jordan <woodennickel@gmail.com> wrote: > On 4/26/05, Andrew Morton <akpm@osdl.org> wrote: > > > Our point is that contemporary microprocessors cannot electrically do what > > you want them to do! > > > > Now, conceeeeeeiveably the kernel could keep track of the state of the > > pages down to the byte level, and could keep track of all COWed pages and > > could look at faulting addresses at the byte level and could copy sub-page > > ranges by hand from one process's address space into another process's > > after I/O completion. I don't think we want to do that. > > > > Methinks your specification is busted. > > I agree in principal. However, I expect this issue will come up with > more and more new specifications, and if it isn't addressed once in > the linux kernel, it will be kludged and broken many times in many > drivers. > > I believe we need an kernel level interface that will pin user pages, > and lock the user vma in a single step. The interface should be used > by drivers when the hardware mappings are done. If the process is > split into a user operation to lock the memory, and a driver operation > to map the hardware, there will always be opportunity for abuse. > > Reference counting needs to be done by this interface to allow > different hardware to interoperate. > > The interface can't overload the VM_LOCKED flag, or rely on any other > attributes that the user can tinker with via any other interface. > > And as much as I hate to admit it, I think on a fork, we will need to > copy parts of pages at the beginning or end of user I/O buffers. > I agree with all but the last part, in my opinion there is no need to deal with fork issues as long as solutions do not result in failures. There is *no* basis for a child process to expect that it will inherit RDMA resources. A child process that uses such resources will get undefined results, nothing further needs to be stated, and no heroic efforts are required to avoid them. What is definitely needed is kernel counting of locks on user pages. Finer granularity is not expected, it is the RDMA hardware that works at finer granularity. All it needs is to know what bus address a given virtual page maps to -- and it needs to know that said mapping will not change without advance notice. Further, any revocation of an existing mapping (to deal with hot page swapping or whatever) cannot expect the RDMA hardware to respond any faster than it would to invalidating a memory region. The RDMA hardware has an inherent need to cache translations. That is why it cannot guarantee that it will cease updating a memory region the nanosecond that a request is made to invalidate an STag. Instead it is allowed to block on such a request and only guarantees to have ceased access when the invalidate request completes. The same need for a delay exists for any interface that moves memory around, or requests to reclaim memory from the application. This also applies on process death. The hardware cannot stop on a dime. The best it can do is stop promptly, and given an unambiguous indication to the OS as to when it has stopped. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-29 15:56 ` Caitlin Bestler @ 2005-04-29 17:04 ` Libor Michalek 2005-04-30 0:31 ` Caitlin Bestler 0 siblings, 1 reply; 59+ messages in thread From: Libor Michalek @ 2005-04-29 17:04 UTC (permalink / raw) To: Caitlin Bestler Cc: Bill Jordan, Andrew Morton, hch, linux-kernel, openib-general, Timur Tabi On Fri, Apr 29, 2005 at 08:56:20AM -0700, Caitlin Bestler wrote: > On 4/29/05, Bill Jordan <woodennickel@gmail.com> wrote: > > On 4/26/05, Andrew Morton <akpm@osdl.org> wrote: > > > > > Our point is that contemporary microprocessors cannot electrically > > > do what you want them to do! > > > > > > Now, conceeeeeeiveably the kernel could keep track of the state of the > > > pages down to the byte level, and could keep track of all COWed pages and > > > could look at faulting addresses at the byte level and could copy sub-page > > > ranges by hand from one process's address space into another process's > > > after I/O completion. I don't think we want to do that. > > > > > > Methinks your specification is busted. > > > > I agree in principal. However, I expect this issue will come up with > > more and more new specifications, and if it isn't addressed once in > > the linux kernel, it will be kludged and broken many times in many > > drivers. > > > > I believe we need an kernel level interface that will pin user pages, > > and lock the user vma in a single step. The interface should be used > > by drivers when the hardware mappings are done. If the process is > > split into a user operation to lock the memory, and a driver operation > > to map the hardware, there will always be opportunity for abuse. > > > > Reference counting needs to be done by this interface to allow > > different hardware to interoperate. > > > > The interface can't overload the VM_LOCKED flag, or rely on any other > > attributes that the user can tinker with via any other interface. > > > > And as much as I hate to admit it, I think on a fork, we will need to > > copy parts of pages at the beginning or end of user I/O buffers. > > > > I agree with all but the last part, in my opinion there is no need to deal > with fork issues as long as solutions do not result in failures. There is > *no* basis for a child process to expect that it will inherit RDMA resources. > A child process that uses such resources will get undefined results, nothing > further needs to be stated, and no heroic efforts are required to avoid them. However, you have a potential problem with registered buffers that do not begin or end on a page boundary, which is common with malloc. If the buffer resides on a portion of a page, and you mark the vm which contains that entire page VM_DONTCOPY, to ensure that the parent has access to the exact physical page after the fork, the child will not be able to access anything on that entire page. So if the child expects to access data on the same page that happens to contain the registered buffer it will get a segment violation. The four situations we've discussed are: 1) Physical page does not get used for anything else. 2) Processes virtual to physical mapping remains fixed. 3) Same virtual to physical mapping after forking a child. 4) Forked child has access to all non-registered memory of the parent. The first two are now taken care of with get_user_pages, (we use to use VM_LOCKED for the second case) third case is handled by setting the vm to VM_DONTCOPY, and on the fourth case we've always punted, but the real answer is to break partial pages into seperate vms and mark them ALWAYS_COPY. -Libor ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-29 17:04 ` Libor Michalek @ 2005-04-30 0:31 ` Caitlin Bestler 2005-05-03 18:43 ` Andy Isaacson 0 siblings, 1 reply; 59+ messages in thread From: Caitlin Bestler @ 2005-04-30 0:31 UTC (permalink / raw) To: Libor Michalek Cc: Bill Jordan, Andrew Morton, hch, linux-kernel, openib-general, Timur Tabi On 4/29/05, Libor Michalek <libor@topspin.com> wrote: > > However, you have a potential problem with registered buffers that > do not begin or end on a page boundary, which is common with malloc. > If the buffer resides on a portion of a page, and you mark the vm > which contains that entire page VM_DONTCOPY, to ensure that the parent > has access to the exact physical page after the fork, the child will > not be able to access anything on that entire page. So if the child > expects to access data on the same page that happens to contain the > registered buffer it will get a segment violation. > > The four situations we've discussed are: > > 1) Physical page does not get used for anything else. > 2) Processes virtual to physical mapping remains fixed. > 3) Same virtual to physical mapping after forking a child. > 4) Forked child has access to all non-registered memory of > the parent. > > The first two are now taken care of with get_user_pages, (we use to > use VM_LOCKED for the second case) third case is handled by setting > the vm to VM_DONTCOPY, and on the fourth case we've always punted, > but the real answer is to break partial pages into seperate vms and > mark them ALWAYS_COPY. > > -Libor > > Attempting to provide *any* support for applications that fork children after doing RDMA registrations is a ratshole best avoided. The general rule that application developers should follow is to do RDMA *only* in the child processes. Keep in mind that it is not only the memory regions that must be dealt with, but control data invisible to the user (the QP context, etc.). This data frequently is interlinked between kernel residente and user resident data (such as a QP context has the PD ID somewhere on-chip or in kernel, which the Send Queue ring needs to be in user memory). Having two different user processes that both think they have the user half to this type of split data structure is just asking for trouble, even if you manage to get the copy on write bit timing problems all solved. All of this can be avoided by a simple rule: don't fork after opening an RDMA device. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-30 0:31 ` Caitlin Bestler @ 2005-05-03 18:43 ` Andy Isaacson 2005-05-03 19:04 ` Caitlin Bestler 2005-05-04 18:22 ` William Jordan 0 siblings, 2 replies; 59+ messages in thread From: Andy Isaacson @ 2005-05-03 18:43 UTC (permalink / raw) To: Caitlin Bestler Cc: Libor Michalek, Bill Jordan, Andrew Morton, hch, linux-kernel, openib-general, Timur Tabi On Fri, Apr 29, 2005 at 05:31:44PM -0700, Caitlin Bestler wrote: > Attempting to provide *any* support for applications that fork children > after doing RDMA registrations is a ratshole best avoided. The general > rule that application developers should follow is to do RDMA *only* > in the child processes. I think it's unreasonable to *prohibit* fork-after-registration; for one thing, there's lots of code that forks under the covers. Setuid helpers like getpty just assume that they're going to be able to fork. Even stuff like get*by*(3) can potentially fork. And with site-configured stuff like PAM, you end up with things that work on the developer's system but break in deployment. I think it's exceedingly reasonable to say "RDMA doesn't work in children". But the child should get a sane memory image: at least zeros in fully-registered pages, and preferably copies of partially-registered pages. Differentiating between fully-registered and partially-registered pages avoids (I think) the pathological case of having to copy a GB of data just to system("/bin/ls > /tmp/tmpfile"). You can still go pathological if you've partially-registered gigabytes of address space (for example a linked list where each node is allocated with malloc and then registered) but that's a case of "Well, don't do that then". Rather than replacing the fully-registered pages with pages of zeros, you could simply unmap them. A consistent statement would be After fork(2), any regions which were registered are UNDEFINED. Region boundaries are byte-accurate; a registration can cover just part of a page, in which case the non-registered part of the page has normal fork COW semantics. Probably the most sane solution is to simply unmap the fully-registered pages at fork time, and copy any partially-registered pages. But the statement above does not require this. > Keep in mind that it is not only the memory regions that must be dealt > with, but control data invisible to the user (the QP context, etc.). This > data frequently is interlinked between kernel residente and user resident > data (such as a QP context has the PD ID somewhere on-chip or in > kernel, which the Send Queue ring needs to be in user memory). Having > two different user processes that both think they have the user half to > this type of split data structure is just asking for trouble, even if you > manage to get the copy on write bit timing problems all solved. Obviously, calling *any* RDMA-userland-stuff in the child is completely undefined [1]. One place where I can see a potential problem is in atexit()-type handlers registered by the RDMA library. Since those aren't performance-critical they can and should do sanity checks with getpid() and/or checking with the kernel driver. [1] You might want to allow the child to start a completely new RDMA context, but I don't see that as necessary. -andy ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-03 18:43 ` Andy Isaacson @ 2005-05-03 19:04 ` Caitlin Bestler 2005-05-04 18:22 ` William Jordan 1 sibling, 0 replies; 59+ messages in thread From: Caitlin Bestler @ 2005-05-03 19:04 UTC (permalink / raw) To: Andy Isaacson Cc: Libor Michalek, Bill Jordan, Andrew Morton, hch, linux-kernel, openib-general, Timur Tabi On 5/3/05, Andy Isaacson <adi@hexapodia.org> wrote: > > A consistent statement would be > > After fork(2), any regions which were registered are UNDEFINED. > Region boundaries are byte-accurate; a registration can cover just > part of a page, in which case the non-registered part of the page > has normal fork COW semantics. > That is a reasonable approach. > > Obviously, calling *any* RDMA-userland-stuff in the child is completely > undefined [1]. One place where I can see a potential problem is in > atexit()-type handlers registered by the RDMA library. Since those > aren't performance-critical they can and should do sanity checks with > getpid() and/or checking with the kernel driver. > That is also reasonable. None of the RDMA libraries I have worked on bothered to use an atexit()-type hook because the user was theoretically *required* to close the rnic, and driver code was already reuqired to clean up in case of a total process failure. Adding an intermediate safety-net for applications that exited cleanly but forget to close just didn't seem worthwhile. If the application wants the cleanup performed optimally then it can close the rnic, otherwise it can't complain about forcing the RNIC vendor to clean up in the driver code. > [1] You might want to allow the child to start a completely new RDMA > context, but I don't see that as necessary. > That should be allowed. It is actually more normal to use the parent as a dispatcher and to actually manage the connection in a child process. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-03 18:43 ` Andy Isaacson 2005-05-03 19:04 ` Caitlin Bestler @ 2005-05-04 18:22 ` William Jordan 2005-05-05 1:27 ` Rik van Riel 1 sibling, 1 reply; 59+ messages in thread From: William Jordan @ 2005-05-04 18:22 UTC (permalink / raw) To: Andy Isaacson Cc: Caitlin Bestler, Andrew Morton, linux-kernel, openib-general, hch, Timur Tabi On 5/3/05, Andy Isaacson <adi@hexapodia.org> wrote: > Rather than replacing the fully-registered pages with pages of zeros, > you could simply unmap them. I don't like this option. It is nearly free to map all of the pages to the zero-page. You never have to allocate a page if the user never writes to it. Buf if you unmap the page, there could be issues. The memory region could be on the stack, or malloc'ed. In these cases, the child should be able to return from the function, or free the memory without setting a timebomb. -- Bill Jordan InfiniCon Systems ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-04 18:22 ` William Jordan @ 2005-05-05 1:27 ` Rik van Riel 2005-05-05 1:57 ` Andy Isaacson 0 siblings, 1 reply; 59+ messages in thread From: Rik van Riel @ 2005-05-05 1:27 UTC (permalink / raw) To: William Jordan Cc: Andy Isaacson, Caitlin Bestler, Andrew Morton, linux-kernel, openib-general, hch, Timur Tabi On Wed, 4 May 2005, William Jordan wrote: > On 5/3/05, Andy Isaacson <adi@hexapodia.org> wrote: > > Rather than replacing the fully-registered pages with pages of zeros, > > you could simply unmap them. > > I don't like this option. It is nearly free to map all of the pages to > the zero-page. You never have to allocate a page if the user never > writes to it. Unmapping should work fine, as long as the VMA flags are set appropriately. The page fault handler can take care of the rest... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-05-05 1:27 ` Rik van Riel @ 2005-05-05 1:57 ` Andy Isaacson 0 siblings, 0 replies; 59+ messages in thread From: Andy Isaacson @ 2005-05-05 1:57 UTC (permalink / raw) To: Rik van Riel Cc: William Jordan, Caitlin Bestler, Andrew Morton, linux-kernel, openib-general, hch, Timur Tabi On Wed, May 04, 2005 at 09:27:21PM -0400, Rik van Riel wrote: > On Wed, 4 May 2005, William Jordan wrote: > > On 5/3/05, Andy Isaacson <adi@hexapodia.org> wrote: > > > Rather than replacing the fully-registered pages with pages of zeros, > > > you could simply unmap them. > > > > I don't like this option. It is nearly free to map all of the pages to > > the zero-page. You never have to allocate a page if the user never > > writes to it. > > Unmapping should work fine, as long as the VMA flags are > set appropriately. The page fault handler can take care > of the rest... I think there may be a difference in terminology here. What I originally proposed (and what I think Bill was reacting to) is the equivalent of sys_munmap() on the range of registered pages. That has the downsides that he mentioned; an address that was valid in the parent will now result in SIGSEGV or SIGBUS in the child, and it's explicitly endorsed by the userland APIs (such as MPI2) that it's valid to register stack addresses (for example). What I think you're proposing, Rik, is that VMA get destroyed (or split, if only part of it had been registered) and replaced with an anonymous one. That's a very low-overhead way of going about it, I think. Then as you say, the page fault handler will automatically give a zero page to the process when it faults on those addresses. Did I understand your suggestion correctly? I think I agree with Bill that having the child fault on pages which happened to have been registered by the parent would be a bad thing. This would, if I understand correctly, be visible in /proc/$$/maps. Which is OK, if a little bit suprising; but the alternatives are worse. -andy ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 20:14 ` Roland Dreier 2005-04-26 20:18 ` Timur Tabi @ 2005-04-26 20:32 ` Andrew Morton 2005-04-26 21:23 ` Roland Dreier 1 sibling, 1 reply; 59+ messages in thread From: Andrew Morton @ 2005-04-26 20:32 UTC (permalink / raw) To: Roland Dreier; +Cc: libor, hch, linux-kernel, openib-general, timur.tabi Roland Dreier <roland@topspin.com> wrote: > > Roland> a) app registers 0x0000 through 0x17ff > Roland> b) app registers 0x1800 through 0x2fff > Roland> c) app unregisters 0x0000 through 0x17ff > Roland> d) the page at 0x1000 must stay pinned > > Andrew> The userspace library should be able to track the tree and > Andrew> the overlaps, etc. Things might become interesting when > Andrew> the memory is MAP_SHARED pagecache and multiple > Andrew> independent processes are involved, although I guess > Andrew> that'd work OK. > > I used to think I knew how to handle this, but in your scheme where > the kernel is doing accounting for pinned memory by marking vmas with > VM_KERNEL_LOCKED, at step c), I don't see why the kernel won't unlock > vmas covering 0x0000 through 0x1fff and credit 8K back to the > process's pinning count. > > Sorry to be so dense but can you spell out what you think should > happen at steps a), b) and c) above? Well I was vaguely proposing that the userspace library keep track of the byteranges and the underlying page states. So in the above scenario userspace would leave the page at 0x1000 registered until all registrations against that page have been undone. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 20:32 ` Andrew Morton @ 2005-04-26 21:23 ` Roland Dreier 2005-04-27 0:05 ` Andrew Morton 0 siblings, 1 reply; 59+ messages in thread From: Roland Dreier @ 2005-04-26 21:23 UTC (permalink / raw) To: Andrew Morton; +Cc: libor, hch, linux-kernel, openib-general, timur.tabi Andrew> Well I was vaguely proposing that the userspace library Andrew> keep track of the byteranges and the underlying page Andrew> states. So in the above scenario userspace would leave Andrew> the page at 0x1000 registered until all registrations Andrew> against that page have been undone. OK, I already have code in userspace that keeps reference counts for overlapping regions, etc. However I'm not sure how to tie this in with reliable accounting of pinned memory -- we don't want malicious userspace code to be able fool the accounting, right? So I'm still trying to puzzle out what to do. I don't want to keep a complicated data structure in the kernel keeping track of what memory has been registered. Right now, I just keep a list of structs, one for each region, and when a process dies, I just go through region by region and do a put_page() to balance off the get_user_pages(). However I don't see how to make it work if I put the reference counting for overlapping regions in userspace but when I want mlock() accounting in the kernel. If a buggy/malicious app does: a) register from 0x0000 to 0x2fff b) register from 0x1000 to 0x1fff c) unregister from 0x0000 to 0x2fff then it seems the kernel is screwed unless it counts how many times a vma has been pinned. And adding a pin_count member to vm_struct seems like a pretty damn major step. We definitely have to make sure that userspace is never able to either unpin a page that is still registered with RDMA hardware, because that can lead to DMA to into memory that someone else owns. On the other hand, we don't want userspace to be able to defeat resource accounting by tricking the kernel into keeping page_count elevated after it credits the memory back to a process's limit on locked pages. The limit on the number of locked pages seems like a natural thing to check against, but perhaps we need a different limit for the number of pages pinned for use by RDMA hardware. Sort of the same way that there's a separate limit on the number of in-flight aios. - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 21:23 ` Roland Dreier @ 2005-04-27 0:05 ` Andrew Morton 2005-04-27 2:13 ` Roland Dreier 2005-04-27 3:21 ` Caitlin Bestler 0 siblings, 2 replies; 59+ messages in thread From: Andrew Morton @ 2005-04-27 0:05 UTC (permalink / raw) To: Roland Dreier; +Cc: libor, hch, linux-kernel, openib-general, timur.tabi Roland Dreier <roland@topspin.com> wrote: > > Andrew> Well I was vaguely proposing that the userspace library > Andrew> keep track of the byteranges and the underlying page > Andrew> states. So in the above scenario userspace would leave > Andrew> the page at 0x1000 registered until all registrations > Andrew> against that page have been undone. > > OK, I already have code in userspace that keeps reference counts for > overlapping regions, etc. However I'm not sure how to tie this in > with reliable accounting of pinned memory -- we don't want malicious > userspace code to be able fool the accounting, right? > > So I'm still trying to puzzle out what to do. I don't want to keep a > complicated data structure in the kernel keeping track of what memory > has been registered. Right now, I just keep a list of structs, one > for each region, and when a process dies, I just go through region by > region and do a put_page() to balance off the get_user_pages(). > > However I don't see how to make it work if I put the reference > counting for overlapping regions in userspace but when I want mlock() > accounting in the kernel. If a buggy/malicious app does: > > a) register from 0x0000 to 0x2fff > b) register from 0x1000 to 0x1fff > c) unregister from 0x0000 to 0x2fff As far as the kernel is concerned, step b) should be a no-op. (The kernel might choose to split the vma, but that's not significant). > then it seems the kernel is screwed unless it counts how many times a > vma has been pinned. And adding a pin_count member to vm_struct seems > like a pretty damn major step. > > We definitely have to make sure that userspace is never able to either > unpin a page that is still registered with RDMA hardware, because that > can lead to DMA to into memory that someone else owns. On the other > hand, we don't want userspace to be able to defeat resource accounting > by tricking the kernel into keeping page_count elevated after it > credits the memory back to a process's limit on locked pages. The kernel can simply register and unregister ranges for RDMA. So effectively a particular page is in either the registered or unregistered state. Kernel accounting counts the number of registered pages and compares this with rlimits. On top of all that, your userspace library needs to keep track of when pages should really be registered and unregistered with the kernel. Using overlap logic and per-page refcounting or whatever. No? ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-27 0:05 ` Andrew Morton @ 2005-04-27 2:13 ` Roland Dreier 2005-04-27 3:21 ` Caitlin Bestler 1 sibling, 0 replies; 59+ messages in thread From: Roland Dreier @ 2005-04-27 2:13 UTC (permalink / raw) To: Andrew Morton; +Cc: libor, hch, linux-kernel, openib-general, timur.tabi Andrew> The kernel can simply register and unregister ranges for Andrew> RDMA. So effectively a particular page is in either the Andrew> registered or unregistered state. Kernel accounting Andrew> counts the number of registered pages and compares this Andrew> with rlimits. Andrew> On top of all that, your userspace library needs to keep Andrew> track of when pages should really be registered and Andrew> unregistered with the kernel. Using overlap logic and Andrew> per-page refcounting or whatever. This is OK as long as userspace is trusted. However I don't see how this works when we don't trust userspace. The problem is that for an RDMA device (IB HCA or iWARP RNIC), a process can create many memory regions, each of which a separate virtual to physical translation map. For example, an app can do: a) register 0x0000 through 0xffff and get memory handle 1 b) register 0x0000 through 0xffff and get memory handle 2 c) use memory handle 1 for communication with remote app A d) use memory handle 2 for communication with remote app B Even though memory handles 1 and 2 both refer to exactly the same memory, they may have different lifetimes, might be attached to different connections, and so on. Clearly the memory at 0x0000 must stay pinned as long as the RDMA device thinks either memory handle 1 or memory handle 2 is valid. Furthermore, the kernel must be the one keeping track of how many regions refer to a given page because we can't allow userspace to be able to tell a device to go DMA to memory it doesn't own any more. Creation and destruction of these memory handles will always go through the kernel driver, so this isn't so bad. And get_user_pages() is almost exactly what we need: it stacks perfectly, since it operates on the page_count rather than just setting a bit in vm_flags. The main problem is that it doesn't check against RLIMIT_MEMLOCK. The most reasonable thing to do would seem to be having the IB kernel memory region code update current->mm->locked_vm and check it against RLIMIT_MEMLOCK. I guess it would be good to figure out an appropriate abstraction to export rather than monkeying with current->mm directly. We could also put this directly in get_user_pages(), but I'd be worried about messing with current users. I just don't see a way to make VM_KERNEL_LOCKED work. It would also be nice to have a way for apps to set VM_DONTCOPY appropriately. Christoph's suggestion of extending mmap() and mprotect() with PROT_DONTCOPY seems good to me, especially since it means we don't have to export do_mlock() functionality to modules. - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-27 0:05 ` Andrew Morton 2005-04-27 2:13 ` Roland Dreier @ 2005-04-27 3:21 ` Caitlin Bestler 1 sibling, 0 replies; 59+ messages in thread From: Caitlin Bestler @ 2005-04-27 3:21 UTC (permalink / raw) To: Andrew Morton Cc: Roland Dreier, hch, linux-kernel, openib-general, timur.tabi On 4/26/05, Andrew Morton <akpm@osdl.org> wrote: > > > > However I don't see how to make it work if I put the reference > > counting for overlapping regions in userspace but when I want mlock() > > accounting in the kernel. If a buggy/malicious app does: > > > > a) register from 0x0000 to 0x2fff > > b) register from 0x1000 to 0x1fff > > c) unregister from 0x0000 to 0x2fff > > As far as the kernel is concerned, step b) should be a no-op. (The kernel > might choose to split the vma, but that's not significant). > If "register" and "unregister" is meant in the RDMA sense then the above sequence is totally reasonable. The b) registration could be for a different protection domain that did not require access to all of the larger region. Unless a full counting lock is available from the kernel, the responsibility of the collective RDMA components would be to a) pin 0x0000 to 0x2fff, b) nothing c) unpin 0x000 to 0x0fff and 0x2000 to 0x2fff ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 19:28 ` Andrew Morton 2005-04-26 20:14 ` Roland Dreier @ 2005-04-27 3:15 ` Caitlin Bestler 1 sibling, 0 replies; 59+ messages in thread From: Caitlin Bestler @ 2005-04-27 3:15 UTC (permalink / raw) To: Andrew Morton Cc: Roland Dreier, hch, linux-kernel, openib-general, timur.tabi On 4/26/05, Andrew Morton <akpm@osdl.org> wrote: > Roland Dreier <roland@topspin.com> wrote: > > > > Libor> Do you mean that the set/clear parameters to do_mlock() > > Libor> are the actual flags which are set/cleared by the caller? > > Libor> Also, the issue remains that the flags are not reference > > Libor> counted which is a problem if you are dealing with > > Libor> overlapping memory region, or even if one region ends and > > Libor> another begins on the same page. Since the desire is to be > > Libor> able to pin any memory that a user can malloc this is a > > Libor> likely scenario. > > > > Good point... we need to figure out how to handle: > > > > a) app registers 0x0000 through 0x17ff > > b) app registers 0x1800 through 0x2fff > > c) app unregisters 0x0000 through 0x17ff > > d) the page at 0x1000 must stay pinned > > The userspace library should be able to track the tree and the overlaps, > etc. Things might become interesting when the memory is MAP_SHARED > pagecache and multiple independent processes are involved, although I guess > that'd work OK. > > But afaict the problem wherein part of a page needs VM_DONTCOPY and the > other part does not cannot be solved. > Which portion of the userspace library? HCA-dependent code, or common code? The HCA-dependent code would fail to count when the same memory was registered to different HCAs (for example to the internal network device and the external network device). The vendor-independent code *could* do it, but only by maintaining a complete list of all registrations that had been issued but not cancelled. That data would be redundant with data kept at the verb layer, and by the kernel. It *would' work, but maintaining highly redundant data at multiple layers is something that I generally try to avoid. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 23:58 ` Roland Dreier 2005-04-26 0:11 ` Andrew Morton @ 2005-04-26 2:03 ` IWAMOTO Toshihiro 2005-04-26 2:26 ` [openib-general] " Stephen Langdon 1 sibling, 1 reply; 59+ messages in thread From: IWAMOTO Toshihiro @ 2005-04-26 2:03 UTC (permalink / raw) To: Roland Dreier Cc: Andrew Morton, Timur Tabi, hch, hozer, linux-kernel, openib-general At Mon, 25 Apr 2005 16:58:03 -0700, Roland Dreier wrote: > Andrew> It would be better to obtain this memory via a mmap() of > Andrew> some special device node, so we can perform appropriate > Andrew> permission checking and clean everything up on unclean > Andrew> application exit. > > This seems to interact poorly with how applications want to use RDMA, > ie typically through a library interface such as MPI. People doing > HPC don't want to recode their apps to use a new allocator, they just > want to link to a new MPI library and have the app go fast. Such HPC users cannot use the memory hotremoval feature, and something needs to be implemented so that the NUMA migration can handle such memory properly, but I see your point. If such memory were allocated by a driver, the memory could be placed in non-hotremovable areas to avoid the above problems. -- IWAMOTO Toshihiro ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 2:03 ` IWAMOTO Toshihiro @ 2005-04-26 2:26 ` Stephen Langdon 0 siblings, 0 replies; 59+ messages in thread From: Stephen Langdon @ 2005-04-26 2:26 UTC (permalink / raw) To: IWAMOTO Toshihiro Cc: Roland Dreier, Andrew Morton, linux-kernel, openib-general, hch, Timur Tabi [-- Attachment #1.1: Type: text/plain, Size: 8697 bytes --] I don't think that we should jump to the conclusion that in the long term HPC users cannot benefit from support of mechanisms such as hotremoval of memory or other forms of page migration in physical memory. In an earlier exchange on the openib-general list Mike Krause sent the message quoted below on very much the same topic. On the other hand I am willing to accept that there is practical value to implementations which are not (yet) sophisticated to enough to support the migration functions. Steve Langdon > Michael Krause wrote: At 05:35 PM 3/14/2005, Caitlin Bestler wrote: > >> >> >> > -----Original Message----- >> > From: Troy Benjegerdes [ mailto:hozer@hozed.org] >> > Sent: Monday, March 14, 2005 5:06 PM >> > To: Caitlin Bestler >> > Cc: openib-general@openib.org >> > Subject: Re: [openib-general] Getting rid of pinned memory requirement >> > >> > > >> > > The key is that the entire operation either has to be fast >> > > enough so that no connection or application session layer >> > > time-outs occur, or an end-to-end agreement to suspend the >> > > connetion is a requirement. The first option seems more >> > > plausible to me, the second essentially >> > > reuqires extending the CM protocol. That's a tall order even for >> > > InfiniBand, and it's even worse for iWARP where the CM >> > > functionality typically ends when the connection is established. >> > >> > I'll buy the good network design argument. > > > I and others designed InfiniBand RNR (Receiver not ready) operations > to allow one to adjust V-to-P mappings (not change the address that > was advertised) in order to allow an OS to safely play some games with > memory and not drop a connection. The time values associated with RNR > allow a solution to tolerate up to infinite amount of time to perform > such operations but the envisioned goal was to do this on the order of > a handful or milliseconds in the worse case. For iWARP, there was no > support for defining RNR functionality as indeed many people claimed > one could just drop in-bound segments and allow the retransmission > protocol to deal with the delay (even if this has performance > implications due to back-off algorithms though some claim SACK would > minimize this to a large extent). Again, the idea was to minimize the > worse case to milliseconds of down time. BTW, all of this assumed > that the OS would not perform these types of changes that often so the > long-term impact on an application would be minimum. > >> > >> > I suppose if the kernel wants to revoke a card's pinned >> > memory, we should be able to guarantee that it gets new >> > pinned memory within a bounded time. What sort of timing do >> > we need? Milliseconds? >> > Microseconds? >> > >> > In the case of iWarp, isn't this just TCP underneath? If so, >> > can't we just drop any packets in the pipe on the floor and >> > let them get retransmitted? (I suppose the same argument goes >> > for infiniband.. >> > what sort of a time window do we have for retransmission?) >> > >> > What are the limits on end-to-end flow control in IB and iWarp? >> > >> >> >From the RDMA Provider's perspective, the short answer is "quick >> enough so that I don't have to do anything heroic to keep the >> connection alive." > > > It should not require anything heroic. What is does require is a > local method to suspend the local QP(s) so that it cannot place or > read memory in the effected area. That can take some time depending > upon the implementation. There is then the time to over write the > mappings which again depending upon the implementation and the number > of mappings could be milliseconds in length. > >> With TCP you also have to add "and healthy". If you've ever had a >> long download that got effectively stalled by a burst of noise and >> you just hit the 'reload' button on your browser then you know what >> I'm talking about. >> >> But in transport neutral terms I would think that one RTT is >> definitely safe -- that much data could have >> been dropped by one switch failure or one nasty spike in inbound noise. >> >> > > >> > > Yes, there are limits on how much memory you can mlock, or even >> > > allocate. Applications are required to reqister memory precisely >> > > because the required guarantess are not there by default. >> > Eliminating >> > > those guarantees *is* effectively rewriting every RDMA application >> > > without even letting them know. >> > >> > Some of this argument is a policy issue, which I would argue >> > shouldn't be hard-coded in the code or in the network hardware. >> > >> > At least in my view, the guarantees are only there to make >> > applications go fast. We are getting low latency and high >> > performance with infiniband by making memory registration go >> > really really slow. If, to make big HPC simulation >> > applications work, we wind up doing memcpy() to put the data >> > into a registered buffer because we can't register half of >> > physical memory, the application isn't going very fast. >> > >> >> What you are looking for is a distinction between registering >> memory to *enable* the RNIC to optimize local access and >> registering memory to enable its being advertised to the >> remote end. >> >> Early implementations of RDMA, both IB and iWARP, have not >> distinquished between the two. But theoretically *applications* >> do not need memory regions that are not enabled for remote >> access to be pinned. That is an RNIC requirement that could >> evolve. But applications themselves *do* need remotely >> accessible memory regions, portions of which they intend >> to advertise with RKeys, to be truly available (i.e., pinned). >> >> You are also making a policy assumption that an application >> that actually needs half of physical memory should be using >> paged memory. Memory is cheap, and if performance is critical >> why should this memory be swapped out to disk? >> >> Is the limitation on not being able to register half of >> physical memory based upon some assumption that swapping >> is a requirement? Or is it a limitation in the memory region >> size? If it's the latter, you need to get the OS to support >> larger page sizes. > > > For some OS, you can pin very large areas. I've seen 15/16 of memory > being able to be pinned with no adverse impacts on the applications. > For these OS, kernel memory is effectively pinned memory. As such, > depending upon the mix of services being provided, the system may > operate quite nicely with such large amounts of memory being pinned. > As more services are "ported" to operate over RDMA technologies, > memory management isn't necessarily any harder; it just becomes > something people have to think more about. Today's VM designs have > allowed people to get sloppy as they assume that swapping will occur > and since many platforms are not that loaded, they don't see any real > adverse impacts. User-space RDMA applications requires people to > think once again about memory management and that swapping isn't a > get-out-of-jail card. One needs to develop resource management tools > to determine who obtains specified amounts of resources and their > priorities. For the most part, this is somewhat a re-invention of > some thinking that went into the micro-kernel work in past years. > These problems are not intractable; they are only constrained by the > legacy inertia inherent in all technologies today. > > Mike > > > IWAMOTO Toshihiro wrote: >At Mon, 25 Apr 2005 16:58:03 -0700, >Roland Dreier wrote: > > >> Andrew> It would be better to obtain this memory via a mmap() of >> Andrew> some special device node, so we can perform appropriate >> Andrew> permission checking and clean everything up on unclean >> Andrew> application exit. >> >>This seems to interact poorly with how applications want to use RDMA, >>ie typically through a library interface such as MPI. People doing >>HPC don't want to recode their apps to use a new allocator, they just >>want to link to a new MPI library and have the app go fast. >> >> > >Such HPC users cannot use the memory hotremoval feature, and something >needs to be implemented so that the NUMA migration can handle such >memory properly, but I see your point. > >If such memory were allocated by a driver, the memory could be placed >in non-hotremovable areas to avoid the above problems. > >-- >IWAMOTO Toshihiro >_______________________________________________ >openib-general mailing list >openib-general@openib.org >http://openib.org/mailman/listinfo/openib-general > >To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > [-- Attachment #1.2: steve.langdon.vcf --] [-- Type: text/x-vcard, Size: 348 bytes --] begin:vcard fn:Steve Langdon n:Langdon;Stephen org:Hewlett-Packard;Consulting & Architecture Group adr:MS LKG1-3/B19;;550 King Street;Littleton;MA;01460;USA email;internet:steve.langdon@hp.com title:Fellow tel;work:+1 978-506-5771 tel;fax:+1 978-742-1144 tel;home:+1 978-456-8177 tel;cell:+1 978-618-8599 x-mozilla-html:TRUE version:2.1 end:vcard [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/x-pkcs7-signature, Size: 6189 bytes --] ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 22:14 ` Andrew Morton 2005-04-25 22:21 ` Timur Tabi @ 2005-04-25 22:23 ` Timur Tabi 2005-04-25 22:35 ` Andrew Morton 2005-04-26 0:02 ` Roland Dreier 2 siblings, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-04-25 22:23 UTC (permalink / raw) To: Andrew Morton; +Cc: Roland Dreier, hch, hozer, linux-kernel, openib-general Andrew Morton wrote: > The way we expect get_user_pages() to be used is that the kernel will use > get_user_pages() once per application I/O request. Are you saying that the mapping obtained by get_user_pages() is valid only within the context of the IOCtl call? That once the driver returns from the IOCtl, the mapping should no longer be used? -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 22:23 ` Timur Tabi @ 2005-04-25 22:35 ` Andrew Morton 2005-04-25 23:17 ` [openib-general] " Libor Michalek 0 siblings, 1 reply; 59+ messages in thread From: Andrew Morton @ 2005-04-25 22:35 UTC (permalink / raw) To: Timur Tabi; +Cc: roland, hch, hozer, linux-kernel, openib-general Timur Tabi <timur.tabi@ammasso.com> wrote: > > Andrew Morton wrote: > > > The way we expect get_user_pages() to be used is that the kernel will use > > get_user_pages() once per application I/O request. > > Are you saying that the mapping obtained by get_user_pages() is valid only within the > context of the IOCtl call? That once the driver returns from the IOCtl, the mapping > should no longer be used? Yes, we expect that all the pages which get_user_pages() pinned will become unpinned within the context of the syscall which pinned the pages. Or shortly after, in the case of async I/O. This is because there is no file descriptor or anything else associated with the pages which permits the kernel to clean stuff up on unclean application exit. Also there are the obvious issues with permitting pinning of unbounded amounts of memory. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 22:35 ` Andrew Morton @ 2005-04-25 23:17 ` Libor Michalek 2005-04-25 23:24 ` Andrew Morton 0 siblings, 1 reply; 59+ messages in thread From: Libor Michalek @ 2005-04-25 23:17 UTC (permalink / raw) To: Andrew Morton; +Cc: Timur Tabi, hch, linux-kernel, openib-general On Mon, Apr 25, 2005 at 03:35:42PM -0700, Andrew Morton wrote: > Timur Tabi <timur.tabi@ammasso.com> wrote: > > > > Andrew Morton wrote: > > > > > The way we expect get_user_pages() to be used is that the kernel will use > > > get_user_pages() once per application I/O request. > > > > Are you saying that the mapping obtained by get_user_pages() is valid only within the > > context of the IOCtl call? That once the driver returns from the IOCtl, the mapping > > should no longer be used? > > Yes, we expect that all the pages which get_user_pages() pinned will become > unpinned within the context of the syscall which pinned the pages. Or > shortly after, in the case of async I/O. When a network protocol is making use of async I/O the amount of time between posting the read request and getting the completion for that request is unbounded since it depends on the other half of the connection sending some data. In this case the buffer that was pinned during the io_submit() may be pinned, and holding the pages, for a long time. During this time the process might fork, at this point any data received will be placed into the wrong spot. > This is because there is no file descriptor or anything else associated > with the pages which permits the kernel to clean stuff up on unclean > application exit. Also there are the obvious issues with permitting > pinning of unbounded amounts of memory. Correct, the driver must be able to determine that the process has died and clean up after it, so the pinned region in most implementations is associated with an open file descriptor. -Libor ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 23:17 ` [openib-general] " Libor Michalek @ 2005-04-25 23:24 ` Andrew Morton 2005-04-25 23:37 ` Caitlin Bestler 2005-04-26 3:55 ` Libor Michalek 0 siblings, 2 replies; 59+ messages in thread From: Andrew Morton @ 2005-04-25 23:24 UTC (permalink / raw) To: Libor Michalek; +Cc: timur.tabi, hch, linux-kernel, openib-general Libor Michalek <libor@topspin.com> wrote: > > On Mon, Apr 25, 2005 at 03:35:42PM -0700, Andrew Morton wrote: > > Timur Tabi <timur.tabi@ammasso.com> wrote: > > > > > > Andrew Morton wrote: > > > > > > > The way we expect get_user_pages() to be used is that the kernel will use > > > > get_user_pages() once per application I/O request. > > > > > > Are you saying that the mapping obtained by get_user_pages() is valid only within the > > > context of the IOCtl call? That once the driver returns from the IOCtl, the mapping > > > should no longer be used? > > > > Yes, we expect that all the pages which get_user_pages() pinned will become > > unpinned within the context of the syscall which pinned the pages. Or > > shortly after, in the case of async I/O. > > When a network protocol is making use of async I/O the amount of time > between posting the read request and getting the completion for that > request is unbounded since it depends on the other half of the connection > sending some data. In this case the buffer that was pinned during the > io_submit() may be pinned, and holding the pages, for a long time. Sure. > During > this time the process might fork, at this point any data received will be > placed into the wrong spot. Well the data is placed in _a_ spot. That's only the "wrong" spot because you've defined it to be wrong! IOW: what behaviour are you actually looking for here, and why, and does it matter? > > This is because there is no file descriptor or anything else associated > > with the pages which permits the kernel to clean stuff up on unclean > > application exit. Also there are the obvious issues with permitting > > pinning of unbounded amounts of memory. > > Correct, the driver must be able to determine that the process has died > and clean up after it, so the pinned region in most implementations is > associated with an open file descriptor. How is that association created? ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 23:24 ` Andrew Morton @ 2005-04-25 23:37 ` Caitlin Bestler 2005-04-26 0:10 ` Andrew Morton 2005-04-26 3:55 ` Libor Michalek 1 sibling, 1 reply; 59+ messages in thread From: Caitlin Bestler @ 2005-04-25 23:37 UTC (permalink / raw) To: Andrew Morton Cc: Libor Michalek, hch, linux-kernel, openib-general, timur.tabi On 4/25/05, Andrew Morton <akpm@osdl.org> wrote: > > > > This is because there is no file descriptor or anything else associated > > > with the pages which permits the kernel to clean stuff up on unclean > > > application exit. Also there are the obvious issues with permitting > > > pinning of unbounded amounts of memory. > > > > Correct, the driver must be able to determine that the process has died > > and clean up after it, so the pinned region in most implementations is > > associated with an open file descriptor. > > How is that association created? There is not a file descrptor, but there is an rnic handle. Both DAPL and IT-API require that process death will result in the handle and all of its dependent objects being released. The rnic handle can always be declared to be a "file descriptor" if that makes it follow normal OS conventions more precisiely. There is also a need for some form of resource manager to approve creation of Memory Regions. Obviously you cannot have multiple applications claiming half of physical memory. But if you merely require the user to have root privileges in order to create a Memory Region, and then take a first-come first-served attitude, I don't think you end up with something that is truly a general purpose capability. A general purpose RDMA capability requires the ability to indefinitely pin large portions of user memory. It makes sense to integrate that with OS policy control over resource utilization and to integrate it with memory suspend/resume capabilities so that hotplug memory can be supported. What you can't do is downgrade a Memory Region so that it is no longer a memory region. Doing that means that you are not truly supporting RDMA. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 23:37 ` Caitlin Bestler @ 2005-04-26 0:10 ` Andrew Morton 0 siblings, 0 replies; 59+ messages in thread From: Andrew Morton @ 2005-04-26 0:10 UTC (permalink / raw) To: Caitlin Bestler; +Cc: libor, hch, linux-kernel, openib-general, timur.tabi Caitlin Bestler <caitlin.bestler@gmail.com> wrote: > > > > > > > This is because there is no file descriptor or anything else associated > > > > with the pages which permits the kernel to clean stuff up on unclean > > > > application exit. Also there are the obvious issues with permitting > > > > pinning of unbounded amounts of memory. > > > > > > Correct, the driver must be able to determine that the process has died > > > and clean up after it, so the pinned region in most implementations is > > > associated with an open file descriptor. > > > > How is that association created? > > > There is not a file descrptor, but there is an rnic handle. Both DAPL > and IT-API require that process death will result in the handle and all > of its dependent objects being released. What's an "rnic handle", in Linux terms? > The rnic handle can always be declared to be a "file descriptor" if > that makes it follow normal OS conventions more precisiely. Does that mean that the code has not yet been implemented? Yes, a Linux fd is appropriate. But we don't have any sane way right now of saying "you need to run put_page() against all these pages in the ->release() handler". That'll need to be coded by yourselves. > There is also a need for some form of resource manager to approve > creation of Memory Regions. Obviously you cannot have multiple > applications claiming half of physical memory. The kernel already has considerable resource management capabilities. Please consider using/extending/generalising those before inventing anything new. RLIMIT_MEMLOCK would be a starting point. > But if you merely require the user to have root privileges in order > to create a Memory Region, and then take a first-come first-served > attitude, I don't think you end up with something that is truly a > general purpose capability. We don't want code in the kernel which will permit hostile unprivileged users to trivially cause the box to lock up. RLIMIT_MEMLOCK and, if necessary, CAP_IPC_LOCK sound appropriate here. > A general purpose RDMA capability requires the ability to indefinitely > pin large portions of user memory. It makes sense to integrate that > with OS policy control over resource utilization and to integrate it with > memory suspend/resume capabilities so that hotplug memory can > be supported. What you can't do is downgrade a Memory Region so > that it is no longer a memory region. Doing that means that you are > not truly supporting RDMA. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 23:24 ` Andrew Morton 2005-04-25 23:37 ` Caitlin Bestler @ 2005-04-26 3:55 ` Libor Michalek 1 sibling, 0 replies; 59+ messages in thread From: Libor Michalek @ 2005-04-26 3:55 UTC (permalink / raw) To: Andrew Morton; +Cc: timur.tabi, hch, linux-kernel, openib-general On Mon, Apr 25, 2005 at 04:24:05PM -0700, Andrew Morton wrote: > Libor Michalek <libor@topspin.com> wrote: > > On Mon, Apr 25, 2005 at 03:35:42PM -0700, Andrew Morton wrote: > > > > > Yes, we expect that all the pages which get_user_pages() pinned > > > will become unpinned within the context of the syscall which pinned > > > the pages. Or shortly after, in the case of async I/O. > > > > When a network protocol is making use of async I/O the amount of time > > between posting the read request and getting the completion for that > > request is unbounded since it depends on the other half of the connection > > sending some data. In this case the buffer that was pinned during the > > io_submit() may be pinned, and holding the pages, for a long time. > > Sure. > > > During > > this time the process might fork, at this point any data received will be > > placed into the wrong spot. > > Well the data is placed in _a_ spot. That's only the "wrong" spot because > you've defined it to be wrong! > > IOW: what behaviour are you actually looking for here, and why, and does it > matter? For example a network server app has an open connection on which it uses async IO to submit two buffers for a read operation. Both buffers are pinned using get_user_pages() and the connection waits for data to arrive. The connection received data, it is written into the first buffer, the app is notified using async IO, and it retreives the async IO completion. The app reads the buffer which happens to contain a command to spawn a child, the app forks a child. Now there is still a buffer posted for read and if more data arrives on the connection that data is copied to the pages which were saved when the buffer was pinned. The app is notified, retrieves the async IO completion, but when it goes to read that buffer it will not have the new data. > > > This is because there is no file descriptor or anything else associated > > > with the pages which permits the kernel to clean stuff up on unclean > > > application exit. Also there are the obvious issues with permitting > > > pinning of unbounded amounts of memory. > > > > Correct, the driver must be able to determine that the process has died > > and clean up after it, so the pinned region in most implementations is > > associated with an open file descriptor. > > How is that association created? The kernel module which pinned the memory is responsible for unpinning it if the file descriptor, which was used to deliver the command that resulted in the pinning, is closed. -Libor ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-25 22:14 ` Andrew Morton 2005-04-25 22:21 ` Timur Tabi 2005-04-25 22:23 ` Timur Tabi @ 2005-04-26 0:02 ` Roland Dreier 2005-04-26 6:12 ` Christoph Hellwig 2 siblings, 1 reply; 59+ messages in thread From: Roland Dreier @ 2005-04-26 0:02 UTC (permalink / raw) To: Andrew Morton; +Cc: timur.tabi, hch, hozer, linux-kernel, openib-general Andrew> Whoa, hang on. Andrew> The way we expect get_user_pages() to be used is that the Andrew> kernel will use get_user_pages() once per application I/O Andrew> request. Andrew> Are you saying that RDMA clients will semi-permanently own Andrew> pages which were pinned by get_user_pages()? That those Andrew> pages will be used for multiple separate I/O operations? Andrew> If so, then that's a significant design departure and it Andrew> would be good to hear why it is necessary. The idea is that applications manage the lifetime of pinned memory regions. They can do things like post multiple I/O operations without any page-walking overhead, or pass a buffer descriptor to a remote host who will send data at some indeterminate time in the future. In addition, InfiniBand has the notion of atomic operations, so a cluster application may be using some memory region to implement a global lock. This might not be the most kernel-friendly design but it is pretty deeply ingrained in the design of RDMA transports like InfiniBand and iWARP (RDMA over IP). I'm also not opposed to implementing some other mechanism to make this work, but the combiniation of get_user_pages() in the kernel and extending mprotect() to allow setting VM_DONTCOPY seems to work fine. - R. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 0:02 ` Roland Dreier @ 2005-04-26 6:12 ` Christoph Hellwig 2005-04-26 13:45 ` [openib-general] " Caitlin Bestler 0 siblings, 1 reply; 59+ messages in thread From: Christoph Hellwig @ 2005-04-26 6:12 UTC (permalink / raw) To: Roland Dreier Cc: Andrew Morton, timur.tabi, hch, hozer, linux-kernel, openib-general On Mon, Apr 25, 2005 at 05:02:36PM -0700, Roland Dreier wrote: > The idea is that applications manage the lifetime of pinned memory > regions. They can do things like post multiple I/O operations without > any page-walking overhead, or pass a buffer descriptor to a remote > host who will send data at some indeterminate time in the future. In > addition, InfiniBand has the notion of atomic operations, so a cluster > application may be using some memory region to implement a global lock. > > This might not be the most kernel-friendly design but it is pretty > deeply ingrained in the design of RDMA transports like InfiniBand and > iWARP (RDMA over IP). Actuallky, no it isn't. All these transports would work just fine with the mmap a character device to hand out memory from the kernel approach I told you to use multiple times and Andrew mentioned in this thread aswell. What doesn't work with that design are the braindead designed by comittee APIs in the RDMA world - but I don't think we should care about them too much. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-26 6:12 ` Christoph Hellwig @ 2005-04-26 13:45 ` Caitlin Bestler 0 siblings, 0 replies; 59+ messages in thread From: Caitlin Bestler @ 2005-04-26 13:45 UTC (permalink / raw) To: Christoph Hellwig, Roland Dreier, Andrew Morton, timur.tabi, hozer, linux-kernel, openib-general On 4/25/05, Christoph Hellwig <hch@infradead.org> wrote: > On Mon, Apr 25, 2005 at 05:02:36PM -0700, Roland Dreier wrote: > > The idea is that applications manage the lifetime of pinned memory > > regions. They can do things like post multiple I/O operations without > > any page-walking overhead, or pass a buffer descriptor to a remote > > host who will send data at some indeterminate time in the future. In > > addition, InfiniBand has the notion of atomic operations, so a cluster > > application may be using some memory region to implement a global lock. > > > > This might not be the most kernel-friendly design but it is pretty > > deeply ingrained in the design of RDMA transports like InfiniBand and > > iWARP (RDMA over IP). > > Actuallky, no it isn't. All these transports would work just fine with > the mmap a character device to hand out memory from the kernel approach > I told you to use multiple times and Andrew mentioned in this thread aswell. > What doesn't work with that design are the braindead designed by comittee > APIs in the RDMA world - but I don't think we should care about them too > much. > RDMA registers and uses the memory the user specifies. That is why byte granularity and multiple redundant registrations are explicitly specified. The mechanism by which this requirement is implemented is of course OS dependent. But the requirements are that the application specifies what portion of their memory they want registered (or what set of physical pages if they have sufficient privilege) and that request is either honored or refused by a resource manager (one preferably as integrated with general OS resource management as possible). The other aspect is that remotely enabled memory regions and memory windows most be enabled for hardware access for the duration of the region or window -- indefinitely until process death or explicit termination by the application layer. Theoretically there is nothing in the wire protocols that requires source buffers to be pinned indefinitely, but that is the only way any RDMA interface has ever worked -- so "brain death" must be pretty widespread. The fact that this problem must be solved for remotely accessible buffers, and that for cluster applications like MPI there is no distinction between buffers used for inbound messages and outbound messages, might have something to do with this. User verbs needs to deal with these actual Memory Registration requirements, including the very real application need for Memory Windows. The solution should map to existing OS controls as much as possible. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-11 15:34 ` Roland Dreier 2005-04-11 16:33 ` Troy Benjegerdes @ 2005-04-18 16:09 ` Timur Tabi 2005-04-18 16:16 ` Arjan van de Ven 1 sibling, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-04-18 16:09 UTC (permalink / raw) To: Roland Dreier; +Cc: Troy Benjegerdes, linux-kernel, openib-general Roland Dreier wrote: > Troy> How is memory pinning handled? (I haven't had time to read > Troy> all the code, so please excuse my ignorance of something > Troy> obvious). > > The userspace library calls mlock() and then the kernel does > get_user_pages(). Why do you call mlock() and get_user_pages()? In our code, we only call mlock(), and the memory is pinned. We have a test case that fails if only get_user_pages() is called, but it passes if only mlock() is called. -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-18 16:09 ` Timur Tabi @ 2005-04-18 16:16 ` Arjan van de Ven 2005-04-18 16:25 ` Timur Tabi 0 siblings, 1 reply; 59+ messages in thread From: Arjan van de Ven @ 2005-04-18 16:16 UTC (permalink / raw) To: Timur Tabi; +Cc: Roland Dreier, Troy Benjegerdes, linux-kernel, openib-general On Mon, 2005-04-18 at 11:09 -0500, Timur Tabi wrote: > Roland Dreier wrote: > > Troy> How is memory pinning handled? (I haven't had time to read > > Troy> all the code, so please excuse my ignorance of something > > Troy> obvious). > > > > The userspace library calls mlock() and then the kernel does > > get_user_pages(). > > Why do you call mlock() and get_user_pages()? In our code, we only call mlock(), and the > memory is pinned. this is a myth; linux is free to move the page about in physical memory even if it's mlock()ed!! And even then, the user can munlock the memory from another thread etc etc. Not a good idea. get_user_pages() is used from AIO and other parts of the kernel for similar purposes and in fact is designed for it, so it better work. If it has bugs those should be fixed, not worked around! ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-18 16:16 ` Arjan van de Ven @ 2005-04-18 16:25 ` Timur Tabi 2005-04-18 19:40 ` Arjan van de Ven 0 siblings, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-04-18 16:25 UTC (permalink / raw) To: Arjan van de Ven Cc: Roland Dreier, Troy Benjegerdes, linux-kernel, openib-general Arjan van de Ven wrote: > this is a myth; linux is free to move the page about in physical memory > even if it's mlock()ed!! Then Linux has a very odd definition of the word "locked". > And even then, the user can munlock the memory from another thread etc > etc. Not a good idea. Well, that's okay, because then the app is doing something stupid, so we don't worry about that. > get_user_pages() is used from AIO and other parts of the kernel for > similar purposes and in fact is designed for it, so it better work. If > it has bugs those should be fixed, not worked around! I've been complaining about get_user_pages() not working for a long time now, but I can only demonstrate the problem with our hardware. See my other post in this thread for details. -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-18 16:25 ` Timur Tabi @ 2005-04-18 19:40 ` Arjan van de Ven 2005-04-18 20:07 ` [openib-general] " Bernhard Fischer 0 siblings, 1 reply; 59+ messages in thread From: Arjan van de Ven @ 2005-04-18 19:40 UTC (permalink / raw) To: Timur Tabi; +Cc: Roland Dreier, Troy Benjegerdes, linux-kernel, openib-general On Mon, 2005-04-18 at 11:25 -0500, Timur Tabi wrote: > Arjan van de Ven wrote: > > > this is a myth; linux is free to move the page about in physical memory > > even if it's mlock()ed!! > > Then Linux has a very odd definition of the word "locked". > > > And even then, the user can munlock the memory from another thread etc > > etc. Not a good idea. > > Well, that's okay, because then the app is doing something stupid, so we don't worry about > that. you should since that physical page can be reused, say by a root process, and you'd be majorly screwed ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-18 19:40 ` Arjan van de Ven @ 2005-04-18 20:07 ` Bernhard Fischer 2005-04-21 2:17 ` Troy Benjegerdes 0 siblings, 1 reply; 59+ messages in thread From: Bernhard Fischer @ 2005-04-18 20:07 UTC (permalink / raw) To: Arjan van de Ven; +Cc: Timur Tabi, linux-kernel, openib-general On Mon, Apr 18, 2005 at 09:40:40PM +0200, Arjan van de Ven wrote: >On Mon, 2005-04-18 at 11:25 -0500, Timur Tabi wrote: >> Arjan van de Ven wrote: >> >> > this is a myth; linux is free to move the page about in physical memory >> > even if it's mlock()ed!! darn, yes, this is true. I know people who introduced #define VM_RESERVED 0x00080000 /* Don't unmap it from swap_out */ to vm_flags just because of this. I'll just hold my breath and won't delve further. >> >> Then Linux has a very odd definition of the word "locked". >> >> > And even then, the user can munlock the memory from another thread etc >> > etc. Not a good idea. >> >> Well, that's okay, because then the app is doing something stupid, so we don't worry about >> that. > >you should since that physical page can be reused, say by a root >process, and you'd be majorly screwed ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-18 20:07 ` [openib-general] " Bernhard Fischer @ 2005-04-21 2:17 ` Troy Benjegerdes 2005-04-21 3:07 ` Timur Tabi 0 siblings, 1 reply; 59+ messages in thread From: Troy Benjegerdes @ 2005-04-21 2:17 UTC (permalink / raw) To: Bernhard Fischer Cc: Arjan van de Ven, Timur Tabi, linux-kernel, openib-general On Mon, Apr 18, 2005 at 10:07:12PM +0200, Bernhard Fischer wrote: > On Mon, Apr 18, 2005 at 09:40:40PM +0200, Arjan van de Ven wrote: > >On Mon, 2005-04-18 at 11:25 -0500, Timur Tabi wrote: > >> Arjan van de Ven wrote: > >> > >> > this is a myth; linux is free to move the page about in physical memory > >> > even if it's mlock()ed!! > darn, yes, this is true. > I know people who introduced > #define VM_RESERVED 0x00080000 /* Don't unmap it from swap_out > */ Someone (aka Tospin, infinicon, and Amasso) should probably post a patch adding '#define VM_REGISTERD 0x01000000', and some extensions to something like 'madvise' to set pages to be registered. My preference is said patch will also allow a way for the kernel to reclaim registered memory from an application under memory pressure. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-21 2:17 ` Troy Benjegerdes @ 2005-04-21 3:07 ` Timur Tabi 2005-04-21 17:38 ` Andy Isaacson 0 siblings, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-04-21 3:07 UTC (permalink / raw) To: Troy Benjegerdes Cc: Bernhard Fischer, Arjan van de Ven, linux-kernel, openib-general Troy Benjegerdes wrote: > Someone (aka Tospin, infinicon, and Amasso) should probably post a patch > adding '#define VM_REGISTERD 0x01000000', and some extensions to > something like 'madvise' to set pages to be registered. > > My preference is said patch will also allow a way for the kernel to > reclaim registered memory from an application under memory pressure. I don't know if VM_REGISTERED is a good idea or not, but it should be absolutely impossible for the kernel to reclaim "registered" (aka pinned) memory, no matter what. For RDMA services (such as Infiniband, iWARP, etc), it's normal for non-root processes to pin hundreds of megabytes of memory, and that memory better be locked to those physical pages until the application deregisters them. If kernel really thinks it needs to unpin those pages, then at the very least it should kill the process, and the syslog better have a very clear message indicating why. That way, the application doesn't continue thinking that everything's still going to work. If those pages become unpinned, the applications are going to experience serious data corruption. ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-21 3:07 ` Timur Tabi @ 2005-04-21 17:38 ` Andy Isaacson 2005-04-21 18:39 ` Timur Tabi 0 siblings, 1 reply; 59+ messages in thread From: Andy Isaacson @ 2005-04-21 17:38 UTC (permalink / raw) To: Timur Tabi Cc: Troy Benjegerdes, Bernhard Fischer, Arjan van de Ven, linux-kernel, openib-general On Wed, Apr 20, 2005 at 10:07:45PM -0500, Timur Tabi wrote: > Troy Benjegerdes wrote: > >Someone (aka Tospin, infinicon, and Amasso) should probably post a patch > >adding '#define VM_REGISTERD 0x01000000', and some extensions to > >something like 'madvise' to set pages to be registered. > > > >My preference is said patch will also allow a way for the kernel to > >reclaim registered memory from an application under memory pressure. > > I don't know if VM_REGISTERED is a good idea or not, but it should be > absolutely impossible for the kernel to reclaim "registered" (aka pinned) > memory, no matter what. For RDMA services (such as Infiniband, iWARP, etc), > it's normal for non-root processes to pin hundreds of megabytes of memory, > and that memory better be locked to those physical pages until the > application deregisters them. If you take the hardline position that "the app is the only thing that matters", your code is unlikely to get merged. Linux is a general-purpose OS. I don't think that Troy was suggesting the kernel should deregister memory without notifying the application. Personally, I envision something like the NetBSD Scheduler Activations (SA) work, where the kernel can notify the app of changes to its state in a very efficient manner. (According to the NetBSD design whitepaper, the kernel does an upcall whenever the multithreaded app gains or loses a CPU!) In a Linux context, I doubt that fullblown SA is necessary or appropriate. Rather, I'd suggest two new signals, SIGMEMLOW and SIGMEMCRIT. The userland comms library registers handlers for both. When the kernel decides that it needs to reclaim some memory from the app, it sends SIGMEMLOW. The comms library then has the responsibility to un-reserve some memory in an orderly fashion. If a reasonable [1] time has expired since SIGMEMLOW and the kernel is still hungry, the kernel sends SIGMEMCRIT. At this point, the comms lib *must* unregister some memory [2] even if it has to drop state to do so; if it returns from the signal handler without having unregistered the memory, the kernel will SIGKILL. [1] Part of the interface spec should cover the expectation as to how long the library is allowed to take; I'd guess that 2 timeslices should suffice. [2] Is there a way for the kernel to pass down to userspace how many pages it wants, maybe in the sigcontext? > If kernel really thinks it needs to unpin those pages, then at the very > least it should kill the process, and the syslog better have a very clear > message indicating why. That way, the application doesn't continue > thinking that everything's still going to work. If those pages become > unpinned, the applications are going to experience serious data corruption. You might want to consider what happens with your communication system in a machine running power-saving modes (in the limit, suspend-to-disk). Of course most machines with Infiniband adapters aren't running swsusp, but it's not inconceivable that blade servers might sleep to lower power and cooling costs. -andy ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-21 17:38 ` Andy Isaacson @ 2005-04-21 18:39 ` Timur Tabi 2005-04-21 19:56 ` Andy Isaacson 0 siblings, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-04-21 18:39 UTC (permalink / raw) To: Andy Isaacson Cc: Troy Benjegerdes, Bernhard Fischer, Arjan van de Ven, linux-kernel, openib-general Andy Isaacson wrote: > If you take the hardline position that "the app is the only thing that > matters", your code is unlikely to get merged. Linux is a > general-purpose OS. The problem is that our driver and library implement an API that we don't fully control. The API states that the application allocates the memory and tells the library to register it. The app then goes on its merry way until it's done, at which point it tells the library to deregister the memory. Neither the app nor the API has any provision for the app to be notified that the memory is no longer pinned and therefore can't be trusted. That would be considered a critical failure from the app's perspective, so the kernel would be doing it a favor by killing the process. > You might want to consider what happens with your communication system > in a machine running power-saving modes (in the limit, suspend-to-disk). > Of course most machines with Infiniband adapters aren't running swsusp, > but it's not inconceivable that blade servers might sleep to lower power > and cooling costs. Any application that registers memory, will in all likelihood be running at 100% CPU non-stop. The computer is not going to be doing anything else but whatever that app is trying to do. The application could conceiveable register gigabytes of RAM, and if even a single page becomes unpinned, the whole thing is worthless. The application cannot do anything meaningful if it gets a message saying that some of the memory has become unpinned and should not be used. So the real question is: how important is it to the kernel developers that Linux support these kinds of enterprise-class applications? -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-21 18:39 ` Timur Tabi @ 2005-04-21 19:56 ` Andy Isaacson 2005-04-21 20:07 ` Timur Tabi 0 siblings, 1 reply; 59+ messages in thread From: Andy Isaacson @ 2005-04-21 19:56 UTC (permalink / raw) To: Timur Tabi Cc: Troy Benjegerdes, Bernhard Fischer, Arjan van de Ven, linux-kernel, openib-general On Thu, Apr 21, 2005 at 01:39:35PM -0500, Timur Tabi wrote: > Andy Isaacson wrote: > >If you take the hardline position that "the app is the only thing that > >matters", your code is unlikely to get merged. Linux is a > >general-purpose OS. > > The problem is that our driver and library implement an API that we don't > fully control. The API states that the application allocates the memory and > tells the library to register it. The app then goes on its merry way until > it's done, at which point it tells the library to deregister the memory. > Neither the app nor the API has any provision for the app to be notified > that the memory is no longer pinned and therefore can't be trusted. That > would be considered a critical failure from the app's perspective, so the > kernel would be doing it a favor by killing the process. I'm familiar with MPI 1.0 and 2.0, but I haven't been following the development of modern messaging APIs, so I might not make sense here... Assuming that the app calls into the library on a fairly regular basis, you could implement a fast-path/slow-path scheme where the library normally operates in go-fast mode, but if a "unregister" event has occurred, the library falls back to a less performant mode. But now having written that I'm thinking that it's not worth the bother - if you've got a 512P MPP job, it's basically equivalent to job death for one of the nodes to go away in this manner -- even if the process is still running on the node, the fact that you took a giant performance hiccup is unacceptable. Therefore, cluster admins are going to do their darndest to avoid this behavior, so we might as well just kill the job and make it explicit. > >You might want to consider what happens with your communication system > >in a machine running power-saving modes (in the limit, suspend-to-disk). > >Of course most machines with Infiniband adapters aren't running swsusp, > >but it's not inconceivable that blade servers might sleep to lower power > >and cooling costs. > > Any application that registers memory, will in all likelihood be running at > 100% CPU non-stop. The computer is not going to be doing anything else but > whatever that app is trying to do. The application could conceiveable > register gigabytes of RAM, and if even a single page becomes unpinned, the > whole thing is worthless. The application cannot do anything meaningful if > it gets a message saying that some of the memory has become unpinned and > should not be used. > > So the real question is: how important is it to the kernel developers that > Linux support these kinds of enterprise-class applications? While I understand your arguments, this kind of rhetoric is more likely to harden ears than to convince people you're right. I refer you to the "Live Patching Function" thread. *You* need to come up with a solution that looks good to *the community* if you want it merged. In the long run, this process is likely to result in *your* systems working better than if you had just gone off and done your thing. If you have to do something that "tastes bad" to the average l-k hacker, *justify* it by addressing the alternatives and explaining why your solution is the right one. I'm leaning towards agreeing that mlock()-alicious code is the right way to solve this problem, and it's not clear to me what the benefit of adding a new VM_REGISTERED flag would be. Do you guys simply raise RLIMIT_MEMLOCK to allow apps to lock their pages? Or are you doing something more nasty? (Oh, I see that Libor has contributed to the other branch of this thread... off to read...) -andy ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-21 19:56 ` Andy Isaacson @ 2005-04-21 20:07 ` Timur Tabi 2005-04-21 20:12 ` Chris Wright 2005-04-22 6:14 ` Greg KH 0 siblings, 2 replies; 59+ messages in thread From: Timur Tabi @ 2005-04-21 20:07 UTC (permalink / raw) To: Andy Isaacson Cc: Troy Benjegerdes, Bernhard Fischer, Arjan van de Ven, linux-kernel, openib-general Andy Isaacson wrote: > I'm familiar with MPI 1.0 and 2.0, but I haven't been following the > development of modern messaging APIs, so I might not make sense here... > > Assuming that the app calls into the library on a fairly regular basis, Not really. The whole point is to have the adapter DMA the data directly from memory to the network. That's why it's called RDMA - remote DMA. > Therefore, cluster admins are going to do their > darndest to avoid this behavior, so we might as well just kill the job > and make it explicit. Yes, and if it turns out that the same MPI application dies on Linux but not on Solaris because Linux doesn't really care if the memory stays pinned, then we're going to see a lot of MPI customers transitioning away from Linux. > *You* need to come up with a solution that looks good to *the community* > if you want it merged. True, but I'm not going to waste my time adding this support if the consensus I get from the kernel developers that they don't want Linux to behave this way. > Do you guys simply raise RLIMIT_MEMLOCK to allow apps to lock their > pages? Or are you doing something more nasty? A little more nasty. I raise RLIMIT_MEMLOCK in the driver to "unlimited" and also set cap_raise(IPC_LOCK). I do this because I need to support all 2.4 and 2.6 kernel versions with the same driver, but only 2.6.10 and later have any support for non-root mlock(). If and when our driver is submitted to the official kernel, that nastiness will be removed of course. -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-21 20:07 ` Timur Tabi @ 2005-04-21 20:12 ` Chris Wright 2005-04-21 20:14 ` Timur Tabi 2005-04-22 6:14 ` Greg KH 1 sibling, 1 reply; 59+ messages in thread From: Chris Wright @ 2005-04-21 20:12 UTC (permalink / raw) To: Timur Tabi Cc: Andy Isaacson, Troy Benjegerdes, Bernhard Fischer, Arjan van de Ven, linux-kernel, openib-general * Timur Tabi (timur.tabi@ammasso.com) wrote: > Andy Isaacson wrote: > >Do you guys simply raise RLIMIT_MEMLOCK to allow apps to lock their > >pages? Or are you doing something more nasty? > > A little more nasty. I raise RLIMIT_MEMLOCK in the driver to "unlimited" > and also set cap_raise(IPC_LOCK). I do this because I need to support all > 2.4 and 2.6 kernel versions with the same driver, but only 2.6.10 and later > have any support for non-root mlock(). FYI, that will not work on all 2.6 kernels. Specifically anything that's not using capabilities. thanks, -chris -- Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-21 20:12 ` Chris Wright @ 2005-04-21 20:14 ` Timur Tabi 2005-04-21 20:25 ` Chris Wright 0 siblings, 1 reply; 59+ messages in thread From: Timur Tabi @ 2005-04-21 20:14 UTC (permalink / raw) To: Chris Wright Cc: Andy Isaacson, Troy Benjegerdes, Bernhard Fischer, Arjan van de Ven, linux-kernel, openib-general Chris Wright wrote: > FYI, that will not work on all 2.6 kernels. Specifically anything that's > not using capabilities. It works with every kernel I've tried. I'm sure there are plenty of kernel configuration options that will break our driver. But as long as all the distros our customers use work, as well as reasonably-configured custom kernels, we're happy. -- Timur Tabi Staff Software Engineer timur.tabi@ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-21 20:14 ` Timur Tabi @ 2005-04-21 20:25 ` Chris Wright 2005-04-21 20:30 ` Arjan van de Ven 0 siblings, 1 reply; 59+ messages in thread From: Chris Wright @ 2005-04-21 20:25 UTC (permalink / raw) To: Timur Tabi Cc: Chris Wright, Andy Isaacson, Troy Benjegerdes, Bernhard Fischer, Arjan van de Ven, linux-kernel, openib-general * Timur Tabi (timur.tabi@ammasso.com) wrote: > It works with every kernel I've tried. I'm sure there are plenty of kernel > configuration options that will break our driver. But as long as all the > distros our customers use work, as well as reasonably-configured custom > kernels, we're happy. > Hey, if you're happy (and, as you said, you don't intend to merge that bit), I'm happy ;-) thanks, -chris -- Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-21 20:25 ` Chris Wright @ 2005-04-21 20:30 ` Arjan van de Ven 0 siblings, 0 replies; 59+ messages in thread From: Arjan van de Ven @ 2005-04-21 20:30 UTC (permalink / raw) To: Chris Wright Cc: Timur Tabi, Andy Isaacson, Troy Benjegerdes, Bernhard Fischer, linux-kernel, openib-general On Thu, 2005-04-21 at 13:25 -0700, Chris Wright wrote: > * Timur Tabi (timur.tabi@ammasso.com) wrote: > > It works with every kernel I've tried. I'm sure there are plenty of kernel > > configuration options that will break our driver. But as long as all the > > distros our customers use work, as well as reasonably-configured custom > > kernels, we're happy. > > > > Hey, if you're happy (and, as you said, you don't intend to merge that > bit), I'm happy ;-) yeah... drivers giving unprivileged processes more privs belong on bugtraq though, not in the core kernel :) ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation 2005-04-21 20:07 ` Timur Tabi 2005-04-21 20:12 ` Chris Wright @ 2005-04-22 6:14 ` Greg KH 1 sibling, 0 replies; 59+ messages in thread From: Greg KH @ 2005-04-22 6:14 UTC (permalink / raw) To: Timur Tabi Cc: Andy Isaacson, Troy Benjegerdes, Bernhard Fischer, Arjan van de Ven, linux-kernel, openib-general On Thu, Apr 21, 2005 at 03:07:42PM -0500, Timur Tabi wrote: > >*You* need to come up with a solution that looks good to *the community* > >if you want it merged. > > True, but I'm not going to waste my time adding this support if the > consensus I get from the kernel developers that they don't want Linux to > behave this way. I think we have been giving you that consensus from the very beginning :) The very fact that you tried to trot out the "enterprise" card should have raised a huge flag... thanks, greg k-h ^ permalink raw reply [flat|nested] 59+ messages in thread
end of thread, other threads:[~2005-05-11 23:07 UTC | newest] Thread overview: 59+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <3VAeQ-1To-7@gated-at.bofh.it> [not found] ` <3VNYt-4M4-15@gated-at.bofh.it> 2005-04-22 13:10 ` [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation Bodo Eggert <harvested.in.lkml@posting.7eggert.dyndns.org> 2005-04-22 17:01 ` [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbsimplementation Fab Tillier 2005-04-22 22:01 ` Bodo Eggert 2005-04-04 22:09 [PATCH][RFC][0/4] InfiniBand userspace verbs implementation Roland Dreier 2005-04-11 14:22 ` Troy Benjegerdes 2005-04-11 15:34 ` Roland Dreier 2005-04-11 16:33 ` Troy Benjegerdes 2005-04-11 16:56 ` Roland Dreier 2005-04-11 18:01 ` Troy Benjegerdes 2005-04-11 18:03 ` Roland Dreier 2005-04-12 0:13 ` Andrew Morton 2005-04-13 1:04 ` [openib-general] " Libor Michalek 2005-04-18 17:15 ` Timur Tabi 2005-04-26 3:31 ` Libor Michalek 2005-05-04 18:27 ` Timur Tabi 2005-05-05 18:48 ` Timur Tabi 2005-05-06 23:08 ` Timur Tabi 2005-05-07 13:18 ` Hugh Dickins 2005-05-07 14:45 ` Timur Tabi 2005-05-07 16:30 ` Hugh Dickins 2005-05-11 20:12 ` William Jordan 2005-05-11 20:42 ` Hugh Dickins 2005-05-11 22:52 ` Andrea Arcangeli 2005-05-11 22:49 ` Andrea Arcangeli 2005-05-11 22:53 ` Timur Tabi 2005-05-11 23:05 ` Andrea Arcangeli 2005-05-05 23:34 ` Libor Michalek 2005-04-18 16:22 ` Timur Tabi 2005-04-18 16:43 ` Christoph Hellwig 2005-04-18 16:45 ` Timur Tabi 2005-04-24 2:44 ` Andrew Morton 2005-04-24 14:23 ` Timur Tabi 2005-04-25 13:15 ` Roland Dreier 2005-04-25 20:54 ` Andrew Morton 2005-04-25 21:12 ` Roland Dreier 2005-04-25 22:14 ` Andrew Morton 2005-04-25 22:21 ` Timur Tabi 2005-04-25 22:32 ` Andrew Morton 2005-04-25 23:58 ` Roland Dreier 2005-04-26 0:11 ` Andrew Morton 2005-04-26 0:23 ` Roland Dreier 2005-04-26 0:37 ` Andrew Morton 2005-04-26 15:31 ` Roland Dreier 2005-04-26 15:42 ` [openib-general] " Libor Michalek 2005-04-26 15:49 ` Roland Dreier 2005-04-26 19:28 ` Andrew Morton 2005-04-26 20:14 ` Roland Dreier 2005-04-26 20:18 ` Timur Tabi 2005-04-26 20:37 ` Andrew Morton 2005-04-29 14:26 ` Bill Jordan 2005-04-29 15:56 ` Caitlin Bestler 2005-04-29 17:04 ` Libor Michalek 2005-04-30 0:31 ` Caitlin Bestler 2005-05-03 18:43 ` Andy Isaacson 2005-05-03 19:04 ` Caitlin Bestler 2005-05-04 18:22 ` William Jordan 2005-05-05 1:27 ` Rik van Riel 2005-05-05 1:57 ` Andy Isaacson 2005-04-26 20:32 ` Andrew Morton 2005-04-26 21:23 ` Roland Dreier 2005-04-27 0:05 ` Andrew Morton 2005-04-27 2:13 ` Roland Dreier 2005-04-27 3:21 ` Caitlin Bestler 2005-04-27 3:15 ` Caitlin Bestler 2005-04-26 2:03 ` IWAMOTO Toshihiro 2005-04-26 2:26 ` [openib-general] " Stephen Langdon 2005-04-25 22:23 ` Timur Tabi 2005-04-25 22:35 ` Andrew Morton 2005-04-25 23:17 ` [openib-general] " Libor Michalek 2005-04-25 23:24 ` Andrew Morton 2005-04-25 23:37 ` Caitlin Bestler 2005-04-26 0:10 ` Andrew Morton 2005-04-26 3:55 ` Libor Michalek 2005-04-26 0:02 ` Roland Dreier 2005-04-26 6:12 ` Christoph Hellwig 2005-04-26 13:45 ` [openib-general] " Caitlin Bestler 2005-04-18 16:09 ` Timur Tabi 2005-04-18 16:16 ` Arjan van de Ven 2005-04-18 16:25 ` Timur Tabi 2005-04-18 19:40 ` Arjan van de Ven 2005-04-18 20:07 ` [openib-general] " Bernhard Fischer 2005-04-21 2:17 ` Troy Benjegerdes 2005-04-21 3:07 ` Timur Tabi 2005-04-21 17:38 ` Andy Isaacson 2005-04-21 18:39 ` Timur Tabi 2005-04-21 19:56 ` Andy Isaacson 2005-04-21 20:07 ` Timur Tabi 2005-04-21 20:12 ` Chris Wright 2005-04-21 20:14 ` Timur Tabi 2005-04-21 20:25 ` Chris Wright 2005-04-21 20:30 ` Arjan van de Ven 2005-04-22 6:14 ` Greg KH
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).