From: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> To: Pavel Machek <pavel-+ZI9xUNit7I@public.gmane.org> Cc: Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, general-ZwoEplunGu1OwGhvXhtEPSCwEArCW2h5@public.gmane.org, akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org, torvalds-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org Subject: Re: [ofa-general] Re: [GIT PULL] please pull ummunotify Date: Mon, 28 Sep 2009 15:40:57 -0600 [thread overview] Message-ID: <20090928214057.GX19540@obsidianresearch.com> (raw) In-Reply-To: <20090928204923.GA1960-I/5MKhXcvmPrBKCeMvbIDA@public.gmane.org> On Mon, Sep 28, 2009 at 10:49:23PM +0200, Pavel Machek wrote: > > > I don't remember seeing discussion of this on lkml. Yes it is in > > > -next... > > > > eg http://lkml.org/lkml/2009/7/31/197 and followups, or search for v2 > > and earlier patches. > Well... it seems little overspecialized. Just modifying libc to > provide hooks you want looks like better solution. That is what MPI people are doing today and their feedback is that it doesn't work - there are a lot of ways to mess with memory and no good choices to hook the raw syscalls and keep sensible performance. The main focus of this is high performance MPI apps, so lower overhead on critical paths like memory allocation is part of the point. It is ment to go hand-in-hand with the specialized RDMA memory pinning interfaces.. > > > Basically it allows app to 'trace itself'? ...with interesting mmap() > > > interface, exporting int to userspace, hoping it behaves atomically...? > > > > Yes, it allows app to trace what the kernel does to memory mappings. I > > don't believe there's any real issue to atomicity of mmap'ed memory, > > since userspace really just tests whether read value is == to old read > > value or not. > > That still needs memory barriers etc.. to ensure reliable operation, > no? No, I don't think so.. The application is expected to provide sequencing of some sort between the memory call (mmap/munmap/brk/etc) and the int check - usually just by running in the same thread, or through some kind of locking scheme. As long as the mmu notifiers run immediately in the same context as the mmap/etc then it should be fine. For example, the most common problem to solve looks like this: x = mmap(...) do RDMA with x [..] mmunmap(x); [..] y = mmap(..); do RDMA with y if by chance x == y things explode. So this API puts the int test directly before 'do RDMA with'. Due to the above kind of argument the net requirement is either to completely synchronously (and with low overhead) hook every mmap/munmap/brk/etc call into the kernel and do the accounting work, or have a very low over head check every time the memory region is about to be used. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: Jason Gunthorpe <jgunthorpe@obsidianresearch.com> To: Pavel Machek <pavel@ucw.cz> Cc: Roland Dreier <rdreier@cisco.com>, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, general@lists.openfabrics.org, akpm@linux-foundation.org, torvalds@linux-foundation.org Subject: Re: [ofa-general] Re: [GIT PULL] please pull ummunotify Date: Mon, 28 Sep 2009 15:40:57 -0600 [thread overview] Message-ID: <20090928214057.GX19540@obsidianresearch.com> (raw) In-Reply-To: <20090928204923.GA1960@elf.ucw.cz> On Mon, Sep 28, 2009 at 10:49:23PM +0200, Pavel Machek wrote: > > > I don't remember seeing discussion of this on lkml. Yes it is in > > > -next... > > > > eg http://lkml.org/lkml/2009/7/31/197 and followups, or search for v2 > > and earlier patches. > Well... it seems little overspecialized. Just modifying libc to > provide hooks you want looks like better solution. That is what MPI people are doing today and their feedback is that it doesn't work - there are a lot of ways to mess with memory and no good choices to hook the raw syscalls and keep sensible performance. The main focus of this is high performance MPI apps, so lower overhead on critical paths like memory allocation is part of the point. It is ment to go hand-in-hand with the specialized RDMA memory pinning interfaces.. > > > Basically it allows app to 'trace itself'? ...with interesting mmap() > > > interface, exporting int to userspace, hoping it behaves atomically...? > > > > Yes, it allows app to trace what the kernel does to memory mappings. I > > don't believe there's any real issue to atomicity of mmap'ed memory, > > since userspace really just tests whether read value is == to old read > > value or not. > > That still needs memory barriers etc.. to ensure reliable operation, > no? No, I don't think so.. The application is expected to provide sequencing of some sort between the memory call (mmap/munmap/brk/etc) and the int check - usually just by running in the same thread, or through some kind of locking scheme. As long as the mmu notifiers run immediately in the same context as the mmap/etc then it should be fine. For example, the most common problem to solve looks like this: x = mmap(...) do RDMA with x [..] mmunmap(x); [..] y = mmap(..); do RDMA with y if by chance x == y things explode. So this API puts the int test directly before 'do RDMA with'. Due to the above kind of argument the net requirement is either to completely synchronously (and with low overhead) hook every mmap/munmap/brk/etc call into the kernel and do the accounting work, or have a very low over head check every time the memory region is about to be used. Jason
next prev parent reply other threads:[~2009-09-28 21:40 UTC|newest] Thread overview: 82+ messages / expand[flat|nested] mbox.gz Atom feed top 2009-09-11 4:38 [GIT PULL] please pull ummunotify Roland Dreier 2009-09-11 4:38 ` Roland Dreier 2009-09-15 11:34 ` Pavel Machek [not found] ` <20090915113434.GF1328-+ZI9xUNit7I@public.gmane.org> 2009-09-15 14:57 ` [ofa-general] " Roland Dreier 2009-09-15 14:57 ` Roland Dreier [not found] ` <ada7hw0gsqz.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> 2009-09-28 20:49 ` Pavel Machek 2009-09-28 20:49 ` Pavel Machek [not found] ` <20090928204923.GA1960-I/5MKhXcvmPrBKCeMvbIDA@public.gmane.org> 2009-09-28 21:40 ` Jason Gunthorpe [this message] 2009-09-28 21:40 ` Jason Gunthorpe [not found] ` <aday6omhz9d.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> 2009-09-11 5:56 ` KOSAKI Motohiro 2009-09-11 5:56 ` KOSAKI Motohiro 2009-09-11 6:03 ` [ofa-general] " Roland Dreier 2009-09-11 6:03 ` Roland Dreier [not found] ` <adatyzahvbm.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> 2009-09-11 6:11 ` KOSAKI Motohiro 2009-09-11 6:11 ` KOSAKI Motohiro [not found] ` <20090911150552.DB68.A69D9226-+CUm20s59erQFUHtdCDX3A@public.gmane.org> 2009-09-11 16:42 ` Gleb Natapov 2009-09-11 16:42 ` Gleb Natapov 2009-09-11 6:15 ` Brice Goglin [not found] ` <4AA9EAF7.5010401-MZpvjPyXg2s@public.gmane.org> 2009-09-11 6:21 ` KOSAKI Motohiro 2009-09-11 6:21 ` KOSAKI Motohiro 2009-09-11 6:22 ` Roland Dreier 2009-09-11 6:22 ` Roland Dreier 2009-09-11 6:40 ` [ofa-general] " Jason Gunthorpe 2009-09-11 6:40 ` Jason Gunthorpe [not found] ` <20090911064019.GZ4973-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2009-09-11 16:58 ` Roland Dreier 2009-09-11 16:58 ` Roland Dreier [not found] ` <adaljklifkt.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> 2009-09-15 7:03 ` KOSAKI Motohiro 2009-09-15 7:03 ` KOSAKI Motohiro 2009-09-15 8:27 ` Roland Dreier 2009-09-15 8:27 ` Roland Dreier 2009-09-15 12:38 ` Jeff Squyres 2009-09-15 12:38 ` Jeff Squyres 2009-09-16 16:30 ` Roland Dreier 2009-09-16 16:30 ` Roland Dreier 2009-09-16 16:40 ` [ofa-general] " Linus Torvalds 2009-09-16 16:40 ` Linus Torvalds 2009-09-17 11:30 ` Peter Zijlstra 2009-09-17 11:30 ` Peter Zijlstra 2009-09-17 14:24 ` [ofa-general] " Roland Dreier [not found] ` <adafxalejiq.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> 2009-09-17 14:32 ` Roland Dreier 2009-09-17 14:32 ` Roland Dreier [not found] ` <adaab0tej5c.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> 2009-09-17 14:49 ` Peter Zijlstra 2009-09-17 14:49 ` Peter Zijlstra 2009-09-17 15:03 ` Roland Dreier 2009-09-17 15:03 ` Roland Dreier [not found] ` <adazl8td35u.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> 2009-09-17 15:22 ` Peter Zijlstra 2009-09-17 15:22 ` Peter Zijlstra 2009-09-17 15:45 ` Roland Dreier 2009-09-17 15:45 ` Roland Dreier [not found] ` <adatyz1d17q.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> 2009-09-18 11:50 ` Ingo Molnar 2009-09-18 11:50 ` Ingo Molnar 2009-09-29 17:13 ` Pavel Machek 2009-09-29 17:13 ` Pavel Machek [not found] ` <20090929171332.GD14405-I/5MKhXcvmPrBKCeMvbIDA@public.gmane.org> 2009-09-30 9:44 ` Ingo Molnar 2009-09-30 9:44 ` Ingo Molnar [not found] ` <20090930094456.GD24621-X9Un+BFzKDI@public.gmane.org> 2009-09-30 16:02 ` Jason Gunthorpe 2009-09-30 16:02 ` Jason Gunthorpe [not found] ` <20090930160232.GZ22310-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2009-10-12 18:19 ` Ingo Molnar 2009-10-12 18:19 ` Ingo Molnar [not found] ` <20091012181944.GF17138-X9Un+BFzKDI@public.gmane.org> 2009-10-12 19:30 ` Jason Gunthorpe 2009-10-12 19:30 ` Jason Gunthorpe 2009-10-12 20:20 ` Ingo Molnar [not found] ` <20091012202046.GA7648-X9Un+BFzKDI@public.gmane.org> 2009-10-13 4:05 ` Jason Gunthorpe 2009-10-13 4:05 ` Jason Gunthorpe 2009-10-13 6:40 ` Ingo Molnar [not found] ` <20091013064006.GC9470-X9Un+BFzKDI@public.gmane.org> 2009-10-13 16:27 ` Jason Gunthorpe 2009-10-13 16:27 ` Jason Gunthorpe 2009-10-13 5:43 ` Brice Goglin [not found] ` <4AD41373.8010108-MZpvjPyXg2s@public.gmane.org> 2009-10-13 6:38 ` Ingo Molnar 2009-10-13 6:38 ` Ingo Molnar 2009-09-30 17:06 ` Roland Dreier 2009-09-30 17:06 ` Roland Dreier 2009-10-02 16:32 ` Roland Dreier 2009-10-02 16:32 ` Roland Dreier [not found] ` <ada3a61rc3j.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> 2009-10-02 20:45 ` Pavel Machek 2009-10-02 20:45 ` Pavel Machek 2009-10-07 22:34 ` Roland Dreier 2009-10-07 22:34 ` Roland Dreier [not found] ` <ada3a5uq1dk.fsf-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> 2009-10-12 17:33 ` Peter Zijlstra 2009-10-12 17:33 ` Peter Zijlstra 2009-09-17 14:43 ` Peter Zijlstra 2009-09-17 14:43 ` Peter Zijlstra
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20090928214057.GX19540@obsidianresearch.com \ --to=jgunthorpe-epgobjl8dl3ta4ec/59zmfatqe2ktcn/@public.gmane.org \ --cc=akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org \ --cc=general-ZwoEplunGu1OwGhvXhtEPSCwEArCW2h5@public.gmane.org \ --cc=linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=pavel-+ZI9xUNit7I@public.gmane.org \ --cc=rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org \ --cc=torvalds-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.