From: Andrea Arcangeli <andrea@suse.de>
To: Martin Konold <martin.konold@erfrakon.de>
Cc: Luca Veraldi <luca.veraldi@katamail.com>, linux-kernel@vger.kernel.org
Subject: Re: Efficient IPC mechanism on Linux
Date: Wed, 10 Sep 2003 20:01:28 +0200 [thread overview]
Message-ID: <20030910180128.GP21086@dualathlon.random> (raw)
In-Reply-To: <200309101939.17967.martin.konold@erfrakon.de>
On Wed, Sep 10, 2003 at 07:39:17PM +0200, Martin Konold wrote:
> Am Wednesday 10 September 2003 06:59 pm schrieb Andrea Arcangeli:
>
> Hi,
>
> > design that I'm suggesting. Obviously lots of apps are already using
> > this design and there's no userspace API simply because that's not
> > needed.
>
> HPC people have investigated High performance IPC many times basically it
> boils down to:
>
> 1. Userspace is much more efficient than kernel space. So efficient
> implementions avoid kernel space even for message transfers over networks
> (e.g. DMA directly to userspace).
>
> 2. The optimal protocol to use and the number of copies to do is depending on
> the message size.
>
> Small messages are most efficiently handled with a single/dual copy (short
> protocol / eager protocol) and large messages are more efficient with
> single/zero copy techniques (get protocol) depending if a network is involved
> or not.
>
> Traditionally in a networked environment single copy means PIO and zero copy
> means DMA when using network hardware.
>
> The idea is while DMA has much higher bandwidth than PIO DMA is more expensive
> to initiate than PIO. So DMA is only useful for large messages.
agreed.
>
> In the local SMP case there do exist userspace APIs like MPI which can do
btw, so far we were only discussing IPC in a local box (UP or SMP or
NUMA) w/o networking involved. Luca's currnet implementation as well was
only working locally.
> single copy Message passing at memory speed in pure userspace since many
> years.
>
> The following PDF documents a measurement where the communication between two
> processes running on different CPUs in an SMP system is exactly the memcpy
> bandwidth for large messages using a single copy get protocol.
>
> http://ipdps.eece.unm.edu/1999/pc-now/takahash.pdf
>
> Numbers from a Dual P-II-333, Intel 440LX (100MB/s memcpy)
>
> eager get
> min. Latency µs 8.62 9.98
> max Bandwidth MB/s 48.03 100.02
> half bandwith point KB 0.3 2.5
>
> You can easily see that the eager has better latency for very short messages
> and that the get has a max bandwidth beeing equivalent of a memcpy (single
> copy).
>
> True zero copy has unlimited (sigh!) bandwidth within an SMP and does not
> really make sense in contrast to a network.
if you can avoid to enter kernel, you'd better do that, because entering
kernel will take much more time than the copy itself.
with the shm/futex approch you can also have a ring buffer to handle
parallelism better while it's at the same time zerocopy and enterely
userspace based in the best case (thought that's not the common case).
thanks,
Andrea
/*
* If you refuse to depend on closed software for a critical
* part of your business, these links may be useful:
*
* rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.5/
* rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.4/
* http://www.cobite.com/cvsps/
*
* svn://svn.kernel.org/linux-2.6/trunk
* svn://svn.kernel.org/linux-2.4/trunk
*/
next prev parent reply other threads:[~2003-09-10 17:59 UTC|newest]
Thread overview: 74+ messages / expand[flat|nested] mbox.gz Atom feed top
2003-09-09 17:30 Efficient IPC mechanism on Linux Luca Veraldi
2003-09-09 21:17 ` Alan Cox
2003-09-09 21:57 ` Luca Veraldi
2003-09-09 23:11 ` Alan Cox
2003-09-10 9:04 ` Luca Veraldi
2003-09-10 12:56 ` Alan Cox
[not found] ` <20030909175821.GL16080@Synopsys.COM>
[not found] ` <001d01c37703$8edc10e0$36af7450@wssupremo>
[not found] ` <20030910064508.GA25795@Synopsys.COM>
2003-09-10 9:18 ` Luca Veraldi
2003-09-10 9:23 ` Arjan van de Ven
2003-09-10 9:40 ` Luca Veraldi
2003-09-10 9:44 ` Arjan van de Ven
2003-09-10 10:09 ` Luca Veraldi
2003-09-10 10:14 ` Arjan van de Ven
2003-09-10 10:25 ` Luca Veraldi
2003-09-12 18:41 ` Timothy Miller
2003-09-12 19:05 ` Luca Veraldi
2003-09-12 22:37 ` Alan Cox
2003-09-10 12:50 ` Alan Cox
2003-09-10 19:16 ` Shawn
2003-09-10 20:05 ` Rik van Riel
2003-09-17 9:52 ` Rik's list of CS challenges Terje Eggestad
2003-09-17 13:40 ` Alan Cox
2003-09-18 8:26 ` Helge Hafting
2003-09-10 12:47 ` Efficient IPC mechanism on Linux Alan Cox
2003-09-10 13:56 ` Luca Veraldi
2003-09-10 15:59 ` Alan Cox
2003-09-10 9:52 ` Jamie Lokier
2003-09-10 10:07 ` Arjan van de Ven
2003-09-10 10:17 ` Luca Veraldi
2003-09-10 10:37 ` Jamie Lokier
2003-09-10 10:41 ` Arjan van de Ven
2003-09-10 10:54 ` Luca Veraldi
2003-09-10 10:54 ` Arjan van de Ven
2003-09-10 11:16 ` Nick Piggin
2003-09-10 11:30 ` Luca Veraldi
2003-09-10 11:44 ` Nick Piggin
2003-09-10 12:14 ` Luca Veraldi
2003-09-10 12:42 ` Alan Cox
2003-09-10 10:11 ` Luca Veraldi
2003-09-10 19:24 ` Pavel Machek
2003-09-10 19:40 ` Jamie Lokier
2003-09-10 21:35 ` Pavel Machek
2003-09-10 22:06 ` Jamie Lokier
2003-09-10 11:52 ` Alex Riesen
2003-09-10 12:14 ` Luca Veraldi
2003-09-10 12:11 ` Alex Riesen
2003-09-10 12:29 ` Luca Veraldi
2003-09-10 12:28 ` Alex Riesen
2003-09-10 12:36 ` Luca Veraldi
2003-09-10 12:36 ` Alex Riesen
2003-09-10 13:33 ` Gábor Lénárt
2003-09-10 14:04 ` Luca Veraldi
2003-09-10 14:21 ` Stewart Smith
2003-09-10 14:39 ` Luca Veraldi
2003-09-10 16:59 ` Andrea Arcangeli
2003-09-10 17:05 ` Andrea Arcangeli
2003-09-10 17:21 ` Luca Veraldi
2003-09-10 17:41 ` Andrea Arcangeli
2003-09-10 17:39 ` Martin Konold
2003-09-10 18:01 ` Andrea Arcangeli [this message]
2003-09-10 18:05 ` Martin Konold
2003-09-10 18:31 ` Chris Friesen
2003-09-10 18:08 ` Inappropriate signatures Larry McVoy
2003-09-10 18:52 ` Jamie Lokier
2003-09-10 19:54 ` rsync head? [was inappropriate signatures] Joe Perches
2003-09-13 15:39 ` Inappropriate signatures Pavel Machek
2003-09-13 16:49 ` Larry McVoy
2003-09-09 18:59 Efficient IPC mechanism on Linux Luca Veraldi
2003-09-09 22:15 Luca Veraldi
[not found] <F71B37536F3B3D4FA521FEC7FCA17933164A@twinsrv.twinox.se>
2003-09-10 10:36 ` Luca Veraldi
[not found] <E19x3el-0002Fc-Rj@phoenix.hadiko.de>
2003-09-10 12:16 ` Luca Veraldi
2003-09-10 14:53 ` Larry McVoy
[not found] <fa.h06p421.1s00ojt@ifi.uio.no>
[not found] ` <fa.gc37hsp.34id89@ifi.uio.no>
[not found] ` <E19x47V-0002JG-J8@phoenix.hadiko.de>
2003-09-10 12:45 ` Luca Veraldi
[not found] <u9j3.1VB.27@gated-at.bofh.it>
[not found] ` <u9j3.1VB.29@gated-at.bofh.it>
[not found] ` <u9j3.1VB.31@gated-at.bofh.it>
[not found] ` <u9j3.1VB.25@gated-at.bofh.it>
[not found] ` <ubNY.5Ma.19@gated-at.bofh.it>
[not found] ` <uc79.6lg.13@gated-at.bofh.it>
[not found] ` <uc7d.6lg.23@gated-at.bofh.it>
[not found] ` <uch0.6zx.17@gated-at.bofh.it>
[not found] ` <ucqs.6NC.3@gated-at.bofh.it>
[not found] ` <ucqy.6NC.19@gated-at.bofh.it>
[not found] ` <udmB.8eZ.15@gated-at.bofh.it>
[not found] ` <udPF.BD.11@gated-at.bofh.it>
[not found] ` <3F5F37CD.6060808@softhome.net>
2003-09-10 15:28 ` Luca Veraldi
2003-09-10 18:41 Manfred Spraul
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20030910180128.GP21086@dualathlon.random \
--to=andrea@suse.de \
--cc=linux-kernel@vger.kernel.org \
--cc=luca.veraldi@katamail.com \
--cc=martin.konold@erfrakon.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).