linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Christoph Lameter <clameter@sgi.com>
To: Robin Holt <holt@sgi.com>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>,
	"Luck, Tony" <tony.luck@intel.com>, Adam Litke <agl@us.ibm.com>,
	linux-ia64@vger.kernel.org, torvalds@osdl.org,
	linux-mm@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: Increase page fault rate by prezeroing V1 [3/3]: Altix SN2 BTE Zeroing
Date: Wed, 22 Dec 2004 11:56:47 -0800 (PST)	[thread overview]
Message-ID: <Pine.LNX.4.58.0412221154340.9186@schroedinger.engr.sgi.com> (raw)
In-Reply-To: <20041222124647.GB1813@lnx-holt.americas.sgi.com>

I have done some additional tests with a 128 cpu SMP machine and it shows
that the bte slows things down during memory benchmarking by about 10-20%
although its causing less load when the system is not under high stress.

So its not always win and I may drop bte support in a future version. Can
we talk off list about this since this is mostly an SGI thing?

On Wed, 22 Dec 2004, Robin Holt wrote:

> We still need to talk.  This is a much smaller patch, which I like.  The
> problem I see in my 30 second review is you are doing things per-cpu
> when they really need to be done per-node.  It is very likely that
> there will be M-Bricks in the system (cranberry2 has one if you want
> to test your code out there or you can take any altix and disable the
> cpus on a C-Brick).  With M-Bricks, you will essentially limit
> yourself to one zero operation per controlling node instead of one
> per node.
>
> I think the easy answer is to not have the structure allocated
> within bte_copy(), but rather within bte_start_zero and passed
> in as the notification address.
>
> Give me a call sometime today (Wed. I am in the office from about
> 10:00 CDT until around 4:00 CDT)  Maybe we can get this straightened
> out quickly.  If you are not calling from the office, email me with
> other arrangements.
>
> Thanks,
> Robin
>
> On Tue, Dec 21, 2004 at 11:57:57AM -0800, Christoph Lameter wrote:
> > o Use the Block Transfer Engine in the Altix SN2 SHub for background zeroing
> >
> > Index: linux-2.6.9/arch/ia64/sn/kernel/bte.c
> > ===================================================================
> > --- linux-2.6.9.orig/arch/ia64/sn/kernel/bte.c	2004-12-17 14:40:10.000000000 -0800
> > +++ linux-2.6.9/arch/ia64/sn/kernel/bte.c	2004-12-21 11:03:49.000000000 -0800
> > @@ -4,6 +4,8 @@
> >   * for more details.
> >   *
> >   * Copyright (c) 2000-2003 Silicon Graphics, Inc.  All Rights Reserved.
> > + *
> > + * Support for zeroing pages, Christoph Lameter, SGI, December 2004.
> >   */
> >
> >  #include <linux/config.h>
> > @@ -20,6 +22,8 @@
> >  #include <linux/bootmem.h>
> >  #include <linux/string.h>
> >  #include <linux/sched.h>
> > +#include <linux/mm.h>
> > +#include <linux/scrub.h>
> >
> >  #include <asm/sn/bte.h>
> >
> > @@ -30,7 +34,11 @@
> >  /* two interfaces on two btes */
> >  #define MAX_INTERFACES_TO_TRY		4
> >
> > -static struct bteinfo_s *bte_if_on_node(nasid_t nasid, int interface)
> > +DEFINE_PER_CPU(u64 *, bte_zero_notify);
> > +
> > +#define bte_zero_notify __get_cpu_var(bte_zero_notify)
> > +
> > +static inline struct bteinfo_s *bte_if_on_node(nasid_t nasid, int interface)
> >  {
> >  	nodepda_t *tmp_nodepda;
> >
> > @@ -132,7 +140,6 @@
> >  			if (bte == NULL) {
> >  				continue;
> >  			}
> > -
> >  			if (spin_trylock(&bte->spinlock)) {
> >  				if (!(*bte->most_rcnt_na & BTE_WORD_AVAILABLE) ||
> >  				    (BTE_LNSTAT_LOAD(bte) & BTE_ACTIVE)) {
> > @@ -157,7 +164,7 @@
> >  		}
> >  	} while (1);
> >
> > -	if (notification == NULL) {
> > +	if (notification == NULL || (mode & BTE_NOTIFY_AND_GET_POINTER)) {
> >  		/* User does not want to be notified. */
> >  		bte->most_rcnt_na = &bte->notify;
> >  	} else {
> > @@ -192,6 +199,8 @@
> >
> >  	itc_end = ia64_get_itc() + (40000000 * local_cpu_data->cyc_per_usec);
> >
> > +	if (mode & BTE_NOTIFY_AND_GET_POINTER)
> > +		 *(u64 volatile **)(notification) = &bte->notify;
> >  	spin_unlock_irqrestore(&bte->spinlock, irq_flags);
> >
> >  	if (notification != NULL) {
> > @@ -449,5 +458,31 @@
> >  		mynodepda->bte_if[i].cleanup_active = 0;
> >  		mynodepda->bte_if[i].bh_error = 0;
> >  	}
> > +}
> > +
> > +static int bte_check_bzero(void)
> > +{
> > +	return *bte_zero_notify != BTE_WORD_BUSY;
> > +}
> > +
> > +static int bte_start_bzero(void *p, unsigned long len)
> > +{
> > +	/* Check limitations.
> > +		1. System must be running (weird things happen during bootup)
> > +		2. Size >64KB. Smaller requests cause too much bte traffic
> > +	 */
> > +	if (len >= BTE_MAX_XFER || len < 60000 || system_state != SYSTEM_RUNNING)
> > +		return EINVAL;
> > +
> > +	return bte_zero(ia64_tpa(p), len, BTE_NOTIFY_AND_GET_POINTER, &bte_zero_notify);
> > +}
> > +
> > +static struct zero_driver bte_bzero = {
> > +	.start = bte_start_bzero,
> > +	.check = bte_check_bzero,
> > +	.rate = 500000000		/* 500 MB /sec */
> > +};
> >
> > +void sn_bte_bzero_init(void) {
> > +	register_zero_driver(&bte_bzero);
> >  }
> > Index: linux-2.6.9/arch/ia64/sn/kernel/setup.c
> > ===================================================================
> > --- linux-2.6.9.orig/arch/ia64/sn/kernel/setup.c	2004-12-17 14:40:10.000000000 -0800
> > +++ linux-2.6.9/arch/ia64/sn/kernel/setup.c	2004-12-21 11:02:35.000000000 -0800
> > @@ -243,6 +243,7 @@
> >  	int pxm;
> >  	int major = sn_sal_rev_major(), minor = sn_sal_rev_minor();
> >  	extern void sn_cpu_init(void);
> > +	extern void sn_bte_bzero_init(void);
> >
> >  	/*
> >  	 * If the generic code has enabled vga console support - lets
> > @@ -333,6 +334,7 @@
> >  	screen_info = sn_screen_info;
> >
> >  	sn_timer_init();
> > +	sn_bte_bzero_init();
> >  }
> >
> >  /**
> > Index: linux-2.6.9/include/asm-ia64/sn/bte.h
> > ===================================================================
> > --- linux-2.6.9.orig/include/asm-ia64/sn/bte.h	2004-12-17 14:40:16.000000000 -0800
> > +++ linux-2.6.9/include/asm-ia64/sn/bte.h	2004-12-21 11:02:35.000000000 -0800
> > @@ -48,6 +48,8 @@
> >  #define BTE_ZERO_FILL (BTE_NOTIFY | IBCT_ZFIL_MODE)
> >  /* Use a reserved bit to let the caller specify a wait for any BTE */
> >  #define BTE_WACQUIRE (0x4000)
> > +/* Return the pointer to the notification cacheline to the user */
> > +#define BTE_NOTIFY_AND_GET_POINTER (0x8000)
> >  /* Use the BTE on the node with the destination memory */
> >  #define BTE_USE_DEST (BTE_WACQUIRE << 1)
> >  /* Use any available BTE interface on any node for the transfer */
>

  reply	other threads:[~2004-12-22 19:58 UTC|newest]

Thread overview: 87+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <B8E391BBE9FE384DAA4C5C003888BE6F02900FBD@scsmsx401.amr.corp.intel.com>
     [not found] ` <41C20E3E.3070209@yahoo.com.au>
2004-12-21 19:55   ` Increase page fault rate by prezeroing V1 [0/3]: Overview Christoph Lameter
2004-12-21 19:56     ` Increase page fault rate by prezeroing V1 [1/3]: Introduce __GFP_ZERO Christoph Lameter
2004-12-21 19:57     ` Increase page fault rate by prezeroing V1 [2/3]: zeroing and scrubd Christoph Lameter
2005-01-01  2:22       ` Nick Piggin
2005-01-01  2:55         ` pmarques
2004-12-21 19:57     ` Increase page fault rate by prezeroing V1 [3/3]: Altix SN2 BTE Zeroing Christoph Lameter
2004-12-22 12:46       ` Robin Holt
2004-12-22 19:56         ` Christoph Lameter [this message]
2004-12-23 19:29     ` Prezeroing V2 [0/3]: Why and When it works Christoph Lameter
2004-12-23 19:33       ` Prezeroing V2 [1/4]: __GFP_ZERO / clear_page() removal Christoph Lameter
2004-12-23 19:33         ` Prezeroing V2 [2/4]: add second parameter to clear_page() for all arches Christoph Lameter
2004-12-24  8:33           ` Pavel Machek
2004-12-24 16:18             ` Christoph Lameter
2004-12-24 16:27               ` Pavel Machek
2004-12-24 17:02                 ` David S. Miller
2004-12-24 17:05           ` David S. Miller
2004-12-27 22:48             ` David S. Miller
2005-01-03 17:52             ` Christoph Lameter
2005-01-01 10:24           ` Geert Uytterhoeven
2005-01-04 23:12             ` Prezeroing V3 [0/4]: Discussion and i386 performance tests Christoph Lameter
2005-01-04 23:13               ` Prezeroing V3 [1/4]: Allow request for zeroed memory Christoph Lameter
2005-01-04 23:45                 ` Dave Hansen
2005-01-05  1:16                   ` Christoph Lameter
2005-01-05  1:26                     ` Linus Torvalds
2005-01-05 23:11                       ` Christoph Lameter
2005-01-05  0:34                 ` Linus Torvalds
2005-01-05  0:47                   ` Andrew Morton
2005-01-05  1:15                     ` Christoph Lameter
2005-01-08 21:12                 ` Hugh Dickins
2005-01-08 21:56                   ` David S. Miller
2005-01-21 20:09                     ` alloc_zeroed_user_highpage to fix the clear_user_highpage issue Christoph Lameter
2005-02-09  9:58                       ` [Patch] Fix oops in alloc_zeroed_user_highpage() when page is NULL Michael Ellerman
2005-02-10  0:38                         ` Christoph Lameter
2005-01-21 20:12                     ` Extend clear_page by an order parameter Christoph Lameter
2005-01-21 22:29                       ` Paul Mackerras
2005-01-21 23:48                         ` Christoph Lameter
2005-01-22  0:35                           ` Paul Mackerras
2005-01-22  0:43                             ` Andrew Morton
2005-01-22  1:08                               ` Paul Mackerras
2005-01-22  1:20                               ` Roman Zippel
2005-01-22  1:25                               ` Paul Mackerras
2005-01-22  1:54                                 ` Christoph Lameter
2005-01-22  2:53                                   ` Paul Mackerras
2005-01-23  7:45                       ` Andrew Morton
2005-01-24 16:37                         ` Christoph Lameter
2005-01-24 20:23                           ` David S. Miller
2005-01-24 20:33                             ` Christoph Lameter
2005-01-21 20:15                     ` A scrub daemon (prezeroing) Christoph Lameter
2005-01-10 17:16                   ` Prezeroing V3 [1/4]: Allow request for zeroed memory Christoph Lameter
2005-01-10 18:13                     ` Linus Torvalds
2005-01-10 20:17                       ` Christoph Lameter
2005-01-10 23:53                       ` Prezeroing V4 [0/4]: Overview Christoph Lameter
2005-01-10 23:54                         ` Prezeroing V4 [1/4]: Arch specific page zeroing during page fault Christoph Lameter
2005-01-11  0:41                           ` Chris Wright
2005-01-11  0:46                             ` Christoph Lameter
2005-01-11  0:49                               ` Chris Wright
2005-01-10 23:55                         ` Prezeroing V4 [2/4]: Zeroing implementation Christoph Lameter
2005-01-10 23:55                         ` Prezeroing V4 [3/4]: Altix SN2 BTE zero driver Christoph Lameter
2005-01-10 23:56                         ` Prezeroing V4 [4/4]: Extend clear_page to take an order parameter Christoph Lameter
2005-01-04 23:14               ` Prezeroing V3 [2/4]: Extension of " Christoph Lameter
2005-01-05 23:25                 ` Christoph Lameter
2005-01-06 13:52                   ` Andi Kleen
2005-01-06 17:47                     ` Christoph Lameter
2005-01-04 23:15               ` Prezeroing V3 [3/4]: Page zeroing through kscrubd Christoph Lameter
2005-01-04 23:16               ` Prezeroing V3 [4/4]: Driver for hardware zeroing on Altix Christoph Lameter
2005-01-05  2:16                 ` Andi Kleen
2005-01-05 16:24                   ` Christoph Lameter
2004-12-23 19:34         ` Prezeroing V2 [3/4]: Add support for ZEROED and NOT_ZEROED free maps Christoph Lameter
2004-12-23 19:35         ` Prezeroing V2 [4/4]: Hardware Zeroing through SGI BTE Christoph Lameter
2004-12-23 20:08         ` Prezeroing V2 [1/4]: __GFP_ZERO / clear_page() removal Brian Gerst
2004-12-24 16:24           ` Christoph Lameter
2004-12-23 19:49       ` Prezeroing V2 [0/3]: Why and When it works Arjan van de Ven
2004-12-23 20:57       ` Matt Mackall
2004-12-23 21:01       ` Paul Mackerras
2004-12-23 21:11       ` Paul Mackerras
2004-12-23 21:37         ` Andrew Morton
2004-12-23 23:00           ` Paul Mackerras
2004-12-23 21:48         ` Linus Torvalds
2004-12-23 22:34           ` Zwane Mwaikambo
2004-12-24  9:14           ` Arjan van de Ven
2004-12-24 18:21             ` Linus Torvalds
2004-12-24 18:57               ` Arjan van de Ven
2004-12-27 22:50               ` David S. Miller
2004-12-28 11:53                 ` Marcelo Tosatti
2004-12-24 16:17           ` Christoph Lameter
2004-12-24 18:31     ` Increase page fault rate by prezeroing V1 [0/3]: Overview Andrea Arcangeli
2005-01-03 17:54       ` Christoph Lameter

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=Pine.LNX.4.58.0412221154340.9186@schroedinger.engr.sgi.com \
    --to=clameter@sgi.com \
    --cc=agl@us.ibm.com \
    --cc=holt@sgi.com \
    --cc=linux-ia64@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@vger.kernel.org \
    --cc=nickpiggin@yahoo.com.au \
    --cc=tony.luck@intel.com \
    --cc=torvalds@osdl.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).