All of lore.kernel.org
 help / color / mirror / Atom feed
From: Mark Rutland <mark.rutland@arm.com>
To: Kees Cook <keescook@chromium.org>
Cc: Pavel Machek <pavel@ucw.cz>, Laura Abbott <labbott@redhat.com>,
	"linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Heiko Carstens <heiko.carstens@de.ibm.com>,
	"James E.J. Bottomley" <jejb@parisc-linux.org>,
	"H. Peter Anvin" <hpa@zytor.com>,
	"kernel-hardening@lists.openwall.com"
	<kernel-hardening@lists.openwall.com>,
	Rob Herring <robh@kernel.org>, Jessica Yu <jeyu@redhat.com>,
	Jonathan Corbet <corbet@lwn.net>, Helge Deller <deller@gmx.de>,
	"x86@kernel.org" <x86@kernel.org>,
	Russell King <linux@armlinux.org.uk>,
	Ingo Molnar <mingo@redhat.com>, Len Brown <len.brown@intel.com>,
	"linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
	Will Deacon <will.deacon@arm.com>,
	Thomas Gleixner <tglx@linutronix.de>,
Subject: Re: [PATCHv3 2/2] arch: Rename CONFIG_DEBUG_RODATA and CONFIG_DEBUG_MODULE_RONX
Date: Fri, 17 Feb 2017 18:23:13 +0000	[thread overview]
Message-ID: <20170217182313.GB25876@leverpostej> (raw)
In-Reply-To: <CAGXu5jJ4FD=6HLK_21T079JrO569DwWBTeiD18x3WRrYBsaAqw@mail.gmail.com>

On Thu, Feb 16, 2017 at 05:08:20PM -0800, Kees Cook wrote:
> On Thu, Feb 16, 2017 at 2:25 PM, Pavel Machek <pavel@ucw.cz> wrote:
> > Hi!
> >
> >>
> >> -config DEBUG_RODATA
> >> +config STRICT_KERNEL_RWX
> >>       bool "Make kernel text and rodata read-only" if ARCH_OPTIONAL_KERNEL_RWX
> >>       depends on ARCH_HAS_STRICT_KERNEL_RWX
> >>       default !ARCH_OPTIONAL_KERNEL_RWX ||
> >
> > Debug features are expected to have runtime cost, so kconfig help is
> > silent about those. But there are runtime costs, right? It would be
> > nice to mention them in the help text...
> 
> It depends on the architecture. The prior help text for arm said:
> 
>          The tradeoff is that each region is padded to section-size (1MiB)
>          boundaries (because their permissions are different and splitting
>          the 1M pages into 4K ones causes TLB performance problems), which
>          can waste memory.
> 
> parisc (somewhat inaccurately) said:
> 
>          This option may have a slight performance impact because a
>          portion of the kernel code won't be covered by a TLB anymore.
> 
> IIUC, arm64 does what parisc is hinting at: mappings at the end are
> broken down to PAGE_SIZE. On x86, IIUC, there's actually no change to
> TLB performance due to how the mappings are already set up.

On arm64, we split down to page granularity if needed, but use the
largest possible mapping we can (e.g. if we can use a 2M block, we do).

Because of the way we freed the init area, we already couldn't use
larger mappings anyway. Applying the strict permissions didn't come at a
measureable overhead in any real testing.

> I'm not sure the best way to express this in the new help text. Do you
> have some suggestions on wording? Personally, I don't really think
> it's worth mentioning this in Kconfig help, which, in theory, is
> supposed to limit how technical it gets. And I think the performance
> impact is almost entirely negligible compared to the risks addressed.

I also don't see much point in describing some hypothetical architecture
specific overhead here. In most cases this can't be turned off (so there
isn't anything to comapre it to, and hence no cost). Where people want
to turn it off, they already know why they wish to do so.

Thanks,
Mark.

WARNING: multiple messages have this Message-ID (diff)
From: Mark Rutland <mark.rutland@arm.com>
To: Kees Cook <keescook@chromium.org>
Cc: Pavel Machek <pavel@ucw.cz>, Laura Abbott <labbott@redhat.com>,
	"linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Heiko Carstens <heiko.carstens@de.ibm.com>,
	"James E.J. Bottomley" <jejb@parisc-linux.org>,
	"H. Peter Anvin" <hpa@zytor.com>,
	"kernel-hardening@lists.openwall.com" 
	<kernel-hardening@lists.openwall.com>,
	Rob Herring <robh@kernel.org>, Jessica Yu <jeyu@redhat.com>,
	Jonathan Corbet <corbet@lwn.net>, Helge Deller <deller@gmx.de>,
	"x86@kernel.org" <x86@kernel.org>,
	Russell King <linux@armlinux.org.uk>,
	Ingo Molnar <mingo@redhat.com>, Len Brown <len.brown@intel.com>,
	"linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
	Will Deacon <will.deacon@arm.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	"linux-arm-kernel@lists.infradead.org" 
	<linux-arm-kernel@lists.infradead.org>,
	linux-parisc <linux-parisc@vger.kernel.org>,
	Linux PM list <linux-pm@vger.kernel.org>,
	"Rafael J. Wysocki" <rjw@rjwysocki.net>,
	LKML <linux-kernel@vger.kernel.org>,
	Jason Wessel <jason.wessel@windriver.com>,
	Martin Schwidefsky <schwidefsky@de.ibm.com>,
	Robin Murphy <robin.murphy@arm.com>
Subject: Re: [PATCHv3 2/2] arch: Rename CONFIG_DEBUG_RODATA and CONFIG_DEBUG_MODULE_RONX
Date: Fri, 17 Feb 2017 18:23:13 +0000	[thread overview]
Message-ID: <20170217182313.GB25876@leverpostej> (raw)
In-Reply-To: <CAGXu5jJ4FD=6HLK_21T079JrO569DwWBTeiD18x3WRrYBsaAqw@mail.gmail.com>

On Thu, Feb 16, 2017 at 05:08:20PM -0800, Kees Cook wrote:
> On Thu, Feb 16, 2017 at 2:25 PM, Pavel Machek <pavel@ucw.cz> wrote:
> > Hi!
> >
> >>
> >> -config DEBUG_RODATA
> >> +config STRICT_KERNEL_RWX
> >>       bool "Make kernel text and rodata read-only" if ARCH_OPTIONAL_KERNEL_RWX
> >>       depends on ARCH_HAS_STRICT_KERNEL_RWX
> >>       default !ARCH_OPTIONAL_KERNEL_RWX ||
> >
> > Debug features are expected to have runtime cost, so kconfig help is
> > silent about those. But there are runtime costs, right? It would be
> > nice to mention them in the help text...
> 
> It depends on the architecture. The prior help text for arm said:
> 
>          The tradeoff is that each region is padded to section-size (1MiB)
>          boundaries (because their permissions are different and splitting
>          the 1M pages into 4K ones causes TLB performance problems), which
>          can waste memory.
> 
> parisc (somewhat inaccurately) said:
> 
>          This option may have a slight performance impact because a
>          portion of the kernel code won't be covered by a TLB anymore.
> 
> IIUC, arm64 does what parisc is hinting at: mappings at the end are
> broken down to PAGE_SIZE. On x86, IIUC, there's actually no change to
> TLB performance due to how the mappings are already set up.

On arm64, we split down to page granularity if needed, but use the
largest possible mapping we can (e.g. if we can use a 2M block, we do).

Because of the way we freed the init area, we already couldn't use
larger mappings anyway. Applying the strict permissions didn't come at a
measureable overhead in any real testing.

> I'm not sure the best way to express this in the new help text. Do you
> have some suggestions on wording? Personally, I don't really think
> it's worth mentioning this in Kconfig help, which, in theory, is
> supposed to limit how technical it gets. And I think the performance
> impact is almost entirely negligible compared to the risks addressed.

I also don't see much point in describing some hypothetical architecture
specific overhead here. In most cases this can't be turned off (so there
isn't anything to comapre it to, and hence no cost). Where people want
to turn it off, they already know why they wish to do so.

Thanks,
Mark.

WARNING: multiple messages have this Message-ID (diff)
From: Mark Rutland <mark.rutland@arm.com>
To: Kees Cook <keescook@chromium.org>
Cc: Pavel Machek <pavel@ucw.cz>, Laura Abbott <labbott@redhat.com>,
	"linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Heiko Carstens <heiko.carstens@de.ibm.com>,
	"James E.J. Bottomley" <jejb@parisc-linux.org>,
	"H. Peter Anvin" <hpa@zytor.com>,
	"kernel-hardening@lists.openwall.com"
	<kernel-hardening@lists.openwall.com>,
	Rob Herring <robh@kernel.org>, Jessica Yu <jeyu@redhat.com>,
	Jonathan Corbet <corbet@lwn.net>, Helge Deller <deller@gmx.de>,
	"x86@kernel.org" <x86@kernel.org>,
	Russell King <linux@armlinux.org.uk>,
	Ingo Molnar <mingo@redhat.com>, Len Brown <len.brown@intel.com>,
	"linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
	Will Deacon <will.deacon@arm.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>,
	linux-parisc <linux-parisc@vger.kernel.org>,
	Linux PM list <linux-pm@vger.kernel.org>,
	"Rafael J. Wysocki" <rjw@rjwysocki.net>,
	LKML <linux-kernel@vger.kernel.org>,
	Jason Wessel <jason.wessel@windriver.com>,
	Martin Schwidefsky <schwidefsky@de.ibm.com>,
	Robin Murphy <robin.murphy@arm.com>
Subject: [kernel-hardening] Re: [PATCHv3 2/2] arch: Rename CONFIG_DEBUG_RODATA and CONFIG_DEBUG_MODULE_RONX
Date: Fri, 17 Feb 2017 18:23:13 +0000	[thread overview]
Message-ID: <20170217182313.GB25876@leverpostej> (raw)
In-Reply-To: <CAGXu5jJ4FD=6HLK_21T079JrO569DwWBTeiD18x3WRrYBsaAqw@mail.gmail.com>

On Thu, Feb 16, 2017 at 05:08:20PM -0800, Kees Cook wrote:
> On Thu, Feb 16, 2017 at 2:25 PM, Pavel Machek <pavel@ucw.cz> wrote:
> > Hi!
> >
> >>
> >> -config DEBUG_RODATA
> >> +config STRICT_KERNEL_RWX
> >>       bool "Make kernel text and rodata read-only" if ARCH_OPTIONAL_KERNEL_RWX
> >>       depends on ARCH_HAS_STRICT_KERNEL_RWX
> >>       default !ARCH_OPTIONAL_KERNEL_RWX ||
> >
> > Debug features are expected to have runtime cost, so kconfig help is
> > silent about those. But there are runtime costs, right? It would be
> > nice to mention them in the help text...
> 
> It depends on the architecture. The prior help text for arm said:
> 
>          The tradeoff is that each region is padded to section-size (1MiB)
>          boundaries (because their permissions are different and splitting
>          the 1M pages into 4K ones causes TLB performance problems), which
>          can waste memory.
> 
> parisc (somewhat inaccurately) said:
> 
>          This option may have a slight performance impact because a
>          portion of the kernel code won't be covered by a TLB anymore.
> 
> IIUC, arm64 does what parisc is hinting at: mappings at the end are
> broken down to PAGE_SIZE. On x86, IIUC, there's actually no change to
> TLB performance due to how the mappings are already set up.

On arm64, we split down to page granularity if needed, but use the
largest possible mapping we can (e.g. if we can use a 2M block, we do).

Because of the way we freed the init area, we already couldn't use
larger mappings anyway. Applying the strict permissions didn't come at a
measureable overhead in any real testing.

> I'm not sure the best way to express this in the new help text. Do you
> have some suggestions on wording? Personally, I don't really think
> it's worth mentioning this in Kconfig help, which, in theory, is
> supposed to limit how technical it gets. And I think the performance
> impact is almost entirely negligible compared to the risks addressed.

I also don't see much point in describing some hypothetical architecture
specific overhead here. In most cases this can't be turned off (so there
isn't anything to comapre it to, and hence no cost). Where people want
to turn it off, they already know why they wish to do so.

Thanks,
Mark.

WARNING: multiple messages have this Message-ID (diff)
From: Mark Rutland <mark.rutland@arm.com>
To: Kees Cook <keescook@chromium.org>
Cc: Pavel Machek <pavel@ucw.cz>, Laura Abbott <labbott@redhat.com>,
	"linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Heiko Carstens <heiko.carstens@de.ibm.com>,
	"James E.J. Bottomley" <jejb@parisc-linux.org>,
	"H. Peter Anvin" <hpa@zytor.com>,
	"kernel-hardening@lists.openwall.com"
	<kernel-hardening@lists.openwall.com>,
	Rob Herring <robh@kernel.org>, Jessica Yu <jeyu@redhat.com>,
	Jonathan Corbet <corbet@lwn.net>, Helge Deller <deller@gmx.de>,
	"x86@kernel.org" <x86@kernel.org>,
	Russell King <linux@armlinux.org.uk>,
	Ingo Molnar <mingo@redhat.com>, Len Brown <len.brown@intel.com>,
	"linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
	Will Deacon <will.deacon@arm.com>,
	Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [PATCHv3 2/2] arch: Rename CONFIG_DEBUG_RODATA and CONFIG_DEBUG_MODULE_RONX
Date: Fri, 17 Feb 2017 18:23:13 +0000	[thread overview]
Message-ID: <20170217182313.GB25876@leverpostej> (raw)
In-Reply-To: <CAGXu5jJ4FD=6HLK_21T079JrO569DwWBTeiD18x3WRrYBsaAqw@mail.gmail.com>

On Thu, Feb 16, 2017 at 05:08:20PM -0800, Kees Cook wrote:
> On Thu, Feb 16, 2017 at 2:25 PM, Pavel Machek <pavel@ucw.cz> wrote:
> > Hi!
> >
> >>
> >> -config DEBUG_RODATA
> >> +config STRICT_KERNEL_RWX
> >>       bool "Make kernel text and rodata read-only" if ARCH_OPTIONAL_KERNEL_RWX
> >>       depends on ARCH_HAS_STRICT_KERNEL_RWX
> >>       default !ARCH_OPTIONAL_KERNEL_RWX ||
> >
> > Debug features are expected to have runtime cost, so kconfig help is
> > silent about those. But there are runtime costs, right? It would be
> > nice to mention them in the help text...
> 
> It depends on the architecture. The prior help text for arm said:
> 
>          The tradeoff is that each region is padded to section-size (1MiB)
>          boundaries (because their permissions are different and splitting
>          the 1M pages into 4K ones causes TLB performance problems), which
>          can waste memory.
> 
> parisc (somewhat inaccurately) said:
> 
>          This option may have a slight performance impact because a
>          portion of the kernel code won't be covered by a TLB anymore.
> 
> IIUC, arm64 does what parisc is hinting at: mappings at the end are
> broken down to PAGE_SIZE. On x86, IIUC, there's actually no change to
> TLB performance due to how the mappings are already set up.

On arm64, we split down to page granularity if needed, but use the
largest possible mapping we can (e.g. if we can use a 2M block, we do).

Because of the way we freed the init area, we already couldn't use
larger mappings anyway. Applying the strict permissions didn't come at a
measureable overhead in any real testing.

> I'm not sure the best way to express this in the new help text. Do you
> have some suggestions on wording? Personally, I don't really think
> it's worth mentioning this in Kconfig help, which, in theory, is
> supposed to limit how technical it gets. And I think the performance
> impact is almost entirely negligible compared to the risks addressed.

I also don't see much point in describing some hypothetical architecture
specific overhead here. In most cases this can't be turned off (so there
isn't anything to comapre it to, and hence no cost). Where people want
to turn it off, they already know why they wish to do so.

Thanks,
Mark.

WARNING: multiple messages have this Message-ID (diff)
From: mark.rutland@arm.com (Mark Rutland)
To: linux-arm-kernel@lists.infradead.org
Subject: [PATCHv3 2/2] arch: Rename CONFIG_DEBUG_RODATA and CONFIG_DEBUG_MODULE_RONX
Date: Fri, 17 Feb 2017 18:23:13 +0000	[thread overview]
Message-ID: <20170217182313.GB25876@leverpostej> (raw)
In-Reply-To: <CAGXu5jJ4FD=6HLK_21T079JrO569DwWBTeiD18x3WRrYBsaAqw@mail.gmail.com>

On Thu, Feb 16, 2017 at 05:08:20PM -0800, Kees Cook wrote:
> On Thu, Feb 16, 2017 at 2:25 PM, Pavel Machek <pavel@ucw.cz> wrote:
> > Hi!
> >
> >>
> >> -config DEBUG_RODATA
> >> +config STRICT_KERNEL_RWX
> >>       bool "Make kernel text and rodata read-only" if ARCH_OPTIONAL_KERNEL_RWX
> >>       depends on ARCH_HAS_STRICT_KERNEL_RWX
> >>       default !ARCH_OPTIONAL_KERNEL_RWX ||
> >
> > Debug features are expected to have runtime cost, so kconfig help is
> > silent about those. But there are runtime costs, right? It would be
> > nice to mention them in the help text...
> 
> It depends on the architecture. The prior help text for arm said:
> 
>          The tradeoff is that each region is padded to section-size (1MiB)
>          boundaries (because their permissions are different and splitting
>          the 1M pages into 4K ones causes TLB performance problems), which
>          can waste memory.
> 
> parisc (somewhat inaccurately) said:
> 
>          This option may have a slight performance impact because a
>          portion of the kernel code won't be covered by a TLB anymore.
> 
> IIUC, arm64 does what parisc is hinting at: mappings at the end are
> broken down to PAGE_SIZE. On x86, IIUC, there's actually no change to
> TLB performance due to how the mappings are already set up.

On arm64, we split down to page granularity if needed, but use the
largest possible mapping we can (e.g. if we can use a 2M block, we do).

Because of the way we freed the init area, we already couldn't use
larger mappings anyway. Applying the strict permissions didn't come at a
measureable overhead in any real testing.

> I'm not sure the best way to express this in the new help text. Do you
> have some suggestions on wording? Personally, I don't really think
> it's worth mentioning this in Kconfig help, which, in theory, is
> supposed to limit how technical it gets. And I think the performance
> impact is almost entirely negligible compared to the risks addressed.

I also don't see much point in describing some hypothetical architecture
specific overhead here. In most cases this can't be turned off (so there
isn't anything to comapre it to, and hence no cost). Where people want
to turn it off, they already know why they wish to do so.

Thanks,
Mark.

  parent reply	other threads:[~2017-02-17 18:23 UTC|newest]

Thread overview: 55+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-02-07  0:31 [PATCHv3 0/2] Hardening configs refactor/rename Laura Abbott
2017-02-07  0:31 ` Laura Abbott
2017-02-07  0:31 ` [kernel-hardening] " Laura Abbott
2017-02-07  0:31 ` Laura Abbott
2017-02-07  0:31 ` [PATCHv3 1/2] arch: Move CONFIG_DEBUG_RODATA and CONFIG_SET_MODULE_RONX to be common Laura Abbott
2017-02-07  0:31   ` Laura Abbott
2017-02-07  0:31   ` [kernel-hardening] " Laura Abbott
2017-02-07  7:16   ` Ingo Molnar
2017-02-07  7:16     ` Ingo Molnar
2017-02-07  7:16     ` [kernel-hardening] " Ingo Molnar
2017-02-07  7:16     ` Ingo Molnar
2017-02-07 11:42   ` Heiko Carstens
2017-02-07 11:42     ` Heiko Carstens
2017-02-07 11:42     ` [kernel-hardening] " Heiko Carstens
2017-02-07 11:42     ` Heiko Carstens
2017-02-07  0:31 ` [PATCHv3 2/2] arch: Rename CONFIG_DEBUG_RODATA and CONFIG_DEBUG_MODULE_RONX Laura Abbott
2017-02-07  0:31   ` Laura Abbott
2017-02-07  0:31   ` Laura Abbott
2017-02-07  0:31   ` [kernel-hardening] " Laura Abbott
2017-02-07  0:31   ` Laura Abbott
2017-02-07 19:25   ` Jessica Yu
2017-02-07 19:25     ` Jessica Yu
2017-02-07 19:25     ` Jessica Yu
2017-02-07 19:25     ` [kernel-hardening] " Jessica Yu
2017-02-07 19:25     ` Jessica Yu
2017-02-16 22:25   ` [PATCHv3 2/2] " Pavel Machek
2017-02-16 22:25     ` Pavel Machek
2017-02-16 22:25     ` Pavel Machek
2017-02-16 22:25     ` [kernel-hardening] " Pavel Machek
2017-02-16 22:25     ` Pavel Machek
2017-02-17  1:08     ` Kees Cook
2017-02-17  1:08       ` Kees Cook
2017-02-17  1:08       ` Kees Cook
2017-02-17  1:08       ` [kernel-hardening] " Kees Cook
2017-02-17  1:08       ` Kees Cook
2017-02-17  8:22       ` Helge Deller
2017-02-17  8:22         ` Helge Deller
2017-02-17  8:22         ` Helge Deller
2017-02-17  8:22         ` [kernel-hardening] " Helge Deller
2017-02-17  8:22         ` Helge Deller
2017-02-17 18:23       ` Mark Rutland [this message]
2017-02-17 18:23         ` Mark Rutland
2017-02-17 18:23         ` Mark Rutland
2017-02-17 18:23         ` [kernel-hardening] " Mark Rutland
2017-02-17 18:23         ` Mark Rutland
2017-02-20  7:21       ` Ingo Molnar
2017-02-20  7:21         ` Ingo Molnar
2017-02-20  7:21         ` Ingo Molnar
2017-02-20  7:21         ` [kernel-hardening] " Ingo Molnar
2017-02-20  7:21         ` Ingo Molnar
2017-02-07 21:06 ` [PATCHv3 0/2] Hardening configs refactor/rename Kees Cook
2017-02-07 21:06   ` Kees Cook
2017-02-07 21:06   ` Kees Cook
2017-02-07 21:06   ` [kernel-hardening] " Kees Cook
2017-02-07 21:06   ` Kees Cook

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170217182313.GB25876@leverpostej \
    --to=mark.rutland@arm.com \
    --cc=catalin.marinas@arm.com \
    --cc=corbet@lwn.net \
    --cc=deller@gmx.de \
    --cc=heiko.carstens@de.ibm.com \
    --cc=hpa@zytor.com \
    --cc=jejb@parisc-linux.org \
    --cc=jeyu@redhat.com \
    --cc=keescook@chromium.org \
    --cc=kernel-hardening@lists.openwall.com \
    --cc=labbott@redhat.com \
    --cc=len.brown@intel.com \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-s390@vger.kernel.org \
    --cc=linux@armlinux.org.uk \
    --cc=mingo@redhat.com \
    --cc=pavel@ucw.cz \
    --cc=robh@kernel.org \
    --cc=tglx@linutronix.de \
    --cc=will.deacon@arm.com \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.