All of lore.kernel.org
 help / color / mirror / Atom feed
* [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount.
@ 2012-01-25 14:50 Srinivas KANDAGATLA
  2012-01-25 15:45 ` Jim Rees
  2012-01-25 16:20 ` Boaz Harrosh
  0 siblings, 2 replies; 12+ messages in thread
From: Srinivas KANDAGATLA @ 2012-01-25 14:50 UTC (permalink / raw)
  To: linux-nfs; +Cc: akpm, mingo, neilb, stuart.menefy

From: Srinivas Kandagatla <srinivas.kandagatla@st.com>

This patch adds mountretry kernel parameter for nfs root mount.
mount retry indicates the number of times nfs root mount attempts to be
made before giving up. If this option is not specified, the default
value of 3 retries is used.

One noticable change for us from moving from 2.6.32 to 3.x kernel was
nfs root mount was failing all the time, because we use external PHY.
PHY takes about 2-4 seconds get into a Link ready state after ipconfig.
However as nfsroot mount is attempted in this halfway, which finally
fails.
Important thing to notice here is in 2.6.32 nfs mount usally spent more
than 4 seconds Looking up port of RPC, hence we got lucky to get
nfsroot mount working all the time.

Without this patch a rootdelay has to be added to kernel boot
parameters.

Signed-off-by: Srinivas Kandagatla <srinivas.kandagatla@st.com>
---
Hello All,

Recently we have upgraded kernel from 2.6.32 to 3.x, and One noticable change 
in this process was nfs-root was failing all the time on our boards.
The reason being we are using external PHY, which takes about 2-4 seconds to 
get into Link UP state. In the mean time nfsroot mount attempted by kernel fails 
and results in Kernel panic "VFS: Unable to mount root fs via NFS, trying floppy."

In 2.6.32 nfs mount usally spent more than 4 seconds Looking up port of RPC, 
hence we got lucky.

To get nfs-root working on 3.x kernel we have to add rootdelay to the 
kernel parameters. If the kernel had attempted an other try to mount 
nfs root, it would have succeded.

This will be a common problem across all the boards with external phy's, so 
I was thinking of solving the problem with nfs mount retries option.

I would like to seek your inputs on this patch, and are there any other 
methods of solving this issue without rootdelay option?

Comments?

Thanks,
srini


 Documentation/kernel-parameters.txt |    4 ++++
 init/do_mounts.c                    |   25 +++++++++++++++++++++----
 2 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index e00ec14..20a876e 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -1824,6 +1824,10 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			This can be set from sysctl after boot.
 			See Documentation/sysctl/vm.txt for details.
 
+	mountretry=   	Is the number of times nfs root mount retries attempts
+			are made before giving up. If this option is not
+			specified, the default value of 3 is used.
+
 	ohci1394_dma=early	[HW] enable debugging via the ohci1394 driver.
 			See Documentation/debugging-via-ohci1394.txt for more
 			info.
diff --git a/init/do_mounts.c b/init/do_mounts.c
index c0851a8..03b8c05 100644
--- a/init/do_mounts.c
+++ b/init/do_mounts.c
@@ -258,6 +258,19 @@ __setup("rootflags=", root_data_setup);
 __setup("rootfstype=", fs_names_setup);
 __setup("rootdelay=", root_delay_setup);
 
+
+#ifdef CONFIG_ROOT_NFS
+#define DEF_NFS_MOUNT_RETRIES		3
+static unsigned int __initdata mount_retry = DEF_NFS_MOUNT_RETRIES;
+static int __init mount_retry_setup(char *str)
+{
+	mount_retry = kstrtoul(str, 0, NULL);
+	return 1;
+}
+
+__setup("mountretry=", mount_retry_setup);
+#endif
+
 static void __init get_fs_names(char *page)
 {
 	char *s = page;
@@ -363,12 +376,16 @@ out:
 static int __init mount_nfs_root(void)
 {
 	char *root_dev, *root_data;
-
+	unsigned int retries = mount_retry;
 	if (nfs_root_data(&root_dev, &root_data) != 0)
 		return 0;
-	if (do_mount_root(root_dev, "nfs", root_mountflags, root_data) != 0)
-		return 0;
-	return 1;
+
+	while (retries--) {
+		if (do_mount_root(root_dev,
+				"nfs", root_mountflags, root_data) == 0)
+			return 1;
+	}
+	return 0;
 }
 #endif
 
-- 
1.6.3.3


^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount.
  2012-01-25 14:50 [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount Srinivas KANDAGATLA
@ 2012-01-25 15:45 ` Jim Rees
  2012-01-26  7:56   ` Srinivas KANDAGATLA
  2012-01-25 16:20 ` Boaz Harrosh
  1 sibling, 1 reply; 12+ messages in thread
From: Jim Rees @ 2012-01-25 15:45 UTC (permalink / raw)
  To: Srinivas KANDAGATLA; +Cc: linux-nfs, akpm, mingo, neilb, stuart.menefy

Srinivas KANDAGATLA wrote:

  From: Srinivas Kandagatla <srinivas.kandagatla@st.com>
  
  This patch adds mountretry kernel parameter for nfs root mount.
  mount retry indicates the number of times nfs root mount attempts to be
  made before giving up. If this option is not specified, the default
  value of 3 retries is used.

This has already been fixed in commit 43717c7d, "NFS: Retry mounting
NFSROOT".  The retry count is fixed at 5.  Is there some reason you think
the retry count should be configurable?

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount.
  2012-01-25 14:50 [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount Srinivas KANDAGATLA
  2012-01-25 15:45 ` Jim Rees
@ 2012-01-25 16:20 ` Boaz Harrosh
  2012-01-25 18:32   ` Jim Rees
  1 sibling, 1 reply; 12+ messages in thread
From: Boaz Harrosh @ 2012-01-25 16:20 UTC (permalink / raw)
  To: Srinivas KANDAGATLA; +Cc: linux-nfs, akpm, mingo, neilb, stuart.menefy

On 01/25/2012 04:50 PM, Srinivas KANDAGATLA wrote:
> From: Srinivas Kandagatla <srinivas.kandagatla@st.com>
> 
> This patch adds mountretry kernel parameter for nfs root mount.
> mount retry indicates the number of times nfs root mount attempts to be
> made before giving up. If this option is not specified, the default
> value of 3 retries is used.
> 

If the system is set to have root on NFS. And the root is not found
what does it do? does it Just bums out with a recovery console?

So what better options does it have other then retry for ever. die?

I always thought that was inconsistent. With an hard mount NFS
will never give up and will retry for ever freezing all IOers until
the server came back. Only with root-mount it gives up.

Please explain what is the benefits of giving up at all. Can a machine
be at all usable without it's root? 

Thanks
Boaz

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount.
  2012-01-25 16:20 ` Boaz Harrosh
@ 2012-01-25 18:32   ` Jim Rees
  2012-01-25 18:51     ` Boaz Harrosh
  2012-01-25 18:52     ` Chuck Lever
  0 siblings, 2 replies; 12+ messages in thread
From: Jim Rees @ 2012-01-25 18:32 UTC (permalink / raw)
  To: Boaz Harrosh
  Cc: Srinivas KANDAGATLA, linux-nfs, akpm, mingo, neilb, stuart.menefy

Boaz Harrosh wrote:

  On 01/25/2012 04:50 PM, Srinivas KANDAGATLA wrote:
  > From: Srinivas Kandagatla <srinivas.kandagatla@st.com>
  > 
  > This patch adds mountretry kernel parameter for nfs root mount.
  > mount retry indicates the number of times nfs root mount attempts to be
  > made before giving up. If this option is not specified, the default
  > value of 3 retries is used.
  > 
  
  If the system is set to have root on NFS. And the root is not found
  what does it do? does it Just bums out with a recovery console?
  
  So what better options does it have other then retry for ever. die?
  
  I always thought that was inconsistent. With an hard mount NFS
  will never give up and will retry for ever freezing all IOers until
  the server came back. Only with root-mount it gives up.
  
  Please explain what is the benefits of giving up at all. Can a machine
  be at all usable without it's root? 

Doesn't grub have some fallback logic in case the given kernel/root fails?
Also, don't you want some indication that it's not working other than just a
blank screen?

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount.
  2012-01-25 18:32   ` Jim Rees
@ 2012-01-25 18:51     ` Boaz Harrosh
  2012-01-25 18:52     ` Chuck Lever
  1 sibling, 0 replies; 12+ messages in thread
From: Boaz Harrosh @ 2012-01-25 18:51 UTC (permalink / raw)
  To: Jim Rees
  Cc: Srinivas KANDAGATLA, linux-nfs, akpm, mingo, neilb, stuart.menefy

On 01/25/2012 08:32 PM, Jim Rees wrote:
> Boaz Harrosh wrote:
> 
>   On 01/25/2012 04:50 PM, Srinivas KANDAGATLA wrote:
>   > From: Srinivas Kandagatla <srinivas.kandagatla@st.com>
>   > 
>   > This patch adds mountretry kernel parameter for nfs root mount.
>   > mount retry indicates the number of times nfs root mount attempts to be
>   > made before giving up. If this option is not specified, the default
>   > value of 3 retries is used.
>   > 
>   
>   If the system is set to have root on NFS. And the root is not found
>   what does it do? does it Just bums out with a recovery console?
>   
>   So what better options does it have other then retry for ever. die?
>   
>   I always thought that was inconsistent. With an hard mount NFS
>   will never give up and will retry for ever freezing all IOers until
>   the server came back. Only with root-mount it gives up.
>   
>   Please explain what is the benefits of giving up at all. Can a machine
>   be at all usable without it's root? 
> 
> Doesn't grub have some fallback logic in case the given kernel/root fails?

If grub has one set, then it can also put a parameter to specify the retry count.
(I do like this patch)

> Also, don't you want some indication that it's not working other than just a
> blank screen?

Sure print like hell on every loop bigger then 5 (And sleep between trys)
but keep trying until interrupted (alt+ctrl+delete)

I'm just saying that the default should be "max_uint" not 5

Just my $0.017
Boaz



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount.
  2012-01-25 18:32   ` Jim Rees
  2012-01-25 18:51     ` Boaz Harrosh
@ 2012-01-25 18:52     ` Chuck Lever
  2012-01-25 19:04       ` Boaz Harrosh
  1 sibling, 1 reply; 12+ messages in thread
From: Chuck Lever @ 2012-01-25 18:52 UTC (permalink / raw)
  To: Jim Rees
  Cc: Boaz Harrosh, Srinivas KANDAGATLA, linux-nfs, akpm, mingo, neilb,
	stuart.menefy


On Jan 25, 2012, at 1:32 PM, Jim Rees wrote:

> Boaz Harrosh wrote:
> 
>  On 01/25/2012 04:50 PM, Srinivas KANDAGATLA wrote:
>> From: Srinivas Kandagatla <srinivas.kandagatla@st.com>
>> 
>> This patch adds mountretry kernel parameter for nfs root mount.
>> mount retry indicates the number of times nfs root mount attempts to be
>> made before giving up. If this option is not specified, the default
>> value of 3 retries is used.

Is there a reason 3 is not an adequate fixed setting?  For some reason I don't seem to have this patch in my inbox.

>  If the system is set to have root on NFS. And the root is not found
>  what does it do? does it Just bums out with a recovery console?

No.  It tries the next root file system type.

>  So what better options does it have other then retry for ever. die?

If no root file systems are available, then yes, the kernel panics.  This has been the case for a very very long time.

>  I always thought that was inconsistent. With an hard mount NFS
>  will never give up and will retry for ever freezing all IOers until
>  the server came back. Only with root-mount it gives up.

Not true, user space mount can also give up.  See nfs(5), the retry= option.  This is not NFS I/O we're talking about here, there is no risk of data corruption.  So "hard" versus "soft" does not apply.

>  Please explain what is the benefits of giving up at all. Can a machine
>  be at all usable without it's root? 

There are several different root file system options built into the kernel.  NFSROOT is but one.  Each is tried in succession.  If NFSROOT never gives up, then the others that follow it are never tried.

> Doesn't grub have some fallback logic in case the given kernel/root fails?
> Also, don't you want some indication that it's not working other than just a
> blank screen?

The kernel is running at this point, grub is well out of the picture.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com





^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount.
  2012-01-25 18:52     ` Chuck Lever
@ 2012-01-25 19:04       ` Boaz Harrosh
  0 siblings, 0 replies; 12+ messages in thread
From: Boaz Harrosh @ 2012-01-25 19:04 UTC (permalink / raw)
  To: Chuck Lever
  Cc: Jim Rees, Srinivas KANDAGATLA, linux-nfs, akpm, mingo, neilb,
	stuart.menefy

On 01/25/2012 08:52 PM, Chuck Lever wrote:
> 
> 
>>  I always thought that was inconsistent. With an hard mount NFS
>>  will never give up and will retry for ever freezing all IOers until
>>  the server came back. Only with root-mount it gives up.
> 
> Not true, user space mount can also give up. See nfs(5), the retry=
> option. This is not NFS I/O we're talking about here, there is no
> risk of data corruption. So "hard" versus "soft" does not apply.
> 

OK I see your point

>>  Please explain what is the benefits of giving up at all. Can a machine
>>  be at all usable without it's root? 
> 
> There are several different root file system options built into the
> kernel. NFSROOT is but one. Each is tried in succession. If NFSROOT
> never gives up, then the others that follow it are never tried.
> 

OK I did not know that. Thanks for the explanation, good to know.
In the few times I tried NFSROOT I never setup an alternative.

I wish that in the case that the NFSROOT is the only option it would
keep trying.

When the Kernel does not find it's root it panics, I know. If the sata
disk is not there, it's not there. But with NFSROOT there is a good chance it
will come up soon. (We all saw the same power outage and the server is slow
to boot)

I just think it could be more user friendly that's all

Thanks
Boaz

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount.
  2012-01-25 15:45 ` Jim Rees
@ 2012-01-26  7:56   ` Srinivas KANDAGATLA
  2012-01-26  8:37     ` Boaz Harrosh
  0 siblings, 1 reply; 12+ messages in thread
From: Srinivas KANDAGATLA @ 2012-01-26  7:56 UTC (permalink / raw)
  To: Jim Rees; +Cc: linux-nfs, akpm, mingo, neilb, stuart.menefy

Jim Rees wrote:
> Srinivas KANDAGATLA wrote:
>
>   From: Srinivas Kandagatla <srinivas.kandagatla@st.com>
>   
>   This patch adds mountretry kernel parameter for nfs root mount.
>   mount retry indicates the number of times nfs root mount attempts to be
>   made before giving up. If this option is not specified, the default
>   value of 3 retries is used.
>
> This has already been fixed in commit 43717c7d, "NFS: Retry mounting
> NFSROOT".  The retry count is fixed at 5.  Is there some reason you think
> the retry count should be configurable?
>   
Thanks for pointing to the fixed commit. I did realize soon after
sending this patch to mailing list.

Only reason I can think of having retry count configurable is, an
arbitrary number(3 or 5 or n) of retry count may not be enough to
address all the use cases.
So, I think the user should be given a chance to configure the retry
count if he wish to, either timeout quickly and try the succession
root-mount or If he wish to keep kernel trying very harder till he gets
the NFS root mount successful.

Thanks,
srini

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount.
  2012-01-26  7:56   ` Srinivas KANDAGATLA
@ 2012-01-26  8:37     ` Boaz Harrosh
  2012-01-26  8:51       ` Srinivas KANDAGATLA
  0 siblings, 1 reply; 12+ messages in thread
From: Boaz Harrosh @ 2012-01-26  8:37 UTC (permalink / raw)
  To: srinivas.kandagatla
  Cc: Jim Rees, linux-nfs, akpm, mingo, neilb, stuart.menefy

On 01/26/2012 09:56 AM, Srinivas KANDAGATLA wrote:
> Jim Rees wrote:
>> Srinivas KANDAGATLA wrote:
>>
>>   From: Srinivas Kandagatla <srinivas.kandagatla@st.com>
>>   
>>   This patch adds mountretry kernel parameter for nfs root mount.
>>   mount retry indicates the number of times nfs root mount attempts to be
>>   made before giving up. If this option is not specified, the default
>>   value of 3 retries is used.
>>
>> This has already been fixed in commit 43717c7d, "NFS: Retry mounting
>> NFSROOT".  The retry count is fixed at 5.  Is there some reason you think
>> the retry count should be configurable?
>>   
> Thanks for pointing to the fixed commit. I did realize soon after
> sending this patch to mailing list.
> 
> Only reason I can think of having retry count configurable is, an
> arbitrary number(3 or 5 or n) of retry count may not be enough to
> address all the use cases.
> So, I think the user should be given a chance to configure the retry
> count if he wish to, either timeout quickly and try the succession
> root-mount or If he wish to keep kernel trying very harder till he gets
> the NFS root mount successful.
> 

I agree. Specially if there is no "succession root-mount" setup then user
can specify a very large count.

Is there some delay between retries. So not to burn 100% power?

> Thanks,
> srini

Thanks
Boaz

> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount.
  2012-01-26  8:37     ` Boaz Harrosh
@ 2012-01-26  8:51       ` Srinivas KANDAGATLA
  2012-01-26 11:08         ` Boaz Harrosh
  0 siblings, 1 reply; 12+ messages in thread
From: Srinivas KANDAGATLA @ 2012-01-26  8:51 UTC (permalink / raw)
  To: Boaz Harrosh; +Cc: Jim Rees, linux-nfs, akpm, mingo, neilb, stuart.menefy

Boaz Harrosh wrote:
> On 01/26/2012 09:56 AM, Srinivas KANDAGATLA wrote:
>   
>> Jim Rees wrote:
>>     
>>> Srinivas KANDAGATLA wrote:
>>>
>>>   From: Srinivas Kandagatla <srinivas.kandagatla@st.com>
>>>   
>>>   This patch adds mountretry kernel parameter for nfs root mount.
>>>   mount retry indicates the number of times nfs root mount attempts to be
>>>   made before giving up. If this option is not specified, the default
>>>   value of 3 retries is used.
>>>
>>> This has already been fixed in commit 43717c7d, "NFS: Retry mounting
>>> NFSROOT".  The retry count is fixed at 5.  Is there some reason you think
>>> the retry count should be configurable?
>>>   
>>>       
>> Thanks for pointing to the fixed commit. I did realize soon after
>> sending this patch to mailing list.
>>
>> Only reason I can think of having retry count configurable is, an
>> arbitrary number(3 or 5 or n) of retry count may not be enough to
>> address all the use cases.
>> So, I think the user should be given a chance to configure the retry
>> count if he wish to, either timeout quickly and try the succession
>> root-mount or If he wish to keep kernel trying very harder till he gets
>> the NFS root mount successful.
>>
>>     
>
> I agree. Specially if there is no "succession root-mount" setup then user
> can specify a very large count.
>
> Is there some delay between retries. So not to burn 100% power?
>
>   
An nfs root mount attempt timeouts in about 3 sec's and in commit
43717c7d, "NFS: Retry mounting NFSROOT" each retry has additional sleep
which gives sufficient delay between retries.
>> Thanks,
>> srini
>>     
>
> Thanks
> Boaz
>
>   
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>     
>
>   


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount.
  2012-01-26  8:51       ` Srinivas KANDAGATLA
@ 2012-01-26 11:08         ` Boaz Harrosh
  2012-01-26 14:35           ` Srinivas KANDAGATLA
  0 siblings, 1 reply; 12+ messages in thread
From: Boaz Harrosh @ 2012-01-26 11:08 UTC (permalink / raw)
  To: srinivas.kandagatla
  Cc: Jim Rees, linux-nfs, akpm, mingo, neilb, stuart.menefy

On 01/26/2012 10:51 AM, Srinivas KANDAGATLA wrote:
> An nfs root mount attempt timeouts in about 3 sec's and in commit
> 43717c7d, "NFS: Retry mounting NFSROOT" each retry has additional sleep
> which gives sufficient delay between retries.

Grate thanks. Make perfect sense

>>> Thanks,
>>> srini
>>>     
>>

Thanks
Boaz

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount.
  2012-01-26 11:08         ` Boaz Harrosh
@ 2012-01-26 14:35           ` Srinivas KANDAGATLA
  0 siblings, 0 replies; 12+ messages in thread
From: Srinivas KANDAGATLA @ 2012-01-26 14:35 UTC (permalink / raw)
  To: Boaz Harrosh; +Cc: Jim Rees, linux-nfs, akpm, mingo, neilb, stuart.menefy

Hi All,
Thankyou for your feedback on my patch.
 Am going to re-send the rebased patch in different email-thread with
few other updates to existing nfs-retry code.

Thanks
srini
Boaz Harrosh wrote:
> On 01/26/2012 10:51 AM, Srinivas KANDAGATLA wrote:
>   
>> An nfs root mount attempt timeouts in about 3 sec's and in commit
>> 43717c7d, "NFS: Retry mounting NFSROOT" each retry has additional sleep
>> which gives sufficient delay between retries.
>>     
>
> Grate thanks. Make perfect sense
>
>   
>>>> Thanks,
>>>> srini
>>>>     
>>>>         
>
> Thanks
> Boaz
>   


^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2012-01-26 14:43 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-01-25 14:50 [RFC:PATCH 3.1.0] do_mount: Add mount retry option for nfs root mount Srinivas KANDAGATLA
2012-01-25 15:45 ` Jim Rees
2012-01-26  7:56   ` Srinivas KANDAGATLA
2012-01-26  8:37     ` Boaz Harrosh
2012-01-26  8:51       ` Srinivas KANDAGATLA
2012-01-26 11:08         ` Boaz Harrosh
2012-01-26 14:35           ` Srinivas KANDAGATLA
2012-01-25 16:20 ` Boaz Harrosh
2012-01-25 18:32   ` Jim Rees
2012-01-25 18:51     ` Boaz Harrosh
2012-01-25 18:52     ` Chuck Lever
2012-01-25 19:04       ` Boaz Harrosh

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.