From: Suren Baghdasaryan <surenb@google.com> To: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: akpm@linux-foundation.org, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com Subject: Re: [PATCH v4 18/33] mm: write-lock VMAs before removing them from VMA tree Date: Wed, 1 Mar 2023 18:21:53 -0800 [thread overview] Message-ID: <CAJuCfpEhwye9dn3eUaYE1gSHt+8M5ztBR3ZMoTYyBR7+uSL5aQ@mail.gmail.com> (raw) In-Reply-To: <Y//zmYeu6uexiyOY@localhost> On Wed, Mar 1, 2023 at 4:54 PM Hyeonggon Yoo <42.hyeyoo@gmail.com> wrote: > > On Wed, Mar 01, 2023 at 10:42:48AM -0800, Suren Baghdasaryan wrote: > > On Wed, Mar 1, 2023 at 10:34 AM Suren Baghdasaryan <surenb@google.com> wrote: > > > > > > On Tue, Feb 28, 2023 at 11:57 PM Hyeonggon Yoo <42.hyeyoo@gmail.com> wrote: > > > > > > > > On Wed, Mar 01, 2023 at 07:43:33AM +0000, Hyeonggon Yoo wrote: > > > > > On Mon, Feb 27, 2023 at 09:36:17AM -0800, Suren Baghdasaryan wrote: > > > > > > Write-locking VMAs before isolating them ensures that page fault > > > > > > handlers don't operate on isolated VMAs. > > > > > > > > > > > > Signed-off-by: Suren Baghdasaryan <surenb@google.com> > > > > > > --- > > > > > > mm/mmap.c | 1 + > > > > > > mm/nommu.c | 5 +++++ > > > > > > 2 files changed, 6 insertions(+) > > > > > > > > > > > > diff --git a/mm/mmap.c b/mm/mmap.c > > > > > > index 1f42b9a52b9b..f7ed357056c4 100644 > > > > > > --- a/mm/mmap.c > > > > > > +++ b/mm/mmap.c > > > > > > @@ -2255,6 +2255,7 @@ int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, > > > > > > static inline int munmap_sidetree(struct vm_area_struct *vma, > > > > > > struct ma_state *mas_detach) > > > > > > { > > > > > > + vma_start_write(vma); > > > > > > mas_set_range(mas_detach, vma->vm_start, vma->vm_end - 1); > > > > > > > > > > I may be missing something, but have few questions: > > > > > > > > > > 1) Why does a writer need to both write-lock a VMA and mark the VMA detached > > > > > when unmapping it, isn't it enough to just only write-lock a VMA? > > > > > > We need to mark the VMA detached to avoid handling page fault in a > > > detached VMA. The possible scenario is: > > > > > > lock_vma_under_rcu > > > vma = mas_walk(&mas) > > > munmap_sidetree > > > vma_start_write(vma) > > > > > > mas_store_gfp() // remove VMA from the tree > > > vma_end_write_all() > > > vma_start_read(vma) > > > // we locked the VMA but it is not part of the tree anymore. > > > > > > So, marking the VMA locked before vma_end_write_all() and checking > > > > Sorry, I should have said "marking the VMA *detached* before > > vma_end_write_all() and checking vma->detached after vma_start_read() > > helps us avoid handling faults in the detached VMA." > > > > > vma->detached after vma_start_read() helps us avoid handling faults in > > > the detached VMA. > > Thank you for explanation, that makes sense! > > By the way, if there are no 32bit users of Per-VMA lock (are there?), > "detached" bool could be a VMA flag (i.e. making it depend on 64BIT > and selecting ARCH_USES_HIGH_VMA_FLAGS) Yeah, I thought about it but didn't want to make assumptions about potential users just yet. Besides, I heard there are attempts to make vm_flags to be always 64-bit (I think Matthew mentioned that to me once). If that happens, we won't need any dependencies here. Either way, this conversion into a flag can be done as an additional optimization later on. I prefer to keep the main patchset as simple as possible for now. Thanks, Suren. > > Thanks, > Hyeonggon > > -- > To unsubscribe from this group and stop receiving emails from it, send an email to kernel-team+unsubscribe@android.com. >
WARNING: multiple messages have this Message-ID (diff)
From: Suren Baghdasaryan <surenb@google.com> To: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: michel@lespinasse.org, joelaf@google.com, songliubraving@fb.com, mhocko@suse.com, leewalsh@google.com, david@redhat.com, peterz@infradead.org, bigeasy@linutronix.de, peterx@redhat.com, dhowells@redhat.com, linux-mm@kvack.org, edumazet@google.com, jglisse@google.com, punit.agrawal@bytedance.com, will@kernel.org, arjunroy@google.com, chriscli@google.com, dave@stgolabs.net, minchan@google.com, x86@kernel.org, hughd@google.com, willy@infradead.org, gurua@google.com, mingo@redhat.com, linux-arm-kernel@lists.infradead.org, rientjes@google.com, axelrasmussen@google.com, kernel-team@android.com, michalechner92@googlemail.com, soheil@google.com, paulmck@kernel.org, jannh@google.com, liam.howlett@oracle.com, shakeelb@google.com, luto@kernel.org, gthelen@google.com, ldufour@linux.ibm.com, vbabka@suse.cz, posk@google.com, lstoakes@gmail.com, peterjung1337@gmail.com, linuxppc-dev@lists.ozlabs.org, kent.overstreet@linux.dev, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, akpm@linux-foundati on.org, tatashin@google.com, mgorman@techsingularity.net, rppt@kernel.org Subject: Re: [PATCH v4 18/33] mm: write-lock VMAs before removing them from VMA tree Date: Wed, 1 Mar 2023 18:21:53 -0800 [thread overview] Message-ID: <CAJuCfpEhwye9dn3eUaYE1gSHt+8M5ztBR3ZMoTYyBR7+uSL5aQ@mail.gmail.com> (raw) In-Reply-To: <Y//zmYeu6uexiyOY@localhost> On Wed, Mar 1, 2023 at 4:54 PM Hyeonggon Yoo <42.hyeyoo@gmail.com> wrote: > > On Wed, Mar 01, 2023 at 10:42:48AM -0800, Suren Baghdasaryan wrote: > > On Wed, Mar 1, 2023 at 10:34 AM Suren Baghdasaryan <surenb@google.com> wrote: > > > > > > On Tue, Feb 28, 2023 at 11:57 PM Hyeonggon Yoo <42.hyeyoo@gmail.com> wrote: > > > > > > > > On Wed, Mar 01, 2023 at 07:43:33AM +0000, Hyeonggon Yoo wrote: > > > > > On Mon, Feb 27, 2023 at 09:36:17AM -0800, Suren Baghdasaryan wrote: > > > > > > Write-locking VMAs before isolating them ensures that page fault > > > > > > handlers don't operate on isolated VMAs. > > > > > > > > > > > > Signed-off-by: Suren Baghdasaryan <surenb@google.com> > > > > > > --- > > > > > > mm/mmap.c | 1 + > > > > > > mm/nommu.c | 5 +++++ > > > > > > 2 files changed, 6 insertions(+) > > > > > > > > > > > > diff --git a/mm/mmap.c b/mm/mmap.c > > > > > > index 1f42b9a52b9b..f7ed357056c4 100644 > > > > > > --- a/mm/mmap.c > > > > > > +++ b/mm/mmap.c > > > > > > @@ -2255,6 +2255,7 @@ int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, > > > > > > static inline int munmap_sidetree(struct vm_area_struct *vma, > > > > > > struct ma_state *mas_detach) > > > > > > { > > > > > > + vma_start_write(vma); > > > > > > mas_set_range(mas_detach, vma->vm_start, vma->vm_end - 1); > > > > > > > > > > I may be missing something, but have few questions: > > > > > > > > > > 1) Why does a writer need to both write-lock a VMA and mark the VMA detached > > > > > when unmapping it, isn't it enough to just only write-lock a VMA? > > > > > > We need to mark the VMA detached to avoid handling page fault in a > > > detached VMA. The possible scenario is: > > > > > > lock_vma_under_rcu > > > vma = mas_walk(&mas) > > > munmap_sidetree > > > vma_start_write(vma) > > > > > > mas_store_gfp() // remove VMA from the tree > > > vma_end_write_all() > > > vma_start_read(vma) > > > // we locked the VMA but it is not part of the tree anymore. > > > > > > So, marking the VMA locked before vma_end_write_all() and checking > > > > Sorry, I should have said "marking the VMA *detached* before > > vma_end_write_all() and checking vma->detached after vma_start_read() > > helps us avoid handling faults in the detached VMA." > > > > > vma->detached after vma_start_read() helps us avoid handling faults in > > > the detached VMA. > > Thank you for explanation, that makes sense! > > By the way, if there are no 32bit users of Per-VMA lock (are there?), > "detached" bool could be a VMA flag (i.e. making it depend on 64BIT > and selecting ARCH_USES_HIGH_VMA_FLAGS) Yeah, I thought about it but didn't want to make assumptions about potential users just yet. Besides, I heard there are attempts to make vm_flags to be always 64-bit (I think Matthew mentioned that to me once). If that happens, we won't need any dependencies here. Either way, this conversion into a flag can be done as an additional optimization later on. I prefer to keep the main patchset as simple as possible for now. Thanks, Suren. > > Thanks, > Hyeonggon > > -- > To unsubscribe from this group and stop receiving emails from it, send an email to kernel-team+unsubscribe@android.com. >
next prev parent reply other threads:[~2023-03-02 2:22 UTC|newest] Thread overview: 134+ messages / expand[flat|nested] mbox.gz Atom feed top 2023-02-27 17:35 [PATCH v4 00/33] Per-VMA locks Suren Baghdasaryan 2023-02-27 17:35 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 01/33] maple_tree: Be more cautious about dead nodes Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 02/33] maple_tree: Detect dead nodes in mas_start() Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 03/33] maple_tree: Fix freeing of nodes in rcu mode Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 04/33] maple_tree: remove extra smp_wmb() from mas_dead_leaves() Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 05/33] maple_tree: Fix write memory barrier of nodes once dead for RCU mode Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 06/33] maple_tree: Add smp_rmb() to dead node detection Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 07/33] maple_tree: Add RCU lock checking to rcu callback functions Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 08/33] mm: Enable maple tree RCU mode by default Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 09/33] mm: introduce CONFIG_PER_VMA_LOCK Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 10/33] mm: rcu safe VMA freeing Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 11/33] mm: move mmap_lock assert function definitions Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 12/33] mm: add per-VMA lock and helper functions to control it Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 13/33] mm: mark VMA as being written when changing vm_flags Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 14/33] mm/mmap: move vma_prepare before vma_adjust_trans_huge Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 15/33] mm/khugepaged: write-lock VMA while collapsing a huge page Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 16/33] mm/mmap: write-lock VMAs in vma_prepare before modifying them Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 17/33] mm/mremap: write-lock VMA while remapping it to a new address range Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-03-01 7:01 ` Hyeonggon Yoo 2023-03-01 7:01 ` Hyeonggon Yoo 2023-02-27 17:36 ` [PATCH v4 18/33] mm: write-lock VMAs before removing them from VMA tree Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-03-01 7:43 ` Hyeonggon Yoo 2023-03-01 7:43 ` Hyeonggon Yoo 2023-03-01 7:56 ` Hyeonggon Yoo 2023-03-01 7:56 ` Hyeonggon Yoo 2023-03-01 18:34 ` Suren Baghdasaryan 2023-03-01 18:34 ` Suren Baghdasaryan 2023-03-01 18:42 ` Suren Baghdasaryan 2023-03-01 18:42 ` Suren Baghdasaryan 2023-03-02 0:53 ` Hyeonggon Yoo 2023-03-02 0:53 ` Hyeonggon Yoo 2023-03-02 2:21 ` Suren Baghdasaryan [this message] 2023-03-02 2:21 ` Suren Baghdasaryan 2023-03-01 19:07 ` Suren Baghdasaryan 2023-03-01 19:07 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 19/33] mm: conditionally write-lock VMA in free_pgtables Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 20/33] kernel/fork: assert no VMA readers during its destruction Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 21/33] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 22/33] mm: introduce vma detached flag Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 23/33] mm: introduce lock_vma_under_rcu to be used from arch-specific code Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 24/33] mm: fall back to mmap_lock if vma->anon_vma is not yet set Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-03-01 9:54 ` Hyeonggon Yoo 2023-03-01 9:54 ` Hyeonggon Yoo 2023-02-27 17:36 ` [PATCH v4 25/33] mm: add FAULT_FLAG_VMA_LOCK flag Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 26/33] mm: prevent do_swap_page from handling page faults under VMA lock Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 27/33] mm: prevent userfaults to be handled under per-vma lock Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 28/33] mm: introduce per-VMA lock statistics Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 29/33] x86/mm: try VMA lock-based page fault handling first Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-06-29 14:40 ` Jiri Slaby 2023-06-29 14:40 ` Jiri Slaby 2023-06-29 15:30 ` Suren Baghdasaryan 2023-06-29 15:30 ` Suren Baghdasaryan 2023-06-30 6:35 ` Jiri Slaby 2023-06-30 6:35 ` Jiri Slaby 2023-06-30 8:28 ` Jiri Slaby 2023-06-30 8:28 ` Jiri Slaby 2023-06-30 8:43 ` Jiri Slaby 2023-06-30 8:43 ` Jiri Slaby 2023-06-30 17:40 ` Suren Baghdasaryan 2023-06-30 17:40 ` Suren Baghdasaryan 2023-07-03 10:47 ` Jiri Slaby 2023-07-03 10:47 ` Jiri Slaby 2023-07-03 13:52 ` Holger Hoffstätte 2023-07-03 14:45 ` Suren Baghdasaryan 2023-07-03 15:24 ` Suren Baghdasaryan 2023-07-03 18:28 ` Suren Baghdasaryan 2023-07-05 22:15 ` Suren Baghdasaryan 2023-07-05 22:37 ` Holger Hoffstätte 2023-07-05 22:55 ` Suren Baghdasaryan 2023-07-06 14:27 ` Holger Hoffstätte 2023-07-06 16:11 ` Suren Baghdasaryan 2023-07-07 2:23 ` Suren Baghdasaryan 2023-07-07 4:40 ` Suren Baghdasaryan 2023-07-11 6:20 ` Jiri Slaby 2023-06-29 17:06 ` Linux regression tracking #adding (Thorsten Leemhuis) 2023-06-29 17:06 ` Linux regression tracking #adding (Thorsten Leemhuis) 2023-07-10 10:45 ` Linux regression tracking #update (Thorsten Leemhuis) 2023-07-03 9:58 ` Linux regression tracking (Thorsten Leemhuis) 2023-07-03 9:58 ` Linux regression tracking (Thorsten Leemhuis) 2023-02-27 17:36 ` [PATCH v4 30/33] arm64/mm: " Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 31/33] powerc/mm: " Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-03-06 15:42 ` [PATCH] powerpc/mm: fix mmap_lock bad unlock Laurent Dufour 2023-03-06 15:42 ` Laurent Dufour 2023-03-06 20:25 ` [PATCH v4 31/33] powerc/mm: try VMA lock-based page fault handling first Suren Baghdasaryan 2023-03-06 20:25 ` Suren Baghdasaryan 2023-03-06 20:25 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 32/33] mm/mmap: free vm_area_struct without call_rcu in exit_mmap Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-02-27 17:36 ` [PATCH v4 33/33] mm: separate vma->lock from vm_area_struct Suren Baghdasaryan 2023-02-27 17:36 ` Suren Baghdasaryan 2023-07-11 10:35 ` [PATCH v4 00/33] Per-VMA locks Leon Romanovsky 2023-07-11 10:35 ` Leon Romanovsky 2023-07-11 10:39 ` Vlastimil Babka 2023-07-11 10:39 ` Vlastimil Babka 2023-07-11 11:01 ` Leon Romanovsky 2023-07-11 11:01 ` Leon Romanovsky 2023-07-11 11:09 ` Leon Romanovsky 2023-07-11 11:09 ` Leon Romanovsky 2023-07-11 16:35 ` Suren Baghdasaryan 2023-07-11 16:35 ` Suren Baghdasaryan 2023-07-11 17:14 ` Leon Romanovsky 2023-07-11 17:14 ` Leon Romanovsky
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=CAJuCfpEhwye9dn3eUaYE1gSHt+8M5ztBR3ZMoTYyBR7+uSL5aQ@mail.gmail.com \ --to=surenb@google.com \ --cc=42.hyeyoo@gmail.com \ --cc=akpm@linux-foundation.org \ --cc=arjunroy@google.com \ --cc=axelrasmussen@google.com \ --cc=bigeasy@linutronix.de \ --cc=chriscli@google.com \ --cc=dave@stgolabs.net \ --cc=david@redhat.com \ --cc=dhowells@redhat.com \ --cc=edumazet@google.com \ --cc=gthelen@google.com \ --cc=gurua@google.com \ --cc=hannes@cmpxchg.org \ --cc=hughd@google.com \ --cc=jannh@google.com \ --cc=jglisse@google.com \ --cc=joelaf@google.com \ --cc=kent.overstreet@linux.dev \ --cc=kernel-team@android.com \ --cc=ldufour@linux.ibm.com \ --cc=leewalsh@google.com \ --cc=liam.howlett@oracle.com \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linuxppc-dev@lists.ozlabs.org \ --cc=lstoakes@gmail.com \ --cc=luto@kernel.org \ --cc=mgorman@techsingularity.net \ --cc=mhocko@suse.com \ --cc=michalechner92@googlemail.com \ --cc=michel@lespinasse.org \ --cc=minchan@google.com \ --cc=mingo@redhat.com \ --cc=paulmck@kernel.org \ --cc=peterjung1337@gmail.com \ --cc=peterx@redhat.com \ --cc=peterz@infradead.org \ --cc=posk@google.com \ --cc=punit.agrawal@bytedance.com \ --cc=rientjes@google.com \ --cc=rppt@kernel.org \ --cc=shakeelb@google.com \ --cc=soheil@google.com \ --cc=songliubraving@fb.com \ --cc=tatashin@google.com \ --cc=vbabka@suse.cz \ --cc=will@kernel.org \ --cc=willy@infradead.org \ --cc=x86@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.