From: Johannes Weiner <hannes@cmpxchg.org> To: Michal Hocko <mhocko@suse.cz> Cc: Hugh Dickins <hughd@google.com>, Andrew Morton <akpm@linux-foundation.org>, KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>, KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>, Greg Thelen <gthelen@google.com>, Michel Lespinasse <walken@google.com>, Tejun Heo <tj@kernel.org>, Roman Gushchin <klamm@yandex-team.ru>, LKML <linux-kernel@vger.kernel.org>, linux-mm@kvack.org, Rik van Riel <riel@redhat.com> Subject: Re: [PATCH v2 0/4] memcg: Low-limit reclaim Date: Thu, 5 Jun 2014 14:23:36 -0400 [thread overview] Message-ID: <20140605182336.GA2878@cmpxchg.org> (raw) In-Reply-To: <20140605164355.GA22276@dhcp22.suse.cz> On Thu, Jun 05, 2014 at 06:43:55PM +0200, Michal Hocko wrote: > On Thu 05-06-14 12:10:35, Johannes Weiner wrote: > > On Thu, Jun 05, 2014 at 04:51:09PM +0200, Michal Hocko wrote: > > > On Wed 04-06-14 17:45:53, Johannes Weiner wrote: > > > > On Wed, Jun 04, 2014 at 12:18:59PM -0700, Hugh Dickins wrote: > > > > > On Wed, 4 Jun 2014, Johannes Weiner wrote: > > > > > > On Wed, Jun 04, 2014 at 04:46:58PM +0200, Michal Hocko wrote: > > > > > > > > > > > > > > In the other email I have suggested to add a knob with the configurable > > > > > > > default. Would you be OK with that? > > > > > > > > > > > > No, I want to agree on whether we need that fallback code or not. I'm > > > > > > not interested in merging code that you can't convince anybody else is > > > > > > needed. > > > > > > > > > > I for one would welcome such a knob as Michal is proposing. > > > > > > > > Now we have a tie :-) > > > > > > > > > I thought it was long ago agreed that the low limit was going to fallback > > > > > when it couldn't be satisfied. But you seem implacably opposed to that > > > > > as default, and I can well believe that Google is so accustomed to OOMing > > > > > that it is more comfortable with OOMing as the default. Okay. But I > > > > > would expect there to be many who want the attempt towards isolation that > > > > > low limit offers, without a collapse to OOM at the first misjudgement. > > > > > > > > At the same time, I only see users like Google pushing the limits of > > > > the machine to a point where guarantees cover north of 90% of memory. > > > > > > I can think of in-memory database loads which would use the reclaim > > > protection which is quite high as well (say 80% of available memory). > > > Those would definitely like to see ephemeral reclaim rather than OOM. > > > > The OOM wouldn't apply to the database workload, but to other stuff in > > the system. > > Are we talking about the same thing? If nothing is reclaimable because > everybody is within limit then the database workload will be, of course, > a candidate for OOM. And quite a hot one as the memory consumption will > be dominating. Yes, you're right. > > > > I would expect more casual users to work with much smaller guarantees, > > > > and a good chunk of slack on top - otherwise they already had better > > > > be set up for the occasional OOM. Is this an unreasonable assumption > > > > to make? > > > > > > > > I'm not opposed to this feature per se, but I'm really opposed to > > > > merging it for the partial hard bindings argument > > > > > > This was just an example that even setup which is not overcomiting the > > > limit might be caught in an unreclaimable position. Sure we can mitigate > > > those issues to some point and that would be surely welcome. > > > > > > The more important part, however, is that not all usecases really > > > _require_ hard guarantee. > > > > It's not about whether hard guarantees are necessary, it's about > > getting away without additional fallback semantics. The default > > position is still always simpler semantics, so we are looking for > > reasons for the fallback here, not the other way around. > > This doesn't make much sense to me. So you are pushing for something > that is even not necessary. I have already mentioned that I am aware of > usecases which would prefer ephemeral reclaim rather than OOM and that > is pretty darn good reason to have a fallback mode. I think it's quite clear that there is merit for both behaviors, but because there are less "traps" in the hard guarantee semantics their usefulness is much easier to assess. So can we please explore the situations wherein fallbacks would happen so that we can judge the applicability of both behaviors and pick a reasonable default? > > > They are asking for a reasonable memory isolation which they > > > currently do not have. Having a risk of OOM would be a no-go for > > > them so the feature wouldn't be useful for them. > > > > Let's not go back to Handwaving Land on this, please. What does > > "reasonable memory isolation" mean? > > To not reclaim unless everybody is within limit. This doesn't happen > when the limit is not overcomitted normally but still cannot be ruled out > for different reasons (non-user allocations, NUMA setups and who knows > what else) I'm desperately trying to gather a list of these corner cases to get a feeling for when that "best effort" falls apart, because that is part of the interface and something that we have to know for future kernel development, and the user has to know in order to pick a behavior. Your documentation doesn't mention any of those corner cases and leads on that you're basically guaranteed this memory unless you overcommit the limit. > > > I have repeatedly said that I can see also some use for the hard > > > guarantee. Mainly to support overcommit on the limit. I didn't hear > > > about those usecases yet but it seems that at least Google would like to > > > have really hard guarantees. > > > > Everybody who wants to charge for guaranteed memory can not afford to > > have other workloads break isolation at will. > > I am getting tired of this discussion to be honest. You seem to be > locked up to guarantee ignoring that there are usecases which really do > not need have such requirements. I already wrote Hugh that I'm not against the fall back per se, but I really want to map out the usecases and match them up with the hard and best-effort low limit, so that we know how to document this and what the default behavior should be. It's not the fallback that's bothering me, it's your unwillingness to explore and document the vagueness that is inherent in the semantics. I really don't want to merge more underdefined best-effort features that will be useless to users and a hinderance in future development. "I'm aware of usecases that would prefer reclaim over OOM" is just not cutting it when it comes to designing an interface that we're going to be stuck with indefinitely. We need a concrete understanding of how configurations would behave under common situations in the real world.
WARNING: multiple messages have this Message-ID (diff)
From: Johannes Weiner <hannes@cmpxchg.org> To: Michal Hocko <mhocko@suse.cz> Cc: Hugh Dickins <hughd@google.com>, Andrew Morton <akpm@linux-foundation.org>, KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>, KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>, Greg Thelen <gthelen@google.com>, Michel Lespinasse <walken@google.com>, Tejun Heo <tj@kernel.org>, Roman Gushchin <klamm@yandex-team.ru>, LKML <linux-kernel@vger.kernel.org>, linux-mm@kvack.org, Rik van Riel <riel@redhat.com> Subject: Re: [PATCH v2 0/4] memcg: Low-limit reclaim Date: Thu, 5 Jun 2014 14:23:36 -0400 [thread overview] Message-ID: <20140605182336.GA2878@cmpxchg.org> (raw) In-Reply-To: <20140605164355.GA22276@dhcp22.suse.cz> On Thu, Jun 05, 2014 at 06:43:55PM +0200, Michal Hocko wrote: > On Thu 05-06-14 12:10:35, Johannes Weiner wrote: > > On Thu, Jun 05, 2014 at 04:51:09PM +0200, Michal Hocko wrote: > > > On Wed 04-06-14 17:45:53, Johannes Weiner wrote: > > > > On Wed, Jun 04, 2014 at 12:18:59PM -0700, Hugh Dickins wrote: > > > > > On Wed, 4 Jun 2014, Johannes Weiner wrote: > > > > > > On Wed, Jun 04, 2014 at 04:46:58PM +0200, Michal Hocko wrote: > > > > > > > > > > > > > > In the other email I have suggested to add a knob with the configurable > > > > > > > default. Would you be OK with that? > > > > > > > > > > > > No, I want to agree on whether we need that fallback code or not. I'm > > > > > > not interested in merging code that you can't convince anybody else is > > > > > > needed. > > > > > > > > > > I for one would welcome such a knob as Michal is proposing. > > > > > > > > Now we have a tie :-) > > > > > > > > > I thought it was long ago agreed that the low limit was going to fallback > > > > > when it couldn't be satisfied. But you seem implacably opposed to that > > > > > as default, and I can well believe that Google is so accustomed to OOMing > > > > > that it is more comfortable with OOMing as the default. Okay. But I > > > > > would expect there to be many who want the attempt towards isolation that > > > > > low limit offers, without a collapse to OOM at the first misjudgement. > > > > > > > > At the same time, I only see users like Google pushing the limits of > > > > the machine to a point where guarantees cover north of 90% of memory. > > > > > > I can think of in-memory database loads which would use the reclaim > > > protection which is quite high as well (say 80% of available memory). > > > Those would definitely like to see ephemeral reclaim rather than OOM. > > > > The OOM wouldn't apply to the database workload, but to other stuff in > > the system. > > Are we talking about the same thing? If nothing is reclaimable because > everybody is within limit then the database workload will be, of course, > a candidate for OOM. And quite a hot one as the memory consumption will > be dominating. Yes, you're right. > > > > I would expect more casual users to work with much smaller guarantees, > > > > and a good chunk of slack on top - otherwise they already had better > > > > be set up for the occasional OOM. Is this an unreasonable assumption > > > > to make? > > > > > > > > I'm not opposed to this feature per se, but I'm really opposed to > > > > merging it for the partial hard bindings argument > > > > > > This was just an example that even setup which is not overcomiting the > > > limit might be caught in an unreclaimable position. Sure we can mitigate > > > those issues to some point and that would be surely welcome. > > > > > > The more important part, however, is that not all usecases really > > > _require_ hard guarantee. > > > > It's not about whether hard guarantees are necessary, it's about > > getting away without additional fallback semantics. The default > > position is still always simpler semantics, so we are looking for > > reasons for the fallback here, not the other way around. > > This doesn't make much sense to me. So you are pushing for something > that is even not necessary. I have already mentioned that I am aware of > usecases which would prefer ephemeral reclaim rather than OOM and that > is pretty darn good reason to have a fallback mode. I think it's quite clear that there is merit for both behaviors, but because there are less "traps" in the hard guarantee semantics their usefulness is much easier to assess. So can we please explore the situations wherein fallbacks would happen so that we can judge the applicability of both behaviors and pick a reasonable default? > > > They are asking for a reasonable memory isolation which they > > > currently do not have. Having a risk of OOM would be a no-go for > > > them so the feature wouldn't be useful for them. > > > > Let's not go back to Handwaving Land on this, please. What does > > "reasonable memory isolation" mean? > > To not reclaim unless everybody is within limit. This doesn't happen > when the limit is not overcomitted normally but still cannot be ruled out > for different reasons (non-user allocations, NUMA setups and who knows > what else) I'm desperately trying to gather a list of these corner cases to get a feeling for when that "best effort" falls apart, because that is part of the interface and something that we have to know for future kernel development, and the user has to know in order to pick a behavior. Your documentation doesn't mention any of those corner cases and leads on that you're basically guaranteed this memory unless you overcommit the limit. > > > I have repeatedly said that I can see also some use for the hard > > > guarantee. Mainly to support overcommit on the limit. I didn't hear > > > about those usecases yet but it seems that at least Google would like to > > > have really hard guarantees. > > > > Everybody who wants to charge for guaranteed memory can not afford to > > have other workloads break isolation at will. > > I am getting tired of this discussion to be honest. You seem to be > locked up to guarantee ignoring that there are usecases which really do > not need have such requirements. I already wrote Hugh that I'm not against the fall back per se, but I really want to map out the usecases and match them up with the hard and best-effort low limit, so that we know how to document this and what the default behavior should be. It's not the fallback that's bothering me, it's your unwillingness to explore and document the vagueness that is inherent in the semantics. I really don't want to merge more underdefined best-effort features that will be useless to users and a hinderance in future development. "I'm aware of usecases that would prefer reclaim over OOM" is just not cutting it when it comes to designing an interface that we're going to be stuck with indefinitely. We need a concrete understanding of how configurations would behave under common situations in the real world. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2014-06-05 18:24 UTC|newest] Thread overview: 196+ messages / expand[flat|nested] mbox.gz Atom feed top 2014-04-28 12:26 [PATCH v2 0/4] memcg: Low-limit reclaim Michal Hocko 2014-04-28 12:26 ` Michal Hocko 2014-04-28 12:26 ` [PATCH 1/4] memcg, mm: introduce lowlimit reclaim Michal Hocko 2014-04-28 12:26 ` Michal Hocko 2014-04-30 22:55 ` Johannes Weiner 2014-04-30 22:55 ` Johannes Weiner 2014-05-02 9:36 ` Michal Hocko 2014-05-02 9:36 ` Michal Hocko 2014-05-02 12:07 ` Michal Hocko 2014-05-02 12:07 ` Michal Hocko 2014-05-02 13:01 ` Johannes Weiner 2014-05-02 13:01 ` Johannes Weiner 2014-05-02 14:15 ` Michal Hocko 2014-05-02 14:15 ` Michal Hocko 2014-05-02 15:04 ` Johannes Weiner 2014-05-02 15:04 ` Johannes Weiner 2014-05-02 15:11 ` Michal Hocko 2014-05-02 15:11 ` Michal Hocko 2014-05-02 15:34 ` Johannes Weiner 2014-05-02 15:34 ` Johannes Weiner 2014-05-02 15:48 ` Michal Hocko 2014-05-02 15:48 ` Michal Hocko 2014-05-06 19:58 ` Michal Hocko 2014-05-06 19:58 ` Michal Hocko 2014-05-02 15:58 ` Johannes Weiner 2014-05-02 15:58 ` Johannes Weiner 2014-05-02 16:49 ` Michal Hocko 2014-05-02 16:49 ` Michal Hocko 2014-05-02 22:00 ` Johannes Weiner 2014-05-02 22:00 ` Johannes Weiner 2014-05-05 14:21 ` Michal Hocko 2014-05-05 14:21 ` Michal Hocko 2014-05-19 16:18 ` Michal Hocko 2014-05-19 16:18 ` Michal Hocko 2014-06-11 15:15 ` Johannes Weiner 2014-06-11 15:15 ` Johannes Weiner 2014-06-11 16:08 ` Michal Hocko 2014-06-11 16:08 ` Michal Hocko 2014-05-06 13:29 ` Johannes Weiner 2014-05-06 14:32 ` Michal Hocko 2014-05-06 14:32 ` Michal Hocko 2014-05-06 15:21 ` Johannes Weiner 2014-05-06 15:21 ` Johannes Weiner 2014-05-06 16:12 ` Michal Hocko 2014-05-06 16:12 ` Michal Hocko 2014-05-06 16:51 ` Johannes Weiner 2014-05-06 16:51 ` Johannes Weiner 2014-05-06 18:30 ` Michal Hocko 2014-05-06 18:30 ` Michal Hocko 2014-05-06 19:55 ` Johannes Weiner 2014-05-06 19:55 ` Johannes Weiner 2014-04-28 12:26 ` [PATCH 2/4] memcg: Allow setting low_limit Michal Hocko 2014-04-28 12:26 ` Michal Hocko 2014-04-28 12:26 ` [PATCH 3/4] memcg, doc: clarify global vs. limit reclaims Michal Hocko 2014-04-28 12:26 ` Michal Hocko 2014-04-30 23:03 ` Johannes Weiner 2014-04-30 23:03 ` Johannes Weiner 2014-05-02 9:43 ` Michal Hocko 2014-05-02 9:43 ` Michal Hocko 2014-05-06 19:56 ` Michal Hocko 2014-05-06 19:56 ` Michal Hocko 2014-04-28 12:26 ` [PATCH 4/4] memcg: Document memory.low_limit_in_bytes Michal Hocko 2014-04-28 12:26 ` Michal Hocko 2014-04-30 22:57 ` Johannes Weiner 2014-04-30 22:57 ` Johannes Weiner 2014-05-02 9:46 ` Michal Hocko 2014-05-02 9:46 ` Michal Hocko 2014-04-28 15:46 ` [PATCH v2 0/4] memcg: Low-limit reclaim Roman Gushchin 2014-04-28 15:46 ` Roman Gushchin 2014-04-29 7:42 ` Greg Thelen 2014-04-29 7:42 ` Greg Thelen 2014-04-29 10:50 ` Roman Gushchin 2014-04-29 10:50 ` Roman Gushchin 2014-04-29 12:54 ` Michal Hocko 2014-04-29 12:54 ` Michal Hocko 2014-04-30 21:52 ` Andrew Morton 2014-04-30 21:52 ` Andrew Morton 2014-04-30 22:49 ` Johannes Weiner 2014-04-30 22:49 ` Johannes Weiner 2014-05-02 12:03 ` Michal Hocko 2014-05-02 12:03 ` Michal Hocko 2014-04-30 21:59 ` Andrew Morton 2014-04-30 21:59 ` Andrew Morton 2014-05-02 11:22 ` Michal Hocko 2014-05-02 11:22 ` Michal Hocko 2014-05-28 12:10 ` Michal Hocko 2014-05-28 12:10 ` Michal Hocko 2014-05-28 13:49 ` Johannes Weiner 2014-05-28 13:49 ` Johannes Weiner 2014-05-28 14:21 ` Michal Hocko 2014-05-28 14:21 ` Michal Hocko 2014-05-28 15:28 ` Johannes Weiner 2014-05-28 15:28 ` Johannes Weiner 2014-05-28 15:54 ` Michal Hocko 2014-05-28 15:54 ` Michal Hocko 2014-05-28 16:33 ` Johannes Weiner 2014-05-28 16:33 ` Johannes Weiner 2014-06-03 11:07 ` Michal Hocko 2014-06-03 11:07 ` Michal Hocko 2014-06-03 14:22 ` Johannes Weiner 2014-06-03 14:22 ` Johannes Weiner 2014-06-04 14:46 ` Michal Hocko 2014-06-04 14:46 ` Michal Hocko 2014-06-04 15:44 ` Johannes Weiner 2014-06-04 15:44 ` Johannes Weiner 2014-06-04 19:18 ` Hugh Dickins 2014-06-04 19:18 ` Hugh Dickins 2014-06-04 21:45 ` Johannes Weiner 2014-06-04 21:45 ` Johannes Weiner 2014-06-05 14:51 ` Michal Hocko 2014-06-05 14:51 ` Michal Hocko 2014-06-05 16:10 ` Johannes Weiner 2014-06-05 16:10 ` Johannes Weiner 2014-06-05 16:43 ` Michal Hocko 2014-06-05 16:43 ` Michal Hocko 2014-06-05 18:23 ` Johannes Weiner [this message] 2014-06-05 18:23 ` Johannes Weiner 2014-06-06 14:44 ` Michal Hocko 2014-06-06 14:44 ` Michal Hocko 2014-06-06 14:46 ` [PATCH 1/2] mm, memcg: allow OOM if no memcg is eligible during direct reclaim Michal Hocko 2014-06-06 14:46 ` Michal Hocko 2014-06-06 14:46 ` [PATCH 2/2] memcg: Allow hard guarantee mode for low limit reclaim Michal Hocko 2014-06-06 14:46 ` Michal Hocko 2014-06-06 15:29 ` Tejun Heo 2014-06-06 15:29 ` Tejun Heo 2014-06-06 15:34 ` Tejun Heo 2014-06-06 15:34 ` Tejun Heo 2014-06-09 8:30 ` Michal Hocko 2014-06-09 8:30 ` Michal Hocko 2014-06-09 13:54 ` Tejun Heo 2014-06-09 13:54 ` Tejun Heo 2014-06-09 22:52 ` Greg Thelen 2014-06-09 22:52 ` Greg Thelen 2014-06-10 16:57 ` Johannes Weiner 2014-06-10 16:57 ` Johannes Weiner 2014-06-10 22:16 ` Greg Thelen 2014-06-10 22:16 ` Greg Thelen 2014-06-11 7:57 ` Michal Hocko 2014-06-11 7:57 ` Michal Hocko 2014-06-11 8:00 ` [PATCH 1/2] mm, memcg: allow OOM if no memcg is eligible during direct reclaim Michal Hocko 2014-06-11 8:00 ` Michal Hocko 2014-06-11 8:00 ` [PATCH 2/2] memcg: Allow guarantee reclaim Michal Hocko 2014-06-11 8:00 ` Michal Hocko 2014-06-11 15:36 ` Johannes Weiner 2014-06-11 15:36 ` Johannes Weiner 2014-06-12 13:22 ` Michal Hocko 2014-06-12 13:22 ` Michal Hocko 2014-06-12 13:56 ` Johannes Weiner 2014-06-12 13:56 ` Johannes Weiner 2014-06-12 14:22 ` Michal Hocko 2014-06-12 14:22 ` Michal Hocko 2014-06-12 16:17 ` Tejun Heo 2014-06-12 16:17 ` Tejun Heo 2014-06-16 12:59 ` Michal Hocko 2014-06-16 12:59 ` Michal Hocko 2014-06-16 13:57 ` Tejun Heo 2014-06-16 13:57 ` Tejun Heo 2014-06-16 14:04 ` Michal Hocko 2014-06-16 14:04 ` Michal Hocko 2014-06-16 14:12 ` Tejun Heo 2014-06-16 14:12 ` Tejun Heo 2014-06-16 14:29 ` Michal Hocko 2014-06-16 14:29 ` Michal Hocko 2014-06-16 14:40 ` Tejun Heo 2014-06-16 14:40 ` Tejun Heo 2014-06-12 16:51 ` Johannes Weiner 2014-06-12 16:51 ` Johannes Weiner 2014-06-16 13:22 ` Michal Hocko 2014-06-16 13:22 ` Michal Hocko 2014-06-11 15:20 ` [PATCH 1/2] mm, memcg: allow OOM if no memcg is eligible during direct reclaim Johannes Weiner 2014-06-11 15:20 ` Johannes Weiner 2014-06-11 16:14 ` Michal Hocko 2014-06-11 16:14 ` Michal Hocko 2014-06-11 12:31 ` [PATCH 2/2] memcg: Allow hard guarantee mode for low limit reclaim Tejun Heo 2014-06-11 12:31 ` Tejun Heo 2014-06-11 14:11 ` Michal Hocko 2014-06-11 14:11 ` Michal Hocko 2014-06-11 15:34 ` Tejun Heo 2014-06-11 15:34 ` Tejun Heo 2014-06-05 19:36 ` [PATCH v2 0/4] memcg: Low-limit reclaim Tejun Heo 2014-06-05 19:36 ` Tejun Heo 2014-06-05 14:32 ` Michal Hocko 2014-06-05 14:32 ` Michal Hocko 2014-06-05 15:43 ` Johannes Weiner 2014-06-05 15:43 ` Johannes Weiner 2014-06-05 16:09 ` Michal Hocko 2014-06-05 16:09 ` Michal Hocko 2014-06-05 16:46 ` Johannes Weiner 2014-06-05 16:46 ` Johannes Weiner 2014-05-28 16:17 ` Greg Thelen 2014-05-28 16:17 ` Greg Thelen 2014-06-03 11:09 ` Michal Hocko 2014-06-03 11:09 ` Michal Hocko 2014-06-03 14:01 ` Greg Thelen 2014-06-03 14:44 ` Michal Hocko 2014-06-03 14:44 ` Michal Hocko
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20140605182336.GA2878@cmpxchg.org \ --to=hannes@cmpxchg.org \ --cc=akpm@linux-foundation.org \ --cc=gthelen@google.com \ --cc=hughd@google.com \ --cc=kamezawa.hiroyu@jp.fujitsu.com \ --cc=klamm@yandex-team.ru \ --cc=kosaki.motohiro@jp.fujitsu.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mhocko@suse.cz \ --cc=riel@redhat.com \ --cc=tj@kernel.org \ --cc=walken@google.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.