* [PATCH] drm/i915: Handle full s64 precision for wait-ioctl
@ 2017-08-05 19:19 Chris Wilson
2017-08-05 19:47 ` Chris Wilson
` (3 more replies)
0 siblings, 4 replies; 8+ messages in thread
From: Chris Wilson @ 2017-08-05 19:19 UTC (permalink / raw)
To: intel-gfx; +Cc: Daniel Vetter
The wait-ioctl is optionally supplied a timeout with nanosecond
precision in a s64 field. We use nsecs_to_jiffies64() to convert that
into the jiffies consumed by the scheduler, but internally
nsecs_to_jiffies64() does not guard against overflow (as it's purpose is
for use by the scheduler and not drivers!). So we must guard against the
overflow ourselves, and in the process note that we may then return
much earlier than the timeout selected by the user, so don't report
ETIME unless we do hit the timeout. (Woe betold us though if the user
waits for a year (32bit) and the request is still not complete!)
Reported-by: Jason Ekstrand <jason.ekstrand@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
---
drivers/gpu/drm/i915/i915_drv.h | 6 ++++++
drivers/gpu/drm/i915/i915_gem.c | 4 ++++
2 files changed, 10 insertions(+)
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 2638967211a9..184f4d11de79 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -4144,6 +4144,12 @@ static inline unsigned long msecs_to_jiffies_timeout(const unsigned int m)
static inline unsigned long nsecs_to_jiffies_timeout(const u64 n)
{
+#if NSEC_PER_SEC % HZ
+ /* nsecs_to_jiffies64() does not guard against overflow */
+ if (n >= (u64)MAX_JIFFY_OFFSET * NSEC_PER_SEC / HZ)
+ return MAX_JIFFY_OFFSET;
+#endif
+
return min_t(u64, MAX_JIFFY_OFFSET, nsecs_to_jiffies64(n) + 1);
}
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 44df7dc3f880..b5794add4a3a 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -3543,6 +3543,10 @@ i915_gem_wait_ioctl(struct file *filp,
*/
if (ret == -ETIME && !nsecs_to_jiffies(arg.timeout_ns))
arg.timeout_ns = 0;
+
+ /* Asked to wait beyond the jiffie/scheduler precision */
+ if (ret == -ETIME && arg.timeout_ns)
+ ret = -EAGAIN;
}
i915_gem_object_put(obj);
--
2.13.3
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH] drm/i915: Handle full s64 precision for wait-ioctl
2017-08-05 19:19 [PATCH] drm/i915: Handle full s64 precision for wait-ioctl Chris Wilson
@ 2017-08-05 19:47 ` Chris Wilson
2017-08-11 7:18 ` Joonas Lahtinen
2017-08-11 7:13 ` Joonas Lahtinen
` (2 subsequent siblings)
3 siblings, 1 reply; 8+ messages in thread
From: Chris Wilson @ 2017-08-05 19:47 UTC (permalink / raw)
To: intel-gfx; +Cc: Daniel Vetter
Quoting Chris Wilson (2017-08-05 20:19:24)
> The wait-ioctl is optionally supplied a timeout with nanosecond
> precision in a s64 field. We use nsecs_to_jiffies64() to convert that
> into the jiffies consumed by the scheduler, but internally
> nsecs_to_jiffies64() does not guard against overflow (as it's purpose is
> for use by the scheduler and not drivers!). So we must guard against the
> overflow ourselves, and in the process note that we may then return
> much earlier than the timeout selected by the user, so don't report
> ETIME unless we do hit the timeout. (Woe betold us though if the user
> waits for a year (32bit) and the request is still not complete!)
>
> Reported-by: Jason Ekstrand <jason.ekstrand@intel.com>
> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> ---
> drivers/gpu/drm/i915/i915_drv.h | 6 ++++++
> drivers/gpu/drm/i915/i915_gem.c | 4 ++++
> 2 files changed, 10 insertions(+)
>
> diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
> index 2638967211a9..184f4d11de79 100644
> --- a/drivers/gpu/drm/i915/i915_drv.h
> +++ b/drivers/gpu/drm/i915/i915_drv.h
> @@ -4144,6 +4144,12 @@ static inline unsigned long msecs_to_jiffies_timeout(const unsigned int m)
>
> static inline unsigned long nsecs_to_jiffies_timeout(const u64 n)
> {
> +#if NSEC_PER_SEC % HZ
> + /* nsecs_to_jiffies64() does not guard against overflow */
> + if (n >= (u64)MAX_JIFFY_OFFSET * NSEC_PER_SEC / HZ)
> + return MAX_JIFFY_OFFSET;
This still overflows, we need n / NSEC_PER_SEC >= MAX_JIFFY_OFFSET / HZ
as MAX_JIFFY_OFFSET is ~LONG_MAX/2
Hmm, so div_u64(n, NSEC_PER_SEC) >= MAX_JIFFY_OFFSET / HZ ?
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] drm/i915: Handle full s64 precision for wait-ioctl
2017-08-05 19:19 [PATCH] drm/i915: Handle full s64 precision for wait-ioctl Chris Wilson
2017-08-05 19:47 ` Chris Wilson
@ 2017-08-11 7:13 ` Joonas Lahtinen
2017-08-11 10:57 ` [PATCH v2] " Chris Wilson
2017-08-11 11:27 ` ✓ Fi.CI.BAT: success for drm/i915: Handle full s64 precision for wait-ioctl (rev2) Patchwork
3 siblings, 0 replies; 8+ messages in thread
From: Joonas Lahtinen @ 2017-08-11 7:13 UTC (permalink / raw)
To: Chris Wilson, intel-gfx; +Cc: Daniel Vetter
On la, 2017-08-05 at 20:19 +0100, Chris Wilson wrote:
> The wait-ioctl is optionally supplied a timeout with nanosecond
> precision in a s64 field. We use nsecs_to_jiffies64() to convert that
> into the jiffies consumed by the scheduler, but internally
> nsecs_to_jiffies64() does not guard against overflow (as it's purpose is
> for use by the scheduler and not drivers!). So we must guard against the
> overflow ourselves, and in the process note that we may then return
> much earlier than the timeout selected by the user, so don't report
> ETIME unless we do hit the timeout. (Woe betold us though if the user
> waits for a year (32bit) and the request is still not complete!)
>
> Reported-by: Jason Ekstrand <jason.ekstrand@intel.com>
> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
<SNIP>
> diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
> index 44df7dc3f880..b5794add4a3a 100644
> --- a/drivers/gpu/drm/i915/i915_gem.c
> +++ b/drivers/gpu/drm/i915/i915_gem.c
> @@ -3543,6 +3543,10 @@ i915_gem_wait_ioctl(struct file *filp,
> */
> if (ret == -ETIME && !nsecs_to_jiffies(arg.timeout_ns))
> arg.timeout_ns = 0;
> +
> + /* Asked to wait beyond the jiffie/scheduler precision */
> + if (ret == -ETIME && arg.timeout_ns)
> + ret = -EAGAIN;
-EAGAIN is documented as "GPU wedged" in the ioctl documentation. So
better update that documentation.
Regards, Joonas
--
Joonas Lahtinen
Open Source Technology Center
Intel Corporation
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] drm/i915: Handle full s64 precision for wait-ioctl
2017-08-05 19:47 ` Chris Wilson
@ 2017-08-11 7:18 ` Joonas Lahtinen
0 siblings, 0 replies; 8+ messages in thread
From: Joonas Lahtinen @ 2017-08-11 7:18 UTC (permalink / raw)
To: Chris Wilson, intel-gfx; +Cc: Daniel Vetter
On la, 2017-08-05 at 20:47 +0100, Chris Wilson wrote:
> Quoting Chris Wilson (2017-08-05 20:19:24)
<SNIP>
> > +++ b/drivers/gpu/drm/i915/i915_drv.h
> > @@ -4144,6 +4144,12 @@ static inline unsigned long msecs_to_jiffies_timeout(const unsigned int m)
> >
> > static inline unsigned long nsecs_to_jiffies_timeout(const u64 n)
> > {
> > +#if NSEC_PER_SEC % HZ
> > + /* nsecs_to_jiffies64() does not guard against overflow */
> > + if (n >= (u64)MAX_JIFFY_OFFSET * NSEC_PER_SEC / HZ)
> > + return MAX_JIFFY_OFFSET;
>
> This still overflows, we need n / NSEC_PER_SEC >= MAX_JIFFY_OFFSET / HZ
> as MAX_JIFFY_OFFSET is ~LONG_MAX/2
>
> Hmm, so div_u64(n, NSEC_PER_SEC) >= MAX_JIFFY_OFFSET / HZ ?
This reflects the original test best.
Regards, Joonas
--
Joonas Lahtinen
Open Source Technology Center
Intel Corporation
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH v2] drm/i915: Handle full s64 precision for wait-ioctl
2017-08-05 19:19 [PATCH] drm/i915: Handle full s64 precision for wait-ioctl Chris Wilson
2017-08-05 19:47 ` Chris Wilson
2017-08-11 7:13 ` Joonas Lahtinen
@ 2017-08-11 10:57 ` Chris Wilson
2017-08-14 10:42 ` Joonas Lahtinen
2017-08-11 11:27 ` ✓ Fi.CI.BAT: success for drm/i915: Handle full s64 precision for wait-ioctl (rev2) Patchwork
3 siblings, 1 reply; 8+ messages in thread
From: Chris Wilson @ 2017-08-11 10:57 UTC (permalink / raw)
To: intel-gfx; +Cc: Daniel Vetter
The wait-ioctl is optionally supplied a timeout with nanosecond
precision in a s64 field. We use nsecs_to_jiffies64() to convert that
into the jiffies consumed by the scheduler, but internally
nsecs_to_jiffies64() does not guard against overflow (as it's purpose is
for use by the scheduler and not drivers!). So we must guard against the
overflow ourselves, and in the process note that we may then return
much earlier than the timeout selected by the user, so don't report
ETIME unless we do hit the timeout. (Woe betold us though if the user
waits for a year (32bit) and the request is still not complete!)
v2: Refine overflow detection (to not include an overffow itself)
Reported-by: Jason Ekstrand <jason.ekstrand@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
---
drivers/gpu/drm/i915/i915_drv.h | 5 +++++
drivers/gpu/drm/i915/i915_gem.c | 6 +++++-
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 907603cba447..4f9f7b6ac276 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -4171,6 +4171,11 @@ static inline unsigned long msecs_to_jiffies_timeout(const unsigned int m)
static inline unsigned long nsecs_to_jiffies_timeout(const u64 n)
{
+ /* nsecs_to_jiffies64() does not guard against overflow */
+ if (NSEC_PER_SEC % HZ &&
+ div_u64(n, NSEC_PER_SEC) >= MAX_JIFFY_OFFSET / HZ)
+ return MAX_JIFFY_OFFSET;
+
return min_t(u64, MAX_JIFFY_OFFSET, nsecs_to_jiffies64(n) + 1);
}
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 000a764ee8d9..dbda7d078245 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -3284,7 +3284,7 @@ static unsigned long to_wait_timeout(s64 timeout_ns)
* -ERESTARTSYS: signal interrupted the wait
* -ENONENT: object doesn't exist
* Also possible, but rare:
- * -EAGAIN: GPU wedged
+ * -EAGAIN: incomplete, restart syscall
* -ENOMEM: damn
* -ENODEV: Internal IRQ fail
* -E?: The add request failed
@@ -3332,6 +3332,10 @@ i915_gem_wait_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
*/
if (ret == -ETIME && !nsecs_to_jiffies(args->timeout_ns))
args->timeout_ns = 0;
+
+ /* Asked to wait beyond the jiffie/scheduler precision? */
+ if (ret == -ETIME && args->timeout_ns)
+ ret = -EAGAIN;
}
i915_gem_object_put(obj);
--
2.13.3
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 8+ messages in thread
* ✓ Fi.CI.BAT: success for drm/i915: Handle full s64 precision for wait-ioctl (rev2)
2017-08-05 19:19 [PATCH] drm/i915: Handle full s64 precision for wait-ioctl Chris Wilson
` (2 preceding siblings ...)
2017-08-11 10:57 ` [PATCH v2] " Chris Wilson
@ 2017-08-11 11:27 ` Patchwork
3 siblings, 0 replies; 8+ messages in thread
From: Patchwork @ 2017-08-11 11:27 UTC (permalink / raw)
To: Chris Wilson; +Cc: intel-gfx
== Series Details ==
Series: drm/i915: Handle full s64 precision for wait-ioctl (rev2)
URL : https://patchwork.freedesktop.org/series/28420/
State : success
== Summary ==
Series 28420v2 drm/i915: Handle full s64 precision for wait-ioctl
https://patchwork.freedesktop.org/api/1.0/series/28420/revisions/2/mbox/
Test gem_exec_flush:
Subgroup basic-batch-kernel-default-uc:
pass -> FAIL (fi-snb-2600) fdo#100007
Test kms_flip:
Subgroup basic-flip-vs-modeset:
skip -> PASS (fi-skl-x1585l) fdo#101781
Test kms_pipe_crc_basic:
Subgroup suspend-read-crc-pipe-b:
dmesg-warn -> PASS (fi-byt-n2820) fdo#101705
fdo#100007 https://bugs.freedesktop.org/show_bug.cgi?id=100007
fdo#101781 https://bugs.freedesktop.org/show_bug.cgi?id=101781
fdo#101705 https://bugs.freedesktop.org/show_bug.cgi?id=101705
fi-bdw-5557u total:279 pass:268 dwarn:0 dfail:0 fail:0 skip:11 time:444s
fi-bdw-gvtdvm total:279 pass:265 dwarn:0 dfail:0 fail:0 skip:14 time:431s
fi-blb-e6850 total:279 pass:224 dwarn:1 dfail:0 fail:0 skip:54 time:355s
fi-bsw-n3050 total:279 pass:243 dwarn:0 dfail:0 fail:0 skip:36 time:546s
fi-bxt-j4205 total:279 pass:260 dwarn:0 dfail:0 fail:0 skip:19 time:523s
fi-byt-j1900 total:279 pass:254 dwarn:1 dfail:0 fail:0 skip:24 time:522s
fi-byt-n2820 total:279 pass:251 dwarn:0 dfail:0 fail:0 skip:28 time:508s
fi-glk-2a total:279 pass:260 dwarn:0 dfail:0 fail:0 skip:19 time:602s
fi-hsw-4770 total:279 pass:263 dwarn:0 dfail:0 fail:0 skip:16 time:445s
fi-hsw-4770r total:279 pass:263 dwarn:0 dfail:0 fail:0 skip:16 time:419s
fi-ilk-650 total:279 pass:229 dwarn:0 dfail:0 fail:0 skip:50 time:421s
fi-ivb-3520m total:279 pass:261 dwarn:0 dfail:0 fail:0 skip:18 time:497s
fi-ivb-3770 total:279 pass:261 dwarn:0 dfail:0 fail:0 skip:18 time:474s
fi-kbl-7500u total:279 pass:261 dwarn:0 dfail:0 fail:0 skip:18 time:467s
fi-kbl-7560u total:279 pass:269 dwarn:0 dfail:0 fail:0 skip:10 time:586s
fi-kbl-r total:279 pass:261 dwarn:0 dfail:0 fail:0 skip:18 time:592s
fi-pnv-d510 total:279 pass:223 dwarn:1 dfail:0 fail:0 skip:55 time:529s
fi-skl-6260u total:279 pass:269 dwarn:0 dfail:0 fail:0 skip:10 time:461s
fi-skl-6700k total:279 pass:261 dwarn:0 dfail:0 fail:0 skip:18 time:475s
fi-skl-6770hq total:279 pass:269 dwarn:0 dfail:0 fail:0 skip:10 time:477s
fi-skl-gvtdvm total:279 pass:266 dwarn:0 dfail:0 fail:0 skip:13 time:438s
fi-skl-x1585l total:279 pass:269 dwarn:0 dfail:0 fail:0 skip:10 time:499s
fi-snb-2520m total:279 pass:251 dwarn:0 dfail:0 fail:0 skip:28 time:547s
fi-snb-2600 total:279 pass:249 dwarn:0 dfail:0 fail:1 skip:29 time:405s
fbb8288699ef622bbfc6e10bdca6773a16f93fac drm-tip: 2017y-08m-11d-09h-03m-47s UTC integration manifest
b87953debf0e drm/i915: Handle full s64 precision for wait-ioctl
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_5379/
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2] drm/i915: Handle full s64 precision for wait-ioctl
2017-08-11 10:57 ` [PATCH v2] " Chris Wilson
@ 2017-08-14 10:42 ` Joonas Lahtinen
2017-08-15 16:41 ` Chris Wilson
0 siblings, 1 reply; 8+ messages in thread
From: Joonas Lahtinen @ 2017-08-14 10:42 UTC (permalink / raw)
To: Chris Wilson, intel-gfx; +Cc: Daniel Vetter
On Fri, 2017-08-11 at 11:57 +0100, Chris Wilson wrote:
> The wait-ioctl is optionally supplied a timeout with nanosecond
> precision in a s64 field. We use nsecs_to_jiffies64() to convert that
> into the jiffies consumed by the scheduler, but internally
> nsecs_to_jiffies64() does not guard against overflow (as it's purpose is
> for use by the scheduler and not drivers!). So we must guard against the
> overflow ourselves, and in the process note that we may then return
> much earlier than the timeout selected by the user, so don't report
> ETIME unless we do hit the timeout. (Woe betold us though if the user
> waits for a year (32bit) and the request is still not complete!)
>
> v2: Refine overflow detection (to not include an overffow itself)
>
> Reported-by: Jason Ekstrand <jason.ekstrand@intel.com>
> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Regards, Joonas
--
Joonas Lahtinen
Open Source Technology Center
Intel Corporation
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2] drm/i915: Handle full s64 precision for wait-ioctl
2017-08-14 10:42 ` Joonas Lahtinen
@ 2017-08-15 16:41 ` Chris Wilson
0 siblings, 0 replies; 8+ messages in thread
From: Chris Wilson @ 2017-08-15 16:41 UTC (permalink / raw)
To: Joonas Lahtinen, intel-gfx; +Cc: Daniel Vetter
Quoting Joonas Lahtinen (2017-08-14 11:42:05)
> On Fri, 2017-08-11 at 11:57 +0100, Chris Wilson wrote:
> > The wait-ioctl is optionally supplied a timeout with nanosecond
> > precision in a s64 field. We use nsecs_to_jiffies64() to convert that
> > into the jiffies consumed by the scheduler, but internally
> > nsecs_to_jiffies64() does not guard against overflow (as it's purpose is
> > for use by the scheduler and not drivers!). So we must guard against the
> > overflow ourselves, and in the process note that we may then return
> > much earlier than the timeout selected by the user, so don't report
> > ETIME unless we do hit the timeout. (Woe betold us though if the user
> > waits for a year (32bit) and the request is still not complete!)
> >
> > v2: Refine overflow detection (to not include an overffow itself)
> >
> > Reported-by: Jason Ekstrand <jason.ekstrand@intel.com>
> > Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> > Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
> > Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
>
> Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
And pushed, thanks for the review.
-Chris
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2017-08-15 16:42 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-08-05 19:19 [PATCH] drm/i915: Handle full s64 precision for wait-ioctl Chris Wilson
2017-08-05 19:47 ` Chris Wilson
2017-08-11 7:18 ` Joonas Lahtinen
2017-08-11 7:13 ` Joonas Lahtinen
2017-08-11 10:57 ` [PATCH v2] " Chris Wilson
2017-08-14 10:42 ` Joonas Lahtinen
2017-08-15 16:41 ` Chris Wilson
2017-08-11 11:27 ` ✓ Fi.CI.BAT: success for drm/i915: Handle full s64 precision for wait-ioctl (rev2) Patchwork
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.