* [PATCH bpf-next] bpf,sockmap: fix sk->sk_forward_alloc warn_on in sk_stream_kill_queues
@ 2022-05-24 7:53 Wang Yufen
2022-05-25 8:41 ` Jakub Sitnicki
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Wang Yufen @ 2022-05-24 7:53 UTC (permalink / raw)
To: ast, john.fastabend, andrii, daniel, jakub, lmb, davem, kafai,
dsahern, kuba, songliubraving, yhs, kpsingh
Cc: netdev, bpf, Wang Yufen
During TCP sockmap redirect pressure test, the following warning is triggered:
WARNING: CPU: 3 PID: 2145 at net/core/stream.c:205 sk_stream_kill_queues+0xbc/0xd0
CPU: 3 PID: 2145 Comm: iperf Kdump: loaded Tainted: G W 5.10.0+ #9
Call Trace:
inet_csk_destroy_sock+0x55/0x110
inet_csk_listen_stop+0xbb/0x380
tcp_close+0x41b/0x480
inet_release+0x42/0x80
__sock_release+0x3d/0xa0
sock_close+0x11/0x20
__fput+0x9d/0x240
task_work_run+0x62/0x90
exit_to_user_mode_prepare+0x110/0x120
syscall_exit_to_user_mode+0x27/0x190
entry_SYSCALL_64_after_hwframe+0x44/0xa9
The reason we observed is that:
When the listener is closing, a connection may have completed the three-way
handshake but not accepted, and the client has sent some packets. The child
sks in accept queue release by inet_child_forget()->inet_csk_destroy_sock(),
but psocks of child sks have not released.
To fix, add sock_map_destroy to release psocks.
Signed-off-by: Wang Yufen <wangyufen@huawei.com>
---
include/linux/bpf.h | 1 +
include/linux/skmsg.h | 1 +
net/core/skmsg.c | 1 +
net/core/sock_map.c | 23 +++++++++++++++++++++++
net/ipv4/tcp_bpf.c | 1 +
5 files changed, 27 insertions(+)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index cc4d5e394031..c4de82d5e72c 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -2092,6 +2092,7 @@ int sock_map_bpf_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr);
void sock_map_unhash(struct sock *sk);
+void sock_map_destroy(struct sock *sk);
void sock_map_close(struct sock *sk, long timeout);
#else
static inline int bpf_prog_offload_init(struct bpf_prog *prog,
diff --git a/include/linux/skmsg.h b/include/linux/skmsg.h
index c5a2d6f50f25..153b6dec9b6a 100644
--- a/include/linux/skmsg.h
+++ b/include/linux/skmsg.h
@@ -95,6 +95,7 @@ struct sk_psock {
spinlock_t link_lock;
refcount_t refcnt;
void (*saved_unhash)(struct sock *sk);
+ void (*saved_destroy)(struct sock *sk);
void (*saved_close)(struct sock *sk, long timeout);
void (*saved_write_space)(struct sock *sk);
void (*saved_data_ready)(struct sock *sk);
diff --git a/net/core/skmsg.c b/net/core/skmsg.c
index 22b983ade0e7..7e03f96e441b 100644
--- a/net/core/skmsg.c
+++ b/net/core/skmsg.c
@@ -715,6 +715,7 @@ struct sk_psock *sk_psock_init(struct sock *sk, int node)
psock->eval = __SK_NONE;
psock->sk_proto = prot;
psock->saved_unhash = prot->unhash;
+ psock->saved_destroy = prot->destroy;
psock->saved_close = prot->close;
psock->saved_write_space = sk->sk_write_space;
diff --git a/net/core/sock_map.c b/net/core/sock_map.c
index 81d4b4756a02..9f08ccfaf6da 100644
--- a/net/core/sock_map.c
+++ b/net/core/sock_map.c
@@ -1561,6 +1561,29 @@ void sock_map_unhash(struct sock *sk)
}
EXPORT_SYMBOL_GPL(sock_map_unhash);
+void sock_map_destroy(struct sock *sk)
+{
+ void (*saved_destroy)(struct sock *sk);
+ struct sk_psock *psock;
+
+ rcu_read_lock();
+ psock = sk_psock_get(sk);
+ if (unlikely(!psock)) {
+ rcu_read_unlock();
+ if (sk->sk_prot->destroy)
+ sk->sk_prot->destroy(sk);
+ return;
+ }
+
+ saved_destroy = psock->saved_destroy;
+ sock_map_remove_links(sk, psock);
+ rcu_read_unlock();
+ sk_psock_stop(psock, true);
+ sk_psock_put(sk, psock);
+ saved_destroy(sk);
+}
+EXPORT_SYMBOL_GPL(sock_map_destroy);
+
void sock_map_close(struct sock *sk, long timeout)
{
void (*saved_close)(struct sock *sk, long timeout);
diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c
index be3947e70fec..38550bb1b90b 100644
--- a/net/ipv4/tcp_bpf.c
+++ b/net/ipv4/tcp_bpf.c
@@ -540,6 +540,7 @@ static void tcp_bpf_rebuild_protos(struct proto prot[TCP_BPF_NUM_CFGS],
struct proto *base)
{
prot[TCP_BPF_BASE] = *base;
+ prot[TCP_BPF_BASE].destroy = sock_map_destroy;
prot[TCP_BPF_BASE].close = sock_map_close;
prot[TCP_BPF_BASE].recvmsg = tcp_bpf_recvmsg;
prot[TCP_BPF_BASE].sock_is_readable = sk_msg_is_readable;
--
2.25.1
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH bpf-next] bpf,sockmap: fix sk->sk_forward_alloc warn_on in sk_stream_kill_queues
2022-05-24 7:53 [PATCH bpf-next] bpf,sockmap: fix sk->sk_forward_alloc warn_on in sk_stream_kill_queues Wang Yufen
@ 2022-05-25 8:41 ` Jakub Sitnicki
2022-05-27 21:37 ` Cong Wang
2022-05-31 23:50 ` patchwork-bot+netdevbpf
2 siblings, 0 replies; 7+ messages in thread
From: Jakub Sitnicki @ 2022-05-25 8:41 UTC (permalink / raw)
To: Wang Yufen
Cc: ast, john.fastabend, andrii, daniel, lmb, davem, kafai, dsahern,
kuba, songliubraving, yhs, kpsingh, netdev, bpf
On Tue, May 24, 2022 at 03:53 PM +08, Wang Yufen wrote:
> During TCP sockmap redirect pressure test, the following warning is triggered:
> WARNING: CPU: 3 PID: 2145 at net/core/stream.c:205 sk_stream_kill_queues+0xbc/0xd0
> CPU: 3 PID: 2145 Comm: iperf Kdump: loaded Tainted: G W 5.10.0+ #9
> Call Trace:
> inet_csk_destroy_sock+0x55/0x110
> inet_csk_listen_stop+0xbb/0x380
> tcp_close+0x41b/0x480
> inet_release+0x42/0x80
> __sock_release+0x3d/0xa0
> sock_close+0x11/0x20
> __fput+0x9d/0x240
> task_work_run+0x62/0x90
> exit_to_user_mode_prepare+0x110/0x120
> syscall_exit_to_user_mode+0x27/0x190
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> The reason we observed is that:
> When the listener is closing, a connection may have completed the three-way
> handshake but not accepted, and the client has sent some packets. The child
> sks in accept queue release by inet_child_forget()->inet_csk_destroy_sock(),
> but psocks of child sks have not released.
>
> To fix, add sock_map_destroy to release psocks.
>
> Signed-off-by: Wang Yufen <wangyufen@huawei.com>
> ---
Thanks for the fix.
Acked-by: Jakub Sitnicki <jakub@cloudflare.com>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH bpf-next] bpf,sockmap: fix sk->sk_forward_alloc warn_on in sk_stream_kill_queues
2022-05-24 7:53 [PATCH bpf-next] bpf,sockmap: fix sk->sk_forward_alloc warn_on in sk_stream_kill_queues Wang Yufen
2022-05-25 8:41 ` Jakub Sitnicki
@ 2022-05-27 21:37 ` Cong Wang
2022-05-28 1:54 ` wangyufen
2022-05-31 23:50 ` patchwork-bot+netdevbpf
2 siblings, 1 reply; 7+ messages in thread
From: Cong Wang @ 2022-05-27 21:37 UTC (permalink / raw)
To: Wang Yufen
Cc: ast, john.fastabend, andrii, daniel, jakub, lmb, davem, kafai,
dsahern, kuba, songliubraving, yhs, kpsingh, netdev, bpf
On Tue, May 24, 2022 at 03:53:11PM +0800, Wang Yufen wrote:
> During TCP sockmap redirect pressure test, the following warning is triggered:
> WARNING: CPU: 3 PID: 2145 at net/core/stream.c:205 sk_stream_kill_queues+0xbc/0xd0
> CPU: 3 PID: 2145 Comm: iperf Kdump: loaded Tainted: G W 5.10.0+ #9
> Call Trace:
> inet_csk_destroy_sock+0x55/0x110
> inet_csk_listen_stop+0xbb/0x380
> tcp_close+0x41b/0x480
> inet_release+0x42/0x80
> __sock_release+0x3d/0xa0
> sock_close+0x11/0x20
> __fput+0x9d/0x240
> task_work_run+0x62/0x90
> exit_to_user_mode_prepare+0x110/0x120
> syscall_exit_to_user_mode+0x27/0x190
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> The reason we observed is that:
> When the listener is closing, a connection may have completed the three-way
> handshake but not accepted, and the client has sent some packets. The child
> sks in accept queue release by inet_child_forget()->inet_csk_destroy_sock(),
> but psocks of child sks have not released.
>
Hm, in this scenario, how does the child socket end up in the sockmap?
Clearly user-space does not have a chance to get an fd yet.
And, how does your patch work? Since the child sock does not even inheirt
the sock proto after clone (see the comments above tcp_bpf_clone()) at
all?
Thanks.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH bpf-next] bpf,sockmap: fix sk->sk_forward_alloc warn_on in sk_stream_kill_queues
2022-05-27 21:37 ` Cong Wang
@ 2022-05-28 1:54 ` wangyufen
2022-05-30 16:37 ` Jakub Sitnicki
0 siblings, 1 reply; 7+ messages in thread
From: wangyufen @ 2022-05-28 1:54 UTC (permalink / raw)
To: Cong Wang
Cc: ast, john.fastabend, andrii, daniel, jakub, lmb, davem, kafai,
dsahern, kuba, songliubraving, yhs, kpsingh, netdev, bpf
在 2022/5/28 5:37, Cong Wang 写道:
> On Tue, May 24, 2022 at 03:53:11PM +0800, Wang Yufen wrote:
>> During TCP sockmap redirect pressure test, the following warning is triggered:
>> WARNING: CPU: 3 PID: 2145 at net/core/stream.c:205 sk_stream_kill_queues+0xbc/0xd0
>> CPU: 3 PID: 2145 Comm: iperf Kdump: loaded Tainted: G W 5.10.0+ #9
>> Call Trace:
>> inet_csk_destroy_sock+0x55/0x110
>> inet_csk_listen_stop+0xbb/0x380
>> tcp_close+0x41b/0x480
>> inet_release+0x42/0x80
>> __sock_release+0x3d/0xa0
>> sock_close+0x11/0x20
>> __fput+0x9d/0x240
>> task_work_run+0x62/0x90
>> exit_to_user_mode_prepare+0x110/0x120
>> syscall_exit_to_user_mode+0x27/0x190
>> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>
>> The reason we observed is that:
>> When the listener is closing, a connection may have completed the three-way
>> handshake but not accepted, and the client has sent some packets. The child
>> sks in accept queue release by inet_child_forget()->inet_csk_destroy_sock(),
>> but psocks of child sks have not released.
>>
> Hm, in this scenario, how does the child socket end up in the sockmap?
> Clearly user-space does not have a chance to get an fd yet.
>
> And, how does your patch work? Since the child sock does not even inheirt
> the sock proto after clone (see the comments above tcp_bpf_clone()) at
> all?
>
> Thanks.
> .
My test cases are as follows:
__section("sockops")
int bpf_sockmap(struct bpf_sock_ops *skops)
{
switch (skops->op) {
case BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB:
case BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB:
...
bpf_sock_hash_update(skops, &sock_ops_map, &key, BPF_NOEXIST);
break;
...
}
__section("sk_msg")
int bpf_redir(struct sk_msg_md *msg)
{
...
bpf_msg_redirect_hash(msg, &sock_ops_map, &key, BPF_F_INGRESS);
return SK_PASS;
}
//tcp_server
int main(char **argv)
{
int sk = 0;
int port, ret;
struct sockaddr_in addr;
signal(SIGCHLD, SIG_IGN);
sk = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if (sk < 0) {
perror("Can't create socket");
return -1;
}
port = atoi(argv[1]);
memset(&addr, 0, sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = htonl(INADDR_ANY);
addr.sin_port = htons(port);
printf("Binding to port %d\n", port);
ret = bind(sk, (struct sockaddr *)&addr, sizeof(addr));
if (ret < 0) {
perror("Can't bind socket");
return -1;
}
ret = listen(sk, size);
if (ret < 0) {
perror("Can't put sock to listen");
return -1;
}
printf("Waiting for connections\n");
while (1) {
//not accpet
sleep(1);
}
}
//tcp_client
int main(char **argv)
{
int port, write_size;
int val[10], rval[10];
int sk = 0;
port = atoi(argv[2]);
val[0] = 1;
sk = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if (sk < 0) {
perror("Can't create socket");
return -1;
}
memset(&addr, 0, sizeof(addr));
addr.sin_family = AF_INET;
inet_aton(argv[1], &addr.sin_addr);
addr.sin_port = htons(port);
ret = connect(sk[i], (struct sockaddr *)&addr, sizeof(addr));
if (ret < 0) {
perror("Can't connect");
return -1;
}
while (1) {
printf("send %d -> %d\n", val[0], val[0]);
write(sk, &val, sizeof(val));
val[0]++;
sleep(1);
}
}
1. start tcp_server
2. start tcp_client
3. kill tcp_server
The problem can be reproduced easily.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH bpf-next] bpf,sockmap: fix sk->sk_forward_alloc warn_on in sk_stream_kill_queues
2022-05-28 1:54 ` wangyufen
@ 2022-05-30 16:37 ` Jakub Sitnicki
2022-05-31 23:40 ` John Fastabend
0 siblings, 1 reply; 7+ messages in thread
From: Jakub Sitnicki @ 2022-05-30 16:37 UTC (permalink / raw)
To: Cong Wang, wangyufen
Cc: ast, john.fastabend, andrii, daniel, lmb, davem, kafai, dsahern,
kuba, songliubraving, yhs, kpsingh, netdev, bpf
On Sat, May 28, 2022 at 09:54 AM +08, wangyufen wrote:
> 在 2022/5/28 5:37, Cong Wang 写道:
>> On Tue, May 24, 2022 at 03:53:11PM +0800, Wang Yufen wrote:
>>> During TCP sockmap redirect pressure test, the following warning is triggered:
>>> WARNING: CPU: 3 PID: 2145 at net/core/stream.c:205 sk_stream_kill_queues+0xbc/0xd0
>>> CPU: 3 PID: 2145 Comm: iperf Kdump: loaded Tainted: G W 5.10.0+ #9
>>> Call Trace:
>>> inet_csk_destroy_sock+0x55/0x110
>>> inet_csk_listen_stop+0xbb/0x380
>>> tcp_close+0x41b/0x480
>>> inet_release+0x42/0x80
>>> __sock_release+0x3d/0xa0
>>> sock_close+0x11/0x20
>>> __fput+0x9d/0x240
>>> task_work_run+0x62/0x90
>>> exit_to_user_mode_prepare+0x110/0x120
>>> syscall_exit_to_user_mode+0x27/0x190
>>> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>>
>>> The reason we observed is that:
>>> When the listener is closing, a connection may have completed the three-way
>>> handshake but not accepted, and the client has sent some packets. The child
>>> sks in accept queue release by inet_child_forget()->inet_csk_destroy_sock(),
>>> but psocks of child sks have not released.
>>>
>> Hm, in this scenario, how does the child socket end up in the sockmap?
>> Clearly user-space does not have a chance to get an fd yet.
>>
>> And, how does your patch work? Since the child sock does not even inheirt
>> the sock proto after clone (see the comments above tcp_bpf_clone()) at
>> all?
>>
>> Thanks.
>> .
> My test cases are as follows:
>
> __section("sockops")
> int bpf_sockmap(struct bpf_sock_ops *skops)
> {
> switch (skops->op) {
> case BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB:
> case BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB:
> ...
> bpf_sock_hash_update(skops, &sock_ops_map, &key, BPF_NOEXIST);
> break;
> ...
> }
Right, when processing the final ACK in tcp_rcv_state_process(), we
invoke the BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB BPF callback.
This gives a chance to install sockmap sk_prot callbacks.
An accept() without ever calling accept() ;-)
[...]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH bpf-next] bpf,sockmap: fix sk->sk_forward_alloc warn_on in sk_stream_kill_queues
2022-05-30 16:37 ` Jakub Sitnicki
@ 2022-05-31 23:40 ` John Fastabend
0 siblings, 0 replies; 7+ messages in thread
From: John Fastabend @ 2022-05-31 23:40 UTC (permalink / raw)
To: Jakub Sitnicki, Cong Wang, wangyufen
Cc: ast, john.fastabend, andrii, daniel, lmb, davem, kafai, dsahern,
kuba, songliubraving, yhs, kpsingh, netdev, bpf
Jakub Sitnicki wrote:
> On Sat, May 28, 2022 at 09:54 AM +08, wangyufen wrote:
> > 在 2022/5/28 5:37, Cong Wang 写道:
> >> On Tue, May 24, 2022 at 03:53:11PM +0800, Wang Yufen wrote:
> >>> During TCP sockmap redirect pressure test, the following warning is triggered:
> >>> WARNING: CPU: 3 PID: 2145 at net/core/stream.c:205 sk_stream_kill_queues+0xbc/0xd0
> >>> CPU: 3 PID: 2145 Comm: iperf Kdump: loaded Tainted: G W 5.10.0+ #9
> >>> Call Trace:
> >>> inet_csk_destroy_sock+0x55/0x110
> >>> inet_csk_listen_stop+0xbb/0x380
> >>> tcp_close+0x41b/0x480
> >>> inet_release+0x42/0x80
> >>> __sock_release+0x3d/0xa0
> >>> sock_close+0x11/0x20
> >>> __fput+0x9d/0x240
> >>> task_work_run+0x62/0x90
> >>> exit_to_user_mode_prepare+0x110/0x120
> >>> syscall_exit_to_user_mode+0x27/0x190
> >>> entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >>>
> >>> The reason we observed is that:
> >>> When the listener is closing, a connection may have completed the three-way
> >>> handshake but not accepted, and the client has sent some packets. The child
> >>> sks in accept queue release by inet_child_forget()->inet_csk_destroy_sock(),
> >>> but psocks of child sks have not released.
> >>>
> >> Hm, in this scenario, how does the child socket end up in the sockmap?
> >> Clearly user-space does not have a chance to get an fd yet.
> >>
> >> And, how does your patch work? Since the child sock does not even inheirt
> >> the sock proto after clone (see the comments above tcp_bpf_clone()) at
> >> all?
> >>
> >> Thanks.
> >> .
> > My test cases are as follows:
> >
> > __section("sockops")
> > int bpf_sockmap(struct bpf_sock_ops *skops)
> > {
> > switch (skops->op) {
> > case BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB:
> > case BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB:
> > ...
> > bpf_sock_hash_update(skops, &sock_ops_map, &key, BPF_NOEXIST);
> > break;
> > ...
> > }
>
> Right, when processing the final ACK in tcp_rcv_state_process(), we
> invoke the BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB BPF callback.
>
> This gives a chance to install sockmap sk_prot callbacks.
>
> An accept() without ever calling accept() ;-)
>
> [...]
LGTM as well.
Acked-by: John Fastabend <john.fastabend@gmail.com>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH bpf-next] bpf,sockmap: fix sk->sk_forward_alloc warn_on in sk_stream_kill_queues
2022-05-24 7:53 [PATCH bpf-next] bpf,sockmap: fix sk->sk_forward_alloc warn_on in sk_stream_kill_queues Wang Yufen
2022-05-25 8:41 ` Jakub Sitnicki
2022-05-27 21:37 ` Cong Wang
@ 2022-05-31 23:50 ` patchwork-bot+netdevbpf
2 siblings, 0 replies; 7+ messages in thread
From: patchwork-bot+netdevbpf @ 2022-05-31 23:50 UTC (permalink / raw)
To: Wang Yufen
Cc: ast, john.fastabend, andrii, daniel, jakub, lmb, davem, kafai,
dsahern, kuba, songliubraving, yhs, kpsingh, netdev, bpf
Hello:
This patch was applied to bpf/bpf-next.git (master)
by Daniel Borkmann <daniel@iogearbox.net>:
On Tue, 24 May 2022 15:53:11 +0800 you wrote:
> During TCP sockmap redirect pressure test, the following warning is triggered:
> WARNING: CPU: 3 PID: 2145 at net/core/stream.c:205 sk_stream_kill_queues+0xbc/0xd0
> CPU: 3 PID: 2145 Comm: iperf Kdump: loaded Tainted: G W 5.10.0+ #9
> Call Trace:
> inet_csk_destroy_sock+0x55/0x110
> inet_csk_listen_stop+0xbb/0x380
> tcp_close+0x41b/0x480
> inet_release+0x42/0x80
> __sock_release+0x3d/0xa0
> sock_close+0x11/0x20
> __fput+0x9d/0x240
> task_work_run+0x62/0x90
> exit_to_user_mode_prepare+0x110/0x120
> syscall_exit_to_user_mode+0x27/0x190
> entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
> [...]
Here is the summary with links:
- [bpf-next] bpf,sockmap: fix sk->sk_forward_alloc warn_on in sk_stream_kill_queues
https://git.kernel.org/bpf/bpf-next/c/84dc313f7b79
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2022-05-31 23:50 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-05-24 7:53 [PATCH bpf-next] bpf,sockmap: fix sk->sk_forward_alloc warn_on in sk_stream_kill_queues Wang Yufen
2022-05-25 8:41 ` Jakub Sitnicki
2022-05-27 21:37 ` Cong Wang
2022-05-28 1:54 ` wangyufen
2022-05-30 16:37 ` Jakub Sitnicki
2022-05-31 23:40 ` John Fastabend
2022-05-31 23:50 ` patchwork-bot+netdevbpf
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.