All of lore.kernel.org
 help / color / mirror / Atom feed
* openwrt route_allowed_ips is inprecise
@ 2016-12-18 20:14 Jason A. Donenfeld
  2016-12-19  8:00 ` Jörg Thalheim
  2016-12-20  1:13 ` Baptiste Jonglez
  0 siblings, 2 replies; 16+ messages in thread
From: Jason A. Donenfeld @ 2016-12-18 20:14 UTC (permalink / raw)
  To: Dan Lüdtke; +Cc: WireGuard mailing list

Hey Dan,

The route_allowed_ips directive is not precise enough. I'm CCing Jorg,
the NixOS maintainer, because this same concern probably applies to
the Nix logic.

Your code is:

  if [ ${route_allowed_ips} -ne 0 ]; then
   for allowed_ip in ${allowed_ips}; do
     case "${allowed_ip}" in
       *:*/*)
         proto_add_ipv6_route "${allowed_ip%%/*}" "${allowed_ip##*/}"
       ;;
       */*)
         proto_add_ipv4_route "${allowed_ip%%/*}" "${allowed_ip##*/}"
       ;;
     esac
   done
 fi

The way it should be done is described in wg-config:

https://git.zx2c4.com/WireGuard/tree/contrib/examples/wg-config/wg-config#n130

    if [[ $AUTO_ROUTE -eq 1 ]]; then
        for i in $(wg show "$INTERFACE" allowed-ips | cut -f 2 | tr -d ,); do
            if ! add_default "$i" && [[ $(ip route get "$i") != *dev\
$INTERFACE\ * ]]; then
                add_route "$i"
            fi
        done
    fi

The add_default thing just accounts for dealing with 0/1 128/1, which
you can ignore, since openwrt has the dependency mechanism. But the
important thing is that I run `ip route get` for each one, and only
add a route if necessary.

FYI.

Jason

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-18 20:14 openwrt route_allowed_ips is inprecise Jason A. Donenfeld
@ 2016-12-19  8:00 ` Jörg Thalheim
  2016-12-19 12:32   ` Jason A. Donenfeld
  2016-12-20  1:13 ` Baptiste Jonglez
  1 sibling, 1 reply; 16+ messages in thread
From: Jörg Thalheim @ 2016-12-19  8:00 UTC (permalink / raw)
  To: Jason A. Donenfeld, Dan Lüdtke; +Cc: WireGuard mailing list

On 2016-12-18 21:14, Jason A. Donenfeld wrote:
> Hey Dan,
>
> The route_allowed_ips directive is not precise enough. I'm CCing Jorg,
> the NixOS maintainer, because this same concern probably applies to
> the Nix logic.
>
> Your code is:
>
>   if [ ${route_allowed_ips} -ne 0 ]; then
>    for allowed_ip in ${allowed_ips}; do
>      case "${allowed_ip}" in
>        *:*/*)
>          proto_add_ipv6_route "${allowed_ip%%/*}" "${allowed_ip##*/}"
>        ;;
>        */*)
>          proto_add_ipv4_route "${allowed_ip%%/*}" "${allowed_ip##*/}"
>        ;;
>      esac
>    done
>  fi
>
> The way it should be done is described in wg-config:
>
> https://git.zx2c4.com/WireGuard/tree/contrib/examples/wg-config/wg-conf=
ig#n130
>
>     if [[ $AUTO_ROUTE -eq 1 ]]; then
>         for i in $(wg show "$INTERFACE" allowed-ips | cut -f 2 | tr -d =
,); do
>             if ! add_default "$i" && [[ $(ip route get "$i") !=3D *dev\=

> $INTERFACE\ * ]]; then
>                 add_route "$i"
>             fi
>         done
>     fi
>
> The add_default thing just accounts for dealing with 0/1 128/1, which
> you can ignore, since openwrt has the dependency mechanism. But the
> important thing is that I run `ip route get` for each one, and only
> add a route if necessary.

Should no the output first sorted from the shortest subnet prefix to the =
longest?
What do you mean be precise? Is there a bug in the other logic apart from=

probably unnecessary routes?

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-19  8:00 ` Jörg Thalheim
@ 2016-12-19 12:32   ` Jason A. Donenfeld
  2016-12-19 13:06     ` Baptiste Jonglez
  0 siblings, 1 reply; 16+ messages in thread
From: Jason A. Donenfeld @ 2016-12-19 12:32 UTC (permalink / raw)
  To: Jörg Thalheim; +Cc: WireGuard mailing list

On Mon, Dec 19, 2016 at 9:00 AM, J=C3=B6rg Thalheim <joerg@higgsboson.tk> w=
rote:
> Should no the output first sorted from the shortest subnet prefix to the =
longest?

No; that doesn't matter.

> What do you mean be precise? Is there a bug in the other logic apart from
> probably unnecessary routes?

Performance will suffer unnecessarily.

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-19 12:32   ` Jason A. Donenfeld
@ 2016-12-19 13:06     ` Baptiste Jonglez
  2016-12-19 13:09       ` Jason A. Donenfeld
  0 siblings, 1 reply; 16+ messages in thread
From: Baptiste Jonglez @ 2016-12-19 13:06 UTC (permalink / raw)
  To: Jason A. Donenfeld; +Cc: WireGuard mailing list

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

On Mon, Dec 19, 2016 at 01:32:31PM +0100, Jason A. Donenfeld wrote:
> On Mon, Dec 19, 2016 at 9:00 AM, Jörg Thalheim <joerg@higgsboson.tk> wrote:
> > What do you mean be precise? Is there a bug in the other logic apart from
> > probably unnecessary routes?
> 
> Performance will suffer unnecessarily.

Please provide numbers.  I would be very surprised if a few redundant
routes have any performance impact, given that the kernel can handle 600k
routes without major issues.

As far as I can see, your argument is more about "(pure) technical
correctness" than about performance.  For the OpenWRT proto, I believe
that simplicity of the code is a more important factor, even if it means a
few redundant routes in some corner cases.  These routes do not hurt
functionality.

Baptiste

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-19 13:06     ` Baptiste Jonglez
@ 2016-12-19 13:09       ` Jason A. Donenfeld
  2016-12-19 13:19         ` Baptiste Jonglez
  0 siblings, 1 reply; 16+ messages in thread
From: Jason A. Donenfeld @ 2016-12-19 13:09 UTC (permalink / raw)
  To: Baptiste Jonglez; +Cc: WireGuard mailing list

On Mon, Dec 19, 2016 at 2:06 PM, Baptiste Jonglez
<baptiste@bitsofnetworks.org> wrote:
> Please provide numbers.  I would be very surprised if a few redundant
> routes have any performance impact, given that the kernel can handle 600k
> routes without major issues.

I'm thinking about the case in which a server has a 10/8 of clients,
each of which gets a /32. In this case quite a few routes wind up in
the table...

Fortunately the change is pretty easy. Instead of running
`proto_add_ipv4_route ...` you run:

[[ $(ip route get "$i") != *dev\ $INTERFACE\ * ]] && proto_add_ipv4_route ...

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-19 13:09       ` Jason A. Donenfeld
@ 2016-12-19 13:19         ` Baptiste Jonglez
  2016-12-19 13:21           ` Jason A. Donenfeld
  0 siblings, 1 reply; 16+ messages in thread
From: Baptiste Jonglez @ 2016-12-19 13:19 UTC (permalink / raw)
  To: Jason A. Donenfeld; +Cc: WireGuard mailing list

[-- Attachment #1: Type: text/plain, Size: 1286 bytes --]

On Mon, Dec 19, 2016 at 02:09:33PM +0100, Jason A. Donenfeld wrote:
> On Mon, Dec 19, 2016 at 2:06 PM, Baptiste Jonglez
> <baptiste@bitsofnetworks.org> wrote:
> > Please provide numbers.  I would be very surprised if a few redundant
> > routes have any performance impact, given that the kernel can handle 600k
> > routes without major issues.
> 
> I'm thinking about the case in which a server has a 10/8 of clients,
> each of which gets a /32. In this case quite a few routes wind up in
> the table...

How many?  What is the performance impact?

> Fortunately the change is pretty easy. Instead of running
> `proto_add_ipv4_route ...` you run:
> 
> [[ $(ip route get "$i") != *dev\ $INTERFACE\ * ]] && proto_add_ipv4_route ...

I really don't like this kind of magic: if there are 42 allowed_ips
entries in the config, then I would expect 42 routes to be created.  If
you don't want them, then just disable route_allowed_ips and add static or
interface routes yourself.

Also, are you sure that this works with busybox's version of "ip"?  What
if "ip" is not enabled in the image?

All in all, since this change is not functionally needed, I don't see the
point of adding the extra complexity and spending the time to test and
maintain this.

Baptiste

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-19 13:19         ` Baptiste Jonglez
@ 2016-12-19 13:21           ` Jason A. Donenfeld
  0 siblings, 0 replies; 16+ messages in thread
From: Jason A. Donenfeld @ 2016-12-19 13:21 UTC (permalink / raw)
  To: Baptiste Jonglez; +Cc: WireGuard mailing list

On Mon, Dec 19, 2016 at 2:19 PM, Baptiste Jonglez
<baptiste@bitsofnetworks.org> wrote:
> spending the time to test and
> maintain this.

Can't argue with that, I guess.

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-18 20:14 openwrt route_allowed_ips is inprecise Jason A. Donenfeld
  2016-12-19  8:00 ` Jörg Thalheim
@ 2016-12-20  1:13 ` Baptiste Jonglez
  2016-12-20  3:14   ` Jason A. Donenfeld
  1 sibling, 1 reply; 16+ messages in thread
From: Baptiste Jonglez @ 2016-12-20  1:13 UTC (permalink / raw)
  To: Jason A. Donenfeld; +Cc: WireGuard mailing list

[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]

On Sun, Dec 18, 2016 at 09:14:18PM +0100, Jason A. Donenfeld wrote:
> The way it should be done is described in wg-config:
> 
> https://git.zx2c4.com/WireGuard/tree/contrib/examples/wg-config/wg-config#n130
> 
>     if [[ $AUTO_ROUTE -eq 1 ]]; then
>         for i in $(wg show "$INTERFACE" allowed-ips | cut -f 2 | tr -d ,); do
>             if ! add_default "$i" && [[ $(ip route get "$i") != *dev\
> $INTERFACE\ * ]]; then
>                 add_route "$i"
>             fi
>         done
>     fi

> the important thing is that I run `ip route get` for each one, and only
> add a route if necessary.

By the way, besides the issue of magic, this approach seems incorrect
depending on the order of the routes.  Consider the case where cmd_add()
handles the following sequence of allowed-ips:

    10.0.0.0/8   dev wg0
    10.4.7.0/24  dev wg0
    10.4.0.0/16  dev wg1

Your method would incorrectly drop the second route, and then the third
route would take over traffic for this /24 through the wrong interface.

I'm sure this approach can be made to actually work in all cases (with
great complexity), but really, who cares about a few redundant routes.

Baptiste

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-20  1:13 ` Baptiste Jonglez
@ 2016-12-20  3:14   ` Jason A. Donenfeld
  2016-12-20  3:38     ` Dan Luedtke
  0 siblings, 1 reply; 16+ messages in thread
From: Jason A. Donenfeld @ 2016-12-20  3:14 UTC (permalink / raw)
  To: Baptiste Jonglez; +Cc: WireGuard mailing list

Hey Baptiste,

On Tue, Dec 20, 2016 at 2:13 AM, Baptiste Jonglez
<baptiste@bitsofnetworks.org> wrote:

> By the way, besides the issue of magic, this approach seems incorrect
> depending on the order of the routes.  Consider the case where cmd_add()
> handles the following sequence of allowed-ips:
>
>     10.0.0.0/8   dev wg0
>     10.4.7.0/24  dev wg0
>     10.4.0.0/16  dev wg1
>
> Your method would incorrectly drop the second route, and then the third
> route would take over traffic for this /24 through the wrong interface.

Very smart point and astute observation! This should be fixed here:

https://git.zx2c4.com/WireGuard/commit/?id=5838c950859f1b55ad344e81b77a0b71917ffd61

Unless there are objections, that will ship with the next snapshot.

Jason

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-20  3:14   ` Jason A. Donenfeld
@ 2016-12-20  3:38     ` Dan Luedtke
  2016-12-20  4:33       ` Jason A. Donenfeld
  0 siblings, 1 reply; 16+ messages in thread
From: Dan Luedtke @ 2016-12-20  3:38 UTC (permalink / raw)
  To: Jason A. Donenfeld; +Cc: WireGuard mailing list



> On 20 Dec 2016, at 04:14, Jason A. Donenfeld <Jason@zx2c4.com>=20
> https://git.zx2c4.com/WireGuard/commit/?id=3D5838c950859f1b55ad344e81b77a0=
b71917ffd61
>=20
> Unless there are objections, that will ship with the next snapshot

No objection but a remark: It is now even more complex and I think shell scr=
ipt is not the right solution. This is pretty straightforward with netlink b=
ut a pain to solve in shell.

Regarding LEDE, netifd should track the routes being added and the extra rou=
tes do not really do harm.=

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-20  3:38     ` Dan Luedtke
@ 2016-12-20  4:33       ` Jason A. Donenfeld
  2016-12-20  8:52         ` Dan Lüdtke
  0 siblings, 1 reply; 16+ messages in thread
From: Jason A. Donenfeld @ 2016-12-20  4:33 UTC (permalink / raw)
  To: Dan Luedtke; +Cc: WireGuard mailing list

On Tue, Dec 20, 2016 at 4:38 AM, Dan Luedtke <mail@danrl.com> wrote:
> No objection but a remark: It is now even more complex and I think shell script is not the right solution.

This is *not* a complex algorithm and can be implemented correctly in
shell. It's on my list for tomorrow to actually clean that up.

> This is pretty straightforward with netlink

No, it's much less straight-forward with raw netlink. Raw netlink
involves hundreds of lines of code to do anything at all. A real mess.
Fortunately there are wrapper libraries you can use from various
languages to make it easier.

> Regarding LEDE, netifd should track the routes being added and the extra routes do not really do harm.

Alright then...

Speaking of netifd, did you ever fix that netifd issue with the IP dependency?

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-20  4:33       ` Jason A. Donenfeld
@ 2016-12-20  8:52         ` Dan Lüdtke
  2016-12-20 10:15           ` Dan Lüdtke
  0 siblings, 1 reply; 16+ messages in thread
From: Dan Lüdtke @ 2016-12-20  8:52 UTC (permalink / raw)
  To: Jason A. Donenfeld; +Cc: WireGuard mailing list

Regarding the initial preciseness issue, have you tested that on LEDE? I =
can't manage to get duplicate routes. However, outdated testing =
environment. Will rebuild and test again. I can't quite understand what =
the initial issue was. Wouldn't you get a "rtnetlink: file exists" when =
you try to add an route that already exists?

This also can only occur if someone uses static routes AND decides to =
use route_allowed_ips, right?

>> This is pretty straightforward with netlink
>=20
> No, it's much less straight-forward with raw netlink. Raw netlink
> involves hundreds of lines of code to do anything at all. A real mess.
> Fortunately there are wrapper libraries you can use from various
> languages to make it easier.

True. I was referring to the auto-route option I read on LKML a while =
ago. Of course, when done from userspace, netlink is not the ideal way.=20=


>=20
>> Regarding LEDE, netifd should track the routes being added and the =
extra routes do not really do harm.
>=20
> Alright then...
>=20
> Speaking of netifd, did you ever fix that netifd issue with the IP =
dependency?

I am on it. First version did add dependency for both protocols if the =
endpoint name had A and AAAA records. However, I find it cleaner to =
check which endpoint wg chose to use and only add that IP address as an =
depedency. Patch/PR comes when I am satisfied with stability.=

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-20  8:52         ` Dan Lüdtke
@ 2016-12-20 10:15           ` Dan Lüdtke
  2016-12-20 13:33             ` Jason A. Donenfeld
  0 siblings, 1 reply; 16+ messages in thread
From: Dan Lüdtke @ 2016-12-20 10:15 UTC (permalink / raw)
  To: Jason A. Donenfeld; +Cc: WireGuard mailing list


> On 20 Dec 2016, at 09:52, Dan L=C3=BCdtke <mail@danrl.com> wrote:
>=20
> Regarding the initial preciseness issue, have you tested that on LEDE? =
I can't manage to get duplicate routes. However, outdated testing =
environment. Will rebuild and test again. I can't quite understand what =
the initial issue was. Wouldn't you get a "rtnetlink: file exists" when =
you try to add an route that already exists?

New environment, build from latest sources this morning. Can't =
reproduce. I can't see duplicate routes. Static routes were added via =
LuCI to represent a typical user's approach.

Can we drop this discussion until we can reproduce the problem?

>>=20
>>> Regarding LEDE, netifd should track the routes being added and the =
extra routes do not really do harm.
>>=20
>> Alright then...
>>=20
>> Speaking of netifd, did you ever fix that netifd issue with the IP =
dependency?
>=20
> I am on it. First version did add dependency for both protocols if the =
endpoint name had A and AAAA records. However, I find it cleaner to =
check which endpoint wg chose to use and only add that IP address as an =
depedency. Patch/PR comes when I am satisfied with stability.

https://github.com/openwrt/packages/pull/3680

This is the leanest approach I came up with and my top candidate for =
merging. Review very welcome.

Thanks!

Dan=

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-20 10:15           ` Dan Lüdtke
@ 2016-12-20 13:33             ` Jason A. Donenfeld
  2016-12-20 14:51               ` Dan Lüdtke
  0 siblings, 1 reply; 16+ messages in thread
From: Jason A. Donenfeld @ 2016-12-20 13:33 UTC (permalink / raw)
  To: Dan Lüdtke; +Cc: WireGuard mailing list

Hi Dan,

On Tue, Dec 20, 2016 at 11:15 AM, Dan L=C3=BCdtke <mail@danrl.com> wrote:
> New environment, build from latest sources this morning. Can't reproduce.=
 I can't see duplicate routes. Static routes were added via LuCI to represe=
nt a typical user's approach.
>
> Can we drop this discussion until we can reproduce the problem?

That's a weird way of putting it, because you !=3D we.

config wireguard_wg
       option public_key '0Nz4n2wdhDJtFbkdIFqlJ4vkKIGFgBVQPxyJ0XWE31o=3D'
       list allowed_ips '10.200.100.1/32'
       option route_allowed_ips '1'
config interface 'wgnet'
       option proto 'static'
       option ifname 'wg'
       option ipaddr '10.200.100.4/24'
-->
root@mipsnet:~# ip r
10.200.100.0/24 dev wg  src 10.200.100.4
10.200.100.1 dev wg


We can drop the discussion, though, because Baptiste, the downstream
package maintainer, doesn't want to maintain that kind of logic and
evidently finds Linux's v4 LC-trie and v6 radix trie to be fast enough
for his purposes. I don't find the duplicated entries very desirable,
but I'm totally fine deferring to his judgement.

>> I am on it. First version did add dependency for both protocols if the e=
ndpoint name had A and AAAA records. However, I find it cleaner to check wh=
ich endpoint wg chose to use and only add that IP address as an depedency. =
Patch/PR comes when I am satisfied with stability.

I think calling `wg show $if endpoints` like you do is the correct
solution. Good thinking. I'll put some comments on the PR.

Jason

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-20 13:33             ` Jason A. Donenfeld
@ 2016-12-20 14:51               ` Dan Lüdtke
  2016-12-20 18:27                 ` Jason A. Donenfeld
  0 siblings, 1 reply; 16+ messages in thread
From: Dan Lüdtke @ 2016-12-20 14:51 UTC (permalink / raw)
  To: Jason A. Donenfeld; +Cc: WireGuard mailing list


> On 20 Dec 2016, at 14:33, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> On Tue, Dec 20, 2016 at 11:15 AM, Dan L=C3=BCdtke <mail@danrl.com> =
wrote:
>> New environment, build from latest sources this morning. Can't =
reproduce. I can't see duplicate routes. Static routes were added via =
LuCI to represent a typical user's approach.
>>=20
>> Can we drop this discussion until we can reproduce the problem?
>=20
> That's a weird way of putting it, because you !=3D we.

Sorry, not a native speaker.

> config wireguard_wg
>       option public_key '0Nz4n2wdhDJtFbkdIFqlJ4vkKIGFgBVQPxyJ0XWE31o=3D'=

>       list allowed_ips '10.200.100.1/32'
>       option route_allowed_ips '1'
> config interface 'wgnet'
>       option proto 'static'
>       option ifname 'wg'
>       option ipaddr '10.200.100.4/24'
> -->
> root@mipsnet:~# ip r
> 10.200.100.0/24 dev wg  src 10.200.100.4
> 10.200.100.1 dev wg

I see what you mean now. Don't see a real problem there despite =
ugliness, but let me think of a solution. May not be my highest priority =
right now, though.


Cheers,

Dan=

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: openwrt route_allowed_ips is inprecise
  2016-12-20 14:51               ` Dan Lüdtke
@ 2016-12-20 18:27                 ` Jason A. Donenfeld
  0 siblings, 0 replies; 16+ messages in thread
From: Jason A. Donenfeld @ 2016-12-20 18:27 UTC (permalink / raw)
  To: Dan Lüdtke; +Cc: WireGuard mailing list

Hi Dan,

On Tue, Dec 20, 2016 at 3:51 PM, Dan L=C3=BCdtke <mail@danrl.com> wrote:
> I see what you mean now. Don't see a real problem there despite ugliness,=
 but let me think of a solution. May not be my highest priority right now, =
though.

Solutions:

0) Do nothing.
1) Make netifd deal with it.
2) Sort the endpoints allowed IPs by cidr, and call `ip route get`
before each `ip route add`.
3) Compute the set difference in bash between the addresses of the
interface and the allowed-ips of the interface, and only `ip route
add` the difference.
4) Compute the set difference using sipcalc or ipcalc between the
addresses of the interface and the allowed-ips of the interface, and
only `ip route add` the difference.
5) Bug me to add this functionality to wg(8), and receive frustrating
responses like "wg(8) isn't supposed to touch any of the info that
ip(8) has due to future tool merging plans."

Jason

^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2016-12-20 18:20 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-12-18 20:14 openwrt route_allowed_ips is inprecise Jason A. Donenfeld
2016-12-19  8:00 ` Jörg Thalheim
2016-12-19 12:32   ` Jason A. Donenfeld
2016-12-19 13:06     ` Baptiste Jonglez
2016-12-19 13:09       ` Jason A. Donenfeld
2016-12-19 13:19         ` Baptiste Jonglez
2016-12-19 13:21           ` Jason A. Donenfeld
2016-12-20  1:13 ` Baptiste Jonglez
2016-12-20  3:14   ` Jason A. Donenfeld
2016-12-20  3:38     ` Dan Luedtke
2016-12-20  4:33       ` Jason A. Donenfeld
2016-12-20  8:52         ` Dan Lüdtke
2016-12-20 10:15           ` Dan Lüdtke
2016-12-20 13:33             ` Jason A. Donenfeld
2016-12-20 14:51               ` Dan Lüdtke
2016-12-20 18:27                 ` Jason A. Donenfeld

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.