We have noticed that Windows seems to try to send ARP requests over Wintun interfaces. In our configurations, these don't go anywhere and get no responses, but the ARP table fills up with addresses. For example: $ arp -a -N 10.0.0.100 |wc -l 387 Our Wintun interfaces are created with these properties: PS C:\> Get-NetIPInterface -InterfaceIndex 69 |Select-Object -Property AddressFamily,NeighborDiscoverySupported,NeighborUnreachabilityDetection |fl AddressFamily : IPv6 NeighborDiscoverySupported : Yes NeighborUnreachabilityDetection : Enabled AddressFamily : IPv4 NeighborDiscoverySupported : Yes NeighborUnreachabilityDetection : Enabled We _think_ that the NeighborDiscoverySupported property being Yes means that Windows issues ARP requests for addresses on the Wintun interface. (The second property controls another neighbour behaviour that perhaps could also be disabled for Wintun interfaces.) Microsoft's documentation for SetIpInterfaceEntry() claims that these properties are set "by the network stack", but offers no hint as to what part of the network stack does this. It does not seem possible to change these properties on an existing interface. Are these properties that either the Wintun driver or user-space API is able to control? Should they be set to No and Disabled? I don't see any use of the driver variant of SetIpInterfaceEntry() or MIB_IPINTERFACE_ROW in Wintun's driver or API code, so I'm not sure when these properties get determined. -- Brad Spencer
Hi Brad, That sure is interesting. Indeed UseNeighborUnreachabilityDetection and SupportsNeighborDiscovery can't be set with SetIpInterfaceEntry. I haven't (yet?) found any way for the driver itself to indicate that these should be false, either, via OIDs or similar. This might require some frustrating reverse engineering. I remember noticing this a long time ago, but I deemed it "annoying but harmless". However, you now mention: On Thu, Sep 9, 2021 at 4:33 PM Brad Spencer <bspencer@blackberry.com> wrote: > but the ARP table fills up with addresses. For example: > > $ arp -a -N 10.0.0.100 |wc -l > 387 If that grows indefinitely, that sounds... bad. So this might be worth looking into again. One thing about your message, though, raised a question in my mind, but I'm not sure whether its an artifact of your wording or a real thing you observed: > We have noticed that Windows seems to try to send ARP requests over > Wintun interfaces. In our configurations, these don't go anywhere and > get no responses, Indeed the ARP table fills up, as shown above, but I'm wondering how you're observing ARP requests exactly. ARP is a layer 2 protocol, and Wintun (and WireGuardNT) are layer 3 devices that should, in theory at least, not have anything to do with ARP packets. So how exactly were you "seeing" the ARP requests on the Wintun interface? Did wireshark show it? Or did you read from the Wintun ring and actually see an ARP frame? Or something else? Or was this just a manner of speaking and you didn't actually observe ARP frames themselves? Another small question: > We _think_ that the NeighborDiscoverySupported property being Yes means > that Windows issues ARP requests for addresses on the Wintun interface. That seems like a good intuition. I'm wondering whether that's something you're assuming or something you read on a Microsoft website. I ask because this might provide a good entry point for whatever reverse engineering I wind up doing to fix this. Regards, Jason
On 2021-09-09 2:42 p.m., Jason A. Donenfeld wrote: > So how exactly were > you "seeing" the ARP requests on the Wintun interface? Did wireshark > show it? Or did you read from the Wintun ring and actually see an ARP > frame? Or something else? Or was this just a manner of speaking and > you didn't actually observe ARP frames themselves? You're right to suspect that I was speaking imprecisely here. We have never seen an ARP request appear on the Wintun interface! I meant to say that we have noticed the ARP table for the Wintun interface accumulating entries. >> We _think_ that the NeighborDiscoverySupported property being Yes means >> that Windows issues ARP requests for addresses on the Wintun interface. > That seems like a good intuition. I'm wondering whether that's > something you're assuming or something you read on a Microsoft > website. I ask because this might provide a good entry point for > whatever reverse engineering I wind up doing to fix this. I pieced this together from a few scraps. On the MIB_IPINTERFACE_ROW page[1], the docs only tersely say: "A value that specifies if the IP interface support neighbor discovery." Microsoft seems to use the same terminology when documenting SetIpNetEntry2()[2]: "The SetIpNetEntry2 function sets the physical address of an existing neighbor IP address entry on the local computer." And then, most importantly, MIB_IPNET_ROW2, the structure used by that function says this in its Remarks section[3]: "For IPv4, this includes addresses determined used the Address Resolution Protocol (ARP). For IPv6, this includes addresses determined using the Neighbor Discovery (ND) protocol for IPv6 as specified in RFC 2461. " So, it seems that the "NetEntry" APIs are those that deal with ARP (and ND) entries, and the term Microsoft uses for that is "neighbor discovery". I don't know if the SupportsNeighborDiscovery field of MIB_INTERFACE_ROW is implied by other properties of the network interface (such as "all Ethernet interfaces support ARP") or whether it can be individually set at all. One other detail is that we have the gateway for the tunnel's routes set to 0.0.0.0 (or "::"). I presume that also influences how Windows decides which addresses might be on-link neighbours. 1. https://docs.microsoft.com/en-us/windows/win32/api/netioapi/ns-netioapi-mib_ipinterface_row 2. https://docs.microsoft.com/en-us/windows/win32/api/netioapi/nf-netioapi-setipnetentry2 3. https://docs.microsoft.com/en-us/windows/win32/api/netioapi/ns-netioapi-mib_ipnet_table2 -- Brad Spencer
Hi Brad, Jason,
Adding the 0.0.0.0/0 (or ":::") to the config is what is causing some
growth of the arp table, but it is not growing indefinitely. After
looking around for SupportsNeighborDiscovery and finding nothing, I
decided to check the repro. When not routing all traffic through an
interface the arp cache is basically static:
PS C:\> arp -a -N 100.0.0.2
Interface: 100.0.0.2 --- 0x38
Internet Address Physical Address Type
100.0.0.4 dynamic
224.0.0.22 static
224.0.0.251 static
224.0.0.252 static
239.255.255.250 static
When routing all traffic through it grows more or less like an arp cache should:
PS C:\> arp -a -N 100.0.0.2
Interface: 100.0.0.2 --- 0x38
Internet Address Physical Address Type
0.0.0.0 static
8.8.4.4 dynamic
8.8.8.8 dynamic
13.64.180.106 dynamic
18.211.21.156 dynamic
20.98.103.204 dynamic
23.6.164.43 dynamic
23.40.197.163 dynamic
23.204.103.147 dynamic
35.185.199.42 dynamic
40.83.247.108 dynamic
51.143.14.218 dynamic
54.214.97.217 dynamic
65.8.17.107 dynamic
67.160.11.228 dynamic
69.1.20.242 dynamic
73.118.184.4 dynamic
74.125.142.188 dynamic
74.125.197.188 dynamic
75.75.75.75 dynamic
100.0.0.2 dynamic
100.0.0.4 dynamic
142.250.69.202 dynamic
142.250.217.99 dynamic
142.251.33.74 dynamic
142.251.33.106 dynamic
173.194.202.189 dynamic
192.168.200.2 dynamic
192.228.79.201 dynamic
224.0.0.22 static
224.0.0.251 static
224.0.0.252 static
239.255.255.250 static
So I tend to agree with Jason that this is "harmless" and shouldn't
cause any serious problems. It would be nice for Microsoft to fix
Set-NetIPInterface, it looks like a bug that SupportsNeighborDiscovery
can't be set.
Best regards,
Alan Graham
On Thu, Sep 9, 2021 at 11:17 AM Brad Spencer <bspencer@blackberry.com> wrote:
>
> On 2021-09-09 2:42 p.m., Jason A. Donenfeld wrote:
> > So how exactly were
> > you "seeing" the ARP requests on the Wintun interface? Did wireshark
> > show it? Or did you read from the Wintun ring and actually see an ARP
> > frame? Or something else? Or was this just a manner of speaking and
> > you didn't actually observe ARP frames themselves?
> You're right to suspect that I was speaking imprecisely here. We have
> never seen an ARP request appear on the Wintun interface! I meant to
> say that we have noticed the ARP table for the Wintun interface
> accumulating entries.
>
> >> We _think_ that the NeighborDiscoverySupported property being Yes means
> >> that Windows issues ARP requests for addresses on the Wintun interface.
> > That seems like a good intuition. I'm wondering whether that's
> > something you're assuming or something you read on a Microsoft
> > website. I ask because this might provide a good entry point for
> > whatever reverse engineering I wind up doing to fix this.
> I pieced this together from a few scraps.
>
> On the MIB_IPINTERFACE_ROW page[1], the docs only tersely say:
>
> "A value that specifies if the IP interface support neighbor discovery."
>
> Microsoft seems to use the same terminology when documenting
> SetIpNetEntry2()[2]:
>
> "The SetIpNetEntry2 function sets the physical address of an existing
> neighbor IP address entry on the local computer."
>
> And then, most importantly, MIB_IPNET_ROW2, the structure used by that
> function says this in its Remarks section[3]:
>
> "For IPv4, this includes addresses determined used the Address
> Resolution Protocol (ARP). For IPv6, this includes addresses determined
> using the Neighbor Discovery (ND) protocol for IPv6 as specified in RFC
> 2461. "
>
> So, it seems that the "NetEntry" APIs are those that deal with ARP (and
> ND) entries, and the term Microsoft uses for that is "neighbor discovery".
>
> I don't know if the SupportsNeighborDiscovery field of MIB_INTERFACE_ROW
> is implied by other properties of the network interface (such as "all
> Ethernet interfaces support ARP") or whether it can be individually set
> at all.
>
> One other detail is that we have the gateway for the tunnel's routes set
> to 0.0.0.0 (or "::"). I presume that also influences how Windows
> decides which addresses might be on-link neighbours.
>
> 1.
> https://docs.microsoft.com/en-us/windows/win32/api/netioapi/ns-netioapi-mib_ipinterface_row
> 2.
> https://docs.microsoft.com/en-us/windows/win32/api/netioapi/nf-netioapi-setipnetentry2
> 3.
> https://docs.microsoft.com/en-us/windows/win32/api/netioapi/ns-netioapi-mib_ipnet_table2
>
> --
> Brad Spencer
>
On 2021-09-09 5:23 p.m., Alan Graham wrote: > Adding the 0.0.0.0/0 (or ":::") to the config is what is causing some > growth of the arp table, but it is not growing indefinitely. After > looking around for SupportsNeighborDiscovery and finding nothing, I > decided to check the repro. When not routing all traffic through an > interface the arp cache is basically static: Thanks for looking into this, too. I also suspect having the gateway set like this is probably necessary for Windows to start adding ARP entries. How were you able to determine that it is also sufficient? My guess is that if it is possible to indicate to Windows that the interface does not support neighbour discovery in general, doing so likely prevents ARP entries regardless of the gateway values. BTW, how did you determine that it does not grow indefinitely? > So I tend to agree with Jason that this is "harmless" and shouldn't > cause any serious problems. It would be nice for Microsoft to fix > Set-NetIPInterface, it looks like a bug that SupportsNeighborDiscovery > can't be set. One "harm" might be that the OS keeping an easy-to-query list of all (recent?) destinations in the ARP table, which could be undesirable. I'm not sure it's a bug, per se. It seems by design that you cannot change the value of SupportsNeighborDiscovery after the interface is created. The documentation for SetIpInterfaceEntry()[1] says: > The MaxReassemblySize, MinRouterAdvertisementInterval, > MaxRouterAdvertisementInterval , Connected, SupportsWakeUpPatterns, > SupportsNeighborDiscovery, SupportsRouterDiscovery, ReachableTime, > TransmitOffload, and ReceiveOffload members of the MIB_IPINTERFACE_ROW > structure pointed to by the Row are ignored when the > SetIpInterfaceEntry function is called. These members are set by the > network stack and cannot be changed using the SetIpInterfaceEntry > function. I noticed that the Cmdlet in PowerShell seems to treat the -NeighborDiscoverySupported option as an input filter vs. a value that you can set. While this surprised me, it is at least consistent with the Win32 API docs. 1. https://docs.microsoft.com/en-us/windows/win32/api/netioapi/nf-netioapi-setipinterfaceentry#remarks -- Brad Spencer
I mainly reached the conclusion that it's not growing indefinitely by
looking at the results in the arp table. There weren't duplicates or
other errors which might suggest the arp table wasn't working
properly. I believe it is, in that it should be aging out entries,
etc. I also stopped and restarted the wireguard service and it
cleared the cache as it should.
I also noticed the Powershell Cmdlet used the wrong filter parameters
when trying to set NeighborDiscoverySupported, as you did. My
expectation would be to get an error invalid parameter like with
NeighborUnreachabilityDetection, not that the filter returned zero
results because it used the wrong query.
Best regards,
Alan Graham
On Fri, Sep 10, 2021 at 10:19 AM Brad Spencer <bspencer@blackberry.com> wrote:
>
> On 2021-09-09 5:23 p.m., Alan Graham wrote:
> > Adding the 0.0.0.0/0 (or ":::") to the config is what is causing some
> > growth of the arp table, but it is not growing indefinitely. After
> > looking around for SupportsNeighborDiscovery and finding nothing, I
> > decided to check the repro. When not routing all traffic through an
> > interface the arp cache is basically static:
>
> Thanks for looking into this, too.
>
> I also suspect having the gateway set like this is probably necessary
> for Windows to start adding ARP entries. How were you able to determine
> that it is also sufficient? My guess is that if it is possible to
> indicate to Windows that the interface does not support neighbour
> discovery in general, doing so likely prevents ARP entries regardless of
> the gateway values.
>
> BTW, how did you determine that it does not grow indefinitely?
>
> > So I tend to agree with Jason that this is "harmless" and shouldn't
> > cause any serious problems. It would be nice for Microsoft to fix
> > Set-NetIPInterface, it looks like a bug that SupportsNeighborDiscovery
> > can't be set.
>
> One "harm" might be that the OS keeping an easy-to-query list of all
> (recent?) destinations in the ARP table, which could be undesirable.
>
> I'm not sure it's a bug, per se. It seems by design that you cannot
> change the value of SupportsNeighborDiscovery after the interface is
> created. The documentation for SetIpInterfaceEntry()[1] says:
>
> > The MaxReassemblySize, MinRouterAdvertisementInterval,
> > MaxRouterAdvertisementInterval , Connected, SupportsWakeUpPatterns,
> > SupportsNeighborDiscovery, SupportsRouterDiscovery, ReachableTime,
> > TransmitOffload, and ReceiveOffload members of the MIB_IPINTERFACE_ROW
> > structure pointed to by the Row are ignored when the
> > SetIpInterfaceEntry function is called. These members are set by the
> > network stack and cannot be changed using the SetIpInterfaceEntry
> > function.
>
> I noticed that the Cmdlet in PowerShell seems to treat the
> -NeighborDiscoverySupported option as an input filter vs. a value that
> you can set. While this surprised me, it is at least consistent with
> the Win32 API docs.
>
> 1.
> https://docs.microsoft.com/en-us/windows/win32/api/netioapi/nf-netioapi-setipinterfaceentry#remarks
>
>
> --
> Brad Spencer
>