A Few Minor Complications on connman + connman-vpn + wpa_supplicant + ofono Bringup

A Few Minor Complications on connman + connman-vpn + wpa_supplicant + ofono Bringup

[Grant Erickson]

It’s great to be re-engaged with the connman community after some time away, noting a community and mailing list migration in the intervening time.

I am currently working on modernizing an IP-based communications stack for a Digi i.MX6-based SBC with Ethernet, Wi-Fi, and Cellular (using a Quectel-based “Nimblelink”) connectivity. The infrastructure has migrated from ifupdown/ifplugd/wpa_supplicant/ppp/chat to connmand (1.41) + connman-vpnd + ofono (2.1) + wpa_supplicant (2.10). All in all, standing up a prototype for this more modern stack was very straightforward and simple relative to when I first engaged on this front in late 2010. Ethernet + Wi-Fi + Cellular worked, mostly, straight away with little-to-no configuration and setup on my part. Congratulations to the community on all the progress in the intervening 13 years!

I’ve noted a handful hiccups which I’ll continue to debug and triage and will post updates as I find them. In the interim, if anyone else can confirm seeing the issues below or has suggestion on them, I would welcome the input as I begin to triage each:

1. When ofono indicates to connmand that the Cellular connection has gone down, connmand aborts with signal 11 (SIGABRT) 100% of the time. The logs preceding the abort are:

    Jul 16 08:25:27 19205453 daemon.info connman-vpnd[360]: wwan0 {update} flags 4240 <DOWN>
    Jul 16 08:25:27 19205453 daemon.info connman-vpnd[360]: wwan0 {newlink} index 8 operstate 2 <DOWN>
    Jul 16 08:25:27 19205453 daemon.info connmand[377]: wwan0 {RX} 0 packets 0 bytes
    Jul 16 08:25:27 19205453 daemon.info connmand[377]: wwan0 {TX} 1 packets 48 bytes
    Jul 16 08:25:27 19205453 daemon.info connmand[377]: wwan0 {update} flags 4240 <DOWN>
    Jul 16 08:25:27 19205453 daemon.info connmand[377]: wwan0 {newlink} index 8 address 00:00:00:00:00:00 mtu 1500
    Jul 16 08:25:27 19205453 daemon.info connmand[377]: wwan0 {newlink} index 8 operstate 2 <DOWN>
    Jul 16 08:25:27 19205453 daemon.info connmand[377]: wwan0 {del} address 100.102.174.234/30 label wwan0
    Jul 16 08:25:33 19205453 daemon.err connmand[377]: Aborting (signal 11) [/usr/sbin/connmand]

Has anyone else observed this?

2. I am attempting to keep Ethernet + Wi-Fi + Cellular concurrent at all times, in that priority order. I expect, of course, that the most preferred and connected technology will keep the default route. However, I would expect that if I bind to an interface, I can still get Internet reachability with that interface. This works for Ethernet (expected, since it has the default route) and Cellular and Wi-Fi for IPv6. However, Wi-Fi over IPv4 can only reach its local subnet but not the Internet when it is not primary:

I suspect an issue with the routing setup:

	# ifconfig 
	eth0      Link encap:Ethernet  HWaddr 00:04:F3:1A:5E:E0  
			  inet addr:192.168.2.69  Bcast:192.168.2.255  Mask:255.255.255.0
			  inet6 addr: 2601:647:5a83:8c52:204:f3ff:fe1a:5ee0/64 Scope:Global
			  inet6 addr: fe80::204:f3ff:fe1a:5ee0/64 Scope:Link
			  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
			  RX packets:53 errors:0 dropped:0 overruns:0 frame:0
			  TX packets:112 errors:0 dropped:0 overruns:0 carrier:0
			  collisions:0 txqueuelen:1000 
			  RX bytes:8488 (8.2 KiB)  TX bytes:16201 (15.8 KiB)

	lo        Link encap:Local Loopback  
			  inet addr:127.0.0.1  Mask:255.0.0.0
			  inet6 addr: ::1/128 Scope:Host
			  UP LOOPBACK RUNNING  MTU:65536  Metric:1
			  RX packets:254 errors:0 dropped:0 overruns:0 frame:0
			  TX packets:254 errors:0 dropped:0 overruns:0 carrier:0
			  collisions:0 txqueuelen:1 
			  RX bytes:16495 (16.1 KiB)  TX bytes:16495 (16.1 KiB)

	wlan0     Link encap:Ethernet  HWaddr 00:04:F3:1A:5E:E1  
			  inet addr:192.168.2.46  Bcast:192.168.2.255  Mask:255.255.255.0
			  inet6 addr: 2601:647:5a83:8c52:204:f3ff:fe1a:5ee1/64 Scope:Global
			  inet6 addr: fe80::204:f3ff:fe1a:5ee1/64 Scope:Link
			  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
			  RX packets:41 errors:0 dropped:0 overruns:0 frame:0
			  TX packets:143 errors:0 dropped:0 overruns:0 carrier:0
			  collisions:0 txqueuelen:1000 
			  RX bytes:7538 (7.3 KiB)  TX bytes:15891 (15.5 KiB)

	wwan0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
			  inet addr:100.87.218.182  P-t-P:100.87.218.182  Mask:255.255.255.252
			  UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
			  RX packets:0 errors:0 dropped:0 overruns:0 frame:0
			  TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
			  collisions:0 txqueuelen:1000 
			  RX bytes:0 (0.0 B)  TX bytes:48 (48.0 B)

	# ping -c 4 -I eth0 dns.google.com 
	PING dns.google.com (8.8.8.8): 56 data bytes
	64 bytes from 8.8.8.8: seq=0 ttl=116 time=8.249 ms
	64 bytes from 8.8.8.8: seq=1 ttl=116 time=9.280 ms
	64 bytes from 8.8.8.8: seq=2 ttl=116 time=8.085 ms
	64 bytes from 8.8.8.8: seq=3 ttl=116 time=8.804 ms

	--- dns.google.com ping statistics ---
	4 packets transmitted, 4 packets received, 0% packet loss
	round-trip min/avg/max = 8.085/8.604/9.280 ms

	# ping6 -c 4 -I eth0 dns.google.com 
	PING dns.google.com (2001:4860:4860::8844): 56 data bytes
	64 bytes from 2001:4860:4860::8844: seq=0 ttl=57 time=9.434 ms
	64 bytes from 2001:4860:4860::8844: seq=1 ttl=57 time=9.235 ms
	64 bytes from 2001:4860:4860::8844: seq=2 ttl=57 time=9.297 ms
	64 bytes from 2001:4860:4860::8844: seq=3 ttl=57 time=9.166 ms

	--- dns.google.com ping statistics ---
	4 packets transmitted, 4 packets received, 0% packet loss
	round-trip min/avg/max = 9.166/9.283/9.434 ms

	# ping -c 4 -I wlan0 dns.google.com 
	PING dns.google.com (8.8.8.8): 56 data bytes

	--- dns.google.com ping statistics ---
	4 packets transmitted, 0 packets received, 100% packet loss

	# ping6 -c 4 -I wlan0 dns.google.com 
	PING dns.google.com (2001:4860:4860::8844): 56 data bytes
	64 bytes from 2001:4860:4860::8844: seq=0 ttl=57 time=100.924 ms
	64 bytes from 2001:4860:4860::8844: seq=1 ttl=57 time=124.037 ms
	64 bytes from 2001:4860:4860::8844: seq=2 ttl=57 time=45.249 ms
	64 bytes from 2001:4860:4860::8844: seq=3 ttl=57 time=69.188 ms

	--- dns.google.com ping statistics ---
	4 packets transmitted, 4 packets received, 0% packet loss
	round-trip min/avg/max = 45.249/84.849/124.037 ms

	# ping -c 4 -I wwan0 dns.google.com 
	PING dns.google.com (8.8.8.8): 56 data bytes
	64 bytes from 8.8.8.8: seq=0 ttl=113 time=605.749 ms
	64 bytes from 8.8.8.8: seq=1 ttl=113 time=287.500 ms
	64 bytes from 8.8.8.8: seq=2 ttl=113 time=98.122 ms
	64 bytes from 8.8.8.8: seq=3 ttl=113 time=293.822 ms

	--- dns.google.com ping statistics ---
	4 packets transmitted, 4 packets received, 0% packet loss
	round-trip min/avg/max = 98.122/321.298/605.749 ms

	# ip -4 route show
	default via 192.168.2.1 dev eth0 
	100.87.218.180/30 dev wwan0 proto kernel scope link src 100.87.218.182 
	100.87.218.181 dev wwan0 scope link 
	192.168.2.0/24 dev wlan0 proto kernel scope link src 192.168.2.46 
	192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.69 
	192.168.2.1 dev eth0 scope link 
	192.168.2.1 dev wlan0 scope link 
	198.224.173.135 via 100.87.218.181 dev wwan0 
	198.224.174.135 via 100.87.218.181 dev wwan0 

	# ip -6 route show
	2601:647:5a83:8c52::/64 dev wlan0 proto kernel metric 256 expires 2591856sec pref medium
	2601:647:5a83:8c52::/64 dev eth0 proto kernel metric 256 expires 2591856sec pref medium
	fe80::/64 dev wlan0 proto kernel metric 256 pref medium
	fe80::/64 dev eth0 proto kernel metric 256 pref medium
	ff00::/8 dev wlan0 metric 256 pref medium
	ff00::/8 dev eth0 metric 256 pref medium
	default via fe80::f29f:c2ff:fe10:271e dev wlan0 proto ra metric 1024 expires 1656sec hoplimit 64 pref medium
	default via fe80::f29f:c2ff:fe10:271e dev eth0 proto ra metric 1024 expires 1656sec hoplimit 64 pref medium

Has anyone else observed this?

3. In the aforementioned configuration, connmanctl (or dbus-send) reports that `State = ready` nearly indefinitely without a transition to Online, despite the ipv4.connman.net and ipv6.connman.net URLs being readily reachable.

4. Connman complains about not being able to use the configured Google DNS fallback servers:

    FallbackNameservers=8.8.8.8,8.8.4.4

with:

    Jul 12 18:12:33 19205453 daemon.err connmand[360]: Failed to connect to server 8.8.8.8
    Jul 12 18:12:33 19205453 daemon.err connmand[360]: Failed to connect to server 8.8.4.4

5. Despite having set ResolvConf=/dev/null per the man page, connman seems to be unhappy with and not following the code path for ResolvConf=/dev/null:

    Jul 12 18:02:42 19205453 daemon.warn connmand[360]: Cannot create /var/run/connman/resolv.conf falling back to /etc/resolv.conf

While I realize this is a harmless warning, it seems at odds with the man page and the code path for ‘resolvfile_export'.

I’ve manually-created a static, read-only /etc/resolv.conf which need not be dynamically updated at runtime:

    nameserver 127.0.0.1
    nameserver ::1

6. The openvpn plugin seems to be unable to find the ‘’ symbol:

    Jul 12 18:02:42 19205453 daemon.err connmand[360]: Can't load /usr/lib/connman/plugins/openvpn.so: /usr/lib/connman/plugins/openvpn.so: undefined symbol: vpn_provider_set_nameservers

The contents of /etc/connman/main.conf:

    [General]
    AllowDomainnameUpdates=false
    AllowHostnameUpdates=false
    AlwaysConnectedTechnologies=ethernet,wifi,cellular
    BackgroundScanning=true
    DefaultAutoConnectTechnologies=ethernet,wifi,cellular
    EnableOnlineCheck=true
    EnableOnlineToReadyTransition=true
    FallbackTimeservers=0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org
    FallbackNameservers=8.8.8.8,8.8.4.4
    PreferredTechnologies=ethernet,wifi,cellular
    ResolvConf=/dev/null
    SingleConnectedTechnology=false

Best,

Grant

-- 
Principal
Nuovations

gerickson@nuovations.com
http://www.nuovations.com/

Re: A Few Minor Complications on connman + connman-vpn + wpa_supplicant + ofono Bringup

[Denis Kenzior]

Hi Grant,

On 7/19/23 11:57, Grant Erickson wrote:
> It’s great to be re-engaged with the connman community after some time away, noting a community and mailing list migration in the intervening time.
> 
> I am currently working on modernizing an IP-based communications stack for a Digi i.MX6-based SBC with Ethernet, Wi-Fi, and Cellular (using a Quectel-based “Nimblelink”) connectivity. The infrastructure has migrated from ifupdown/ifplugd/wpa_supplicant/ppp/chat to connmand (1.41) + connman-vpnd + ofono (2.1) + wpa_supplicant (2.10). All in all, standing up a prototype for this more modern stack was very straightforward and simple relative to when I first engaged on this front in late 2010. Ethernet + Wi-Fi + Cellular worked, mostly, straight away with little-to-no configuration and setup on my part. Congratulations to the community on all the progress in the intervening 13 years!

You should check out iwd ;)

> 
> I suspect an issue with the routing setup:

<snip>

> 	# ip -4 route show
> 	default via 192.168.2.1 dev eth0
> 	100.87.218.180/30 dev wwan0 proto kernel scope link src 100.87.218.182
> 	100.87.218.181 dev wwan0 scope link
> 	192.168.2.0/24 dev wlan0 proto kernel scope link src 192.168.2.46
> 	192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.69
> 	192.168.2.1 dev eth0 scope link
> 	192.168.2.1 dev wlan0 scope link
> 	198.224.173.135 via 100.87.218.181 dev wwan0
> 	198.224.174.135 via 100.87.218.181 dev wwan0
>

Correct.  IPv4 routing is missing any gateway information for wlan and wwan 
links.  Hence the packets cannot be routed.  iwd uses a default route with a 
metric set with the assumption that the metric for ethernet devices would be 
higher priority.

You can also try doing something like:

ip route add defult dev wlan0 table 1234

> 	# ip -6 route show
> 	2601:647:5a83:8c52::/64 dev wlan0 proto kernel metric 256 expires 2591856sec pref medium
> 	2601:647:5a83:8c52::/64 dev eth0 proto kernel metric 256 expires 2591856sec pref medium
> 	fe80::/64 dev wlan0 proto kernel metric 256 pref medium
> 	fe80::/64 dev eth0 proto kernel metric 256 pref medium
> 	ff00::/8 dev wlan0 metric 256 pref medium
> 	ff00::/8 dev eth0 metric 256 pref medium
> 	default via fe80::f29f:c2ff:fe10:271e dev wlan0 proto ra metric 1024 expires 1656sec hoplimit 64 pref medium
> 	default via fe80::f29f:c2ff:fe10:271e dev eth0 proto ra metric 1024 expires 1656sec hoplimit 64 pref medium
>

ConnMan (or is ConnMan using the kernel to setup IPv6 routes? I don't remember) 
is setting up default routes for eth0 and wlan0, so things work in this case.

Regards,
-Denis

Re: A Few Minor Complications on connman + connman-vpn + wpa_supplicant + ofono Bringup

[Grant Erickson]

Thanks for the quick and detailed reply! Please see more inline below.

On Jul 19, 2023, at 3:58 PM, Denis Kenzior <denkenz@gmail.com> wrote:
> On 7/19/23 11:57, Grant Erickson wrote:
>> It’s great to be re-engaged with the connman community after some time away, noting a community and mailing list migration in the intervening time.
>> I am currently working on modernizing an IP-based communications stack for a Digi i.MX6-based SBC with Ethernet, Wi-Fi, and Cellular (using a Quectel-based “Nimblelink”) connectivity. The infrastructure has migrated from ifupdown/ifplugd/wpa_supplicant/ppp/chat to connmand (1.41) + connman-vpnd + ofono (2.1) + wpa_supplicant (2.10). All in all, standing up a prototype for this more modern stack was very straightforward and simple relative to when I first engaged on this front in late 2010. Ethernet + Wi-Fi + Cellular worked, mostly, straight away with little-to-no configuration and setup on my part. Congratulations to the community on all the progress in the intervening 13 years!
> 
> You should check out iwd ;)

Definitely on my to-do list as I’ve heard other sources of advocacy for iwd as well. However, for the initial bring-up and prototyping phase of this modernized stack, I wanted to hold a few things constant, wpa_supplicant among them.

>> I suspect an issue with the routing setup:
> 
> <snip>
> 
>> 	# ip -4 route show
>> 	default via 192.168.2.1 dev eth0
>> 	100.87.218.180/30 dev wwan0 proto kernel scope link src 100.87.218.182
>> 	100.87.218.181 dev wwan0 scope link
>> 	192.168.2.0/24 dev wlan0 proto kernel scope link src 192.168.2.46
>> 	192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.69
>> 	192.168.2.1 dev eth0 scope link
>> 	192.168.2.1 dev wlan0 scope link
>> 	198.224.173.135 via 100.87.218.181 dev wwan0
>> 	198.224.174.135 via 100.87.218.181 dev wwan0
>> 
> 
> Correct.  IPv4 routing is missing any gateway information for wlan and wwan links.  Hence the packets cannot be routed.  iwd uses a default route with a metric set with the assumption that the metric for ethernet devices would be higher priority.

Absent iwd doing this, shouldn’t connman be adding such a route for wlan0 as it did / seems to have for eth0 IPv4/IPv6 and wwan0 IPv4? My understand is connman is the maintainer of routing from a link redundancy perspective. Is that not accurate?

> You can also try doing something like:
> 
> ip route add default dev wlan0 table 1234

That didn’t quite work; however, this did:

ip route add default dev wlan0 via 192.168.2.1 metric 1024

where 192.168.2.1 is the next hop gateway.

> ConnMan (or is ConnMan using the kernel to setup IPv6 routes? I don't remember) is setting up default routes for eth0 and wlan0, so things work in this case.

Precisely, it seems like the IPv4 route for wlan0 is a “hole” in routing coverage where it did set IPv{4,6} routes for eth0, an IPv4 route for wwan0 (no IPv6 on that interface), and an IPv6 route for wlan0 but missed the IPv4 route.

My gut says this is a “bug” rather than “as-designed”. Thoughts?

Best,

Grant

-- 
Principal
Nuovations

gerickson@nuovations.com
http://www.nuovations.com/

Re: A Few Minor Complications on connman + connman-vpn + wpa_supplicant + ofono Bringup

[Denis Kenzior]

Hi Grant,

> Absent iwd doing this, shouldn’t connman be adding such a route for wlan0 as it did / seems to have for eth0 IPv4/IPv6 and wwan0 IPv4? My understand is connman is the maintainer of routing from a link redundancy perspective. Is that not accurate?

I'm not sure what the reasoning is?  Perhaps there was an attempt to stop any 
flows to the non-default interface?  Or maybe just an oversight.  Maybe one of 
the maintainers knows.  But I would tend to agree that the behavior you describe 
would be desirable.

> 
>> You can also try doing something like:
>>
>> ip route add default dev wlan0 table 1234
> 
> That didn’t quite work; however, this did:

Ah I think the table id is limited to a uint8, so maybe try a smaller id?

> 
> ip route add default dev wlan0 via 192.168.2.1 metric 1024
> 
> where 192.168.2.1 is the next hop gateway.
> 
>> ConnMan (or is ConnMan using the kernel to setup IPv6 routes? I don't remember) is setting up default routes for eth0 and wlan0, so things work in this case.
> 
> Precisely, it seems like the IPv4 route for wlan0 is a “hole” in routing coverage where it did set IPv{4,6} routes for eth0, an IPv4 route for wwan0 (no IPv6 on that interface), and an IPv6 route for wlan0 but missed the IPv4 route.
> 
> My gut says this is a “bug” rather than “as-designed”. Thoughts?
> 

Sort of.  From what I recall, ConnMan handles IPv4 routes completely internally 
via its own mechanisms.  For IPv6, it lets the kernel logic act on some of the 
information.  Maybe the prefix info from Router Advertisements?  This is why you 
see two default routes with the same metric in your setup.  The routes are being 
setup by the kernel and not ConnMan.  It has been a while since I looked into 
this.  Either way, as you point out, there is an asymmetry between IPv4 and IPv6 
that should be fixed.

By the way, we do have a full DHCPv4/DHCPv6/SLAAC implementation inside ell now. 
  Maybe someone feels brave enough to start using it in ConnMan?  That's one of 
the reasons we made it.

https://git.kernel.org/pub/scm/libs/ell/ell.git/tree/ell/netconfig.c

Regards,
-Denis

Re: A Few Minor Complications on connman + connman-vpn + wpa_supplicant + ofono Bringup

[Grant Erickson]

On Jul 19, 2023, at 4:29 PM, Grant Erickson <gerickson@nuovations.com> wrote:
> On Jul 19, 2023, at 3:58 PM, Denis Kenzior <denkenz@gmail.com> wrote:
>> On 7/19/23 11:57, Grant Erickson wrote:
>>> I suspect an issue with the routing setup:
>> 
>> <snip>
>> 
>>> 	# ip -4 route show
>>> 	default via 192.168.2.1 dev eth0
>>> 	100.87.218.180/30 dev wwan0 proto kernel scope link src 100.87.218.182
>>> 	100.87.218.181 dev wwan0 scope link
>>> 	192.168.2.0/24 dev wlan0 proto kernel scope link src 192.168.2.46
>>> 	192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.69
>>> 	192.168.2.1 dev eth0 scope link
>>> 	192.168.2.1 dev wlan0 scope link
>>> 	198.224.173.135 via 100.87.218.181 dev wwan0
>>> 	198.224.174.135 via 100.87.218.181 dev wwan0
>>> 
>> 
>> Correct.  IPv4 routing is missing any gateway information for wlan and wwan links.  Hence the packets cannot be routed.  iwd uses a default route with a metric set with the assumption that the metric for ethernet devices would be higher priority.
> 
> Absent iwd doing this, shouldn’t connman be adding such a route for wlan0 as it did / seems to have for eth0 IPv4/IPv6 and wwan0 IPv4? My understand is connman is the maintainer of routing from a link redundancy perspective. Is that not accurate?
> 
>> You can also try doing something like:
>> 
>> ip route add default dev wlan0 table 1234
> 
> That didn’t quite work; however, this did:
> 
> ip route add default dev wlan0 via 192.168.2.1 metric 1024
> 
> where 192.168.2.1 is the next hop gateway.

It also turns out that the 'redirect-gateway' “def1" trick employed by OpenVPN also works, albeit it requires two routing table entries to the RTA_PRIORITY / metric approach's one entry:

    ip route add 0.0.0.0/1 dev wlan0 via <gateway>
    ip route add 128.0.0.0/1 dev wlan0 via <gateway>

both routes are more specific than:

    ip route add default dev eth0 via <gateway>

and are thus chosen when doing:

    ping -c4 -Iwlan0 dns.google.com

However, since this requires two entries and may create conflicts with VPNs, I am inclined to stick with the:

    ip route add default dev wlan0 via 192.168.2.1 metric UINT32_MAX

approach and will have patches towards that end shortly.

Best,

Grant

-- 
Principal
Nuovations

gerickson@nuovations.com
http://www.nuovations.com/

There Are No Routes for Non-default Ethernet / Wi-Fi Services (was Re: A Few Minor Complications on connman + connman-vpn + wpa_supplicant + ofono Bringup)

[Grant Erickson]

On Nov 24, 2023, at 11:47 AM, Grant Erickson <gerickson@nuovations.com> wrote:
> On Jul 19, 2023, at 4:29 PM, Grant Erickson <gerickson@nuovations.com> wrote:
>> On Jul 19, 2023, at 3:58 PM, Denis Kenzior <denkenz@gmail.com> wrote:
>>> On 7/19/23 11:57, Grant Erickson wrote:
>>>> I suspect an issue with the routing setup:
>>> 
>>> <snip>
>>> 
>>>> […]
>>> 
>>> Correct.  IPv4 routing is missing any gateway information for wlan and wwan links.  Hence the packets cannot be routed.  iwd uses a default route with a metric set with the assumption that the metric for ethernet devices would be higher priority.
>> 
>> Absent iwd doing this, shouldn’t connman be adding such a route for wlan0 as it did / seems to have for eth0 IPv4/IPv6 and wwan0 IPv4? My understand is connman is the maintainer of routing from a link redundancy perspective. Is that not accurate?
>> 
>>> You can also try doing something like:
>>> 
>>> ip route add default dev wlan0 table 1234
>> 
>> That didn’t quite work; however, this did:
>> 
>> ip route add default dev wlan0 via 192.168.2.1 metric 1024
>> 
>> where 192.168.2.1 is the next hop gateway.
> 
> It also turns out that the 'redirect-gateway' “def1" trick employed by OpenVPN also works, albeit it requires two routing table entries to the RTA_PRIORITY / metric approach's one entry:
> 
>    ip route add 0.0.0.0/1 dev wlan0 via <gateway>
>    ip route add 128.0.0.0/1 dev wlan0 via <gateway>
> 
> both routes are more specific than:
> 
>    ip route add default dev eth0 via <gateway>
> 
> and are thus chosen when doing:
> 
>    ping -c4 -Iwlan0 dns.google.com
> 
> However, since this requires two entries and may create conflicts with VPNs, I am inclined to stick with the:
> 
>    ip route add default dev wlan0 via 192.168.2.1 metric UINT32_MAX
> 
> approach and will have patches towards that end shortly.

ConnMan Community:

At this point, I have low-priority default routes working and mostly production-ready, at least for Ethernet and Wi-Fi. I need to bring Cellular into the mix today for testing to ensure that follows the algorithm (which, aside from VPN as before, is service-neutral).

Hopefully, someone from the list can assist with VPN testing. I’ll post an evaluation GitHub branch along with RFC patches for those interested / capable.

I’ve made every effort to not break VPN support intentionally; however, there was some overloading of the “active” (which I’ve now formalized as one of four states in a gateway lifetime finite state machine) gateway flag and I’ve “un-overloaded” it such that the “active” state behaves as it seems it was originally designed to, without the overloading. The overloading was such that a gateway configuration would be promoted to “active” and demoted out of it in the {unset, set}_default_gateway for non-default cases such that the connection_newgateway/connection_delgateway RTNL notification round-trip was short-circuited. There will now be a “flags” member that will include a VPN flag (formerly a Boolean) to which additional flags can be added for this “overloading” if so warranted / necessary.

The other use case, beyond VPN, I am unsure about and do not have the test DUT for is multiple Ethernet or Wi-Fi interfaces. I suspect my approach of one metric 0 “high-priority” gateway / default route and one metric UINT32_MAX “low-priorty” gateway / default route will fall apart there. A partitioning of the 1 - UINT32_MAX space may need to be partitioned up by “service order” for non-default services for that case. Again, if you have a use and test case for that, testing and feedback would be appreciated. There may also need to be some per-technology plugin introspection interaction, per Denis’ comments above, that say “Don’t set a low-priority default route; we’ve got that on our end”.

My test case thus far has been:

    1. Start with Ethernet and Wi-Fi connected.
	a. Verify "connmanctl services; ifconfig; ip -4 route show; ping -c4 -Iwlan0 dns.google.com; ping6 -c4 -Iwlan0 dns.google.com" against the Ethernet IP address via ssh.
    2. Unplug Ethernet at the DUT.
        b. Verify "connmanctl services; ifconfig; ip -4 route show; ping -c4 -Iwlan0 dns.google.com; ping6 -c4 -Iwlan0 dns.google.com" against the Wi-Fi IP address via ssh.
    3. Replug Ethernet at the DUT.
	c. Verify (1a) above again.

Prior to my pending patches, (1a) and (1c) failed every time for the "ping -c4 -Iwlan0 dns.google.com” case.

In the (1) and (3) cases, the IPv4 routing table ends up looking like:

    default via 192.168.2.1 dev eth0 
    default via 192.168.2.1 dev wlan0 metric 4294967295 
    192.168.2.0/24 dev wlan0 proto kernel scope link src 192.168.2.46 
    192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.69 
    192.168.2.1 dev eth0 scope link 
    192.168.2.1 dev wlan0 scope link 

Best,

Grant

-- 
Principal
Nuovations

gerickson@nuovations.com
http://www.nuovations.com/

Re: A Few Minor Complications on connman + connman-vpn + wpa_supplicant + ofono Bringup

[Grant Erickson]

On Nov 24, 2023, at 11:47 AM, Grant Erickson <gerickson@nuovations.com> wrote:
> On Jul 19, 2023, at 4:29 PM, Grant Erickson <gerickson@nuovations.com> wrote:
>> On Jul 19, 2023, at 3:58 PM, Denis Kenzior <denkenz@gmail.com> wrote:
>>> On 7/19/23 11:57, Grant Erickson wrote:
>>>> I suspect an issue with the routing setup:
>>> 
>>> <snip>
>>> 
>>>> <snip>
>>> 
>>> Correct.  IPv4 routing is missing any gateway information for wlan and wwan links.  Hence the packets cannot be routed.  iwd uses a default route with a metric set with the assumption that the metric for ethernet devices would be higher priority.
>> 
>> Absent iwd doing this, shouldn’t connman be adding such a route for wlan0 as it did / seems to have for eth0 IPv4/IPv6 and wwan0 IPv4? My understand is connman is the maintainer of routing from a link redundancy perspective. Is that not accurate?
>> 
>>> You can also try doing something like:
>>> 
>>> ip route add default dev wlan0 table 1234
>> 
>> That didn’t quite work; however, this did:
>> 
>> ip route add default dev wlan0 via 192.168.2.1 metric 1024
>> 
>> where 192.168.2.1 is the next hop gateway.

ConnMan Community:

After the substantial number of patches submitted between 2023-11-07 and 2023-12-20, I am pleased to convey that not only does the above use case (successfully binding to and using a non-default service network interface) work:

# ifconfig 
eth0      Link encap:Ethernet  HWaddr 00:04:F3:1A:5E:E0  
          inet addr:192.168.2.69  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: 2601:647:5a00:15c2:204:f3ff:fe1a:5ee0/64 Scope:Global
          inet6 addr: fe80::204:f3ff:fe1a:5ee0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:54310 errors:0 dropped:0 overruns:0 frame:0
          TX packets:37996 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:8580232 (8.1 MiB)  TX bytes:3603832 (3.4 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:34349 errors:0 dropped:0 overruns:0 frame:0
          TX packets:34349 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:1871507 (1.7 MiB)  TX bytes:1871507 (1.7 MiB)

wlan0     Link encap:Ethernet  HWaddr 00:04:F3:1A:5E:E1  
          inet addr:192.168.2.46  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::204:f3ff:fe1a:5ee1/64 Scope:Link
          inet6 addr: 2601:647:5a00:15c2:204:f3ff:fe1a:5ee1/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:20500 errors:0 dropped:0 overruns:0 frame:0
          TX packets:450 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:4118209 (3.9 MiB)  TX bytes:52914 (51.6 KiB)

wwan0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:100.69.213.13  P-t-P:100.69.213.13  Mask:255.255.255.252
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:48 (48.0 B)

# ip -4 route show
default via 192.168.2.1 dev eth0 
default via 100.69.213.14 dev wwan0 metric 4294959103 
default via 192.168.2.1 dev wlan0 metric 4294960127 
100.69.213.12/30 dev wwan0 proto kernel scope link src 100.69.213.13 
100.69.213.14 dev wwan0 scope link 
192.168.2.0/24 dev wlan0 proto kernel scope link src 192.168.2.46 
192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.69 
192.168.2.1 dev eth0 scope link 
192.168.2.1 dev wlan0 scope link 
198.224.173.135 via 100.69.213.14 dev wwan0 
198.224.174.135 via 100.69.213.14 dev wwan0 

# ping -c4 -Iwlan0 dns.google.com
PING dns.google.com (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=116 time=12.208 ms
64 bytes from 8.8.8.8: seq=1 ttl=116 time=40.421 ms
64 bytes from 8.8.8.8: seq=2 ttl=116 time=64.158 ms
64 bytes from 8.8.8.8: seq=3 ttl=116 time=87.914 ms

--- dns.google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 12.208/51.175/87.914 ms

# ping -c4 -Ieth0 dns.google.com
PING dns.google.com (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=116 time=8.409 ms
64 bytes from 8.8.8.8: seq=1 ttl=116 time=9.209 ms
64 bytes from 8.8.8.8: seq=2 ttl=116 time=8.847 ms
64 bytes from 8.8.8.8: seq=3 ttl=116 time=8.414 ms

--- dns.google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 8.409/8.719/9.209 ms

# ping -c4 -Iwwan0 dns.google.com
PING dns.google.com (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=114 time=665.069 ms
64 bytes from 8.8.8.8: seq=1 ttl=114 time=42.188 ms
64 bytes from 8.8.8.8: seq=2 ttl=114 time=82.943 ms
64 bytes from 8.8.8.8: seq=3 ttl=114 time=121.834 ms

--- dns.google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 42.188/228.008/665.069 ms

as desired, but so does the ‘OnlineCheckMode=“continuous”’ (formerly EnableOnlineCheck=true, EnableOnlineToReadyTransition=true) use case in which end-to-end reachability failures that are not next-hop link related are recognized and cause failover to the next “online” service with end-to-end reachability.

Next, onto investigating VPN failures with the most recent patches for Christian Hewitt and looking into why cellular fails after some period of time with oFono 2.1 and 2.2.

Best,

Grant

-- 
Principal
Nuovations

gerickson@nuovations.com
http://www.nuovations.com/