[Buildroot] Handling delays in network interface visibility in Raspberry Pi 3

[Buildroot] Handling delays in network interface visibility in Raspberry Pi 3

[Ivo Grondman]

Hi all,

Today I?ve been trying to debug the following issue. Using Buildroot 2020.11.2 and the Vagrantfile that comes with it, I tried to build a ?vanilla? Buildroot image in the vagrant VM for my Raspberry Pi 3 Model B+ in the following way:

$ make raspberrypi3_defconfig
$ make

i.e. without any customisation. The image builds fine and the Pi boots without errors or warnings, but the network interface eth0 (configured with DHCP) will not have an IPv4 address and I manually have to perform an `ifup -a` or `ifup eth0` to get it going. So I went through all the initialisation files that have a part in bringing up the network interface and soon came to conclude that the script in /etc/network/if-pre-up.d/wait_iface has the right intentions, but still doesn?t solve the slow-to-appear interface issue.

It looks as if that particular script still needs to wait just that bit longer when it sees that /sys/class/net/eth0 exists before actually exiting. Using the file as is will not make eth0 retrieve an address from the DHCP server, but if I add a bit of extra sleep in the if-statement where it detects the presence of /sys/class/net/eth0 then it *will* work.

Now my problem with this approach is that it feels like a hack. Moreover, I?d like to know if something else could be the issue here. So:

1) Did anyone of you experience similar issues with a Raspberry Pi?
2) Are there any alternative, better solutions to this problem?
3) If the answer to 2) is a ?no?, should I submit a patch for this particular script? I?m a bit hesitant to put in a patch with an extra delay on detection of the interface, as this is obviously not something you want to do for *all* possible platforms?

Any help is greatly appreciated.

Best regards,

Ivo Grondman

[Buildroot] Handling delays in network interface visibility in Raspberry Pi 3

[Nicolas Cavallari]

On 15/02/2021 17:34, Ivo Grondman wrote:
> Hi all,
> 
> Today I?ve been trying to debug the following issue. Using Buildroot 2020.11.2 and the Vagrantfile that comes with it, I tried to build a ?vanilla? Buildroot image in the vagrant VM for my Raspberry Pi 3 Model B+ in the following way:
> 
> $ make raspberrypi3_defconfig
> $ make
> 
> i.e. without any customisation. The image builds fine and the Pi boots without errors or warnings, but the network interface eth0 (configured with DHCP) will not have an IPv4 address and I manually have to perform an `ifup -a` or `ifup eth0` to get it going. So I went through all the initialisation files that have a part in bringing up the network interface and soon came to conclude that the script in /etc/network/if-pre-up.d/wait_iface has the right intentions, but still doesn?t solve the slow-to-appear interface issue.
> 
> It looks as if that particular script still needs to wait just that bit longer when it sees that /sys/class/net/eth0 exists before actually exiting. Using the file as is will not make eth0 retrieve an address from the DHCP server, but if I add a bit of extra sleep in the if-statement where it detects the presence of /sys/class/net/eth0 then it *will* work.
> 
> Now my problem with this approach is that it feels like a hack. Moreover, I?d like to know if something else could be the issue here.

I'm curious.

Could you try to:
- Enable the iproute2 package (BR2_PACKAGE_IPROUTE2) and compile it.
- Modify wait_iface to add "ip -t monitor > /tmp/ip_monitor.log &" 
before the loop. And disable your hack.
- Let it boot, then kill the "ip" process.
- Report the content of /tmp/ip_monitor.log ? It will contain a log of 
everything that happened to network interfaces.

[Buildroot] Handling delays in network interface visibility in Raspberry Pi 3

[Peter Korsgaard]

>>>>> "Ivo" == Ivo Grondman <buildroot@grondman.net> writes:

 > Hi all,
 > Today I?ve been trying to debug the following issue. Using Buildroot
 > 2020.11.2 and the Vagrantfile that comes with it, I tried to build a
 > ?vanilla? Buildroot image in the vagrant VM for my Raspberry Pi 3
 > Model B+ in the following way:

 > $ make raspberrypi3_defconfig
 > $ make

 > i.e. without any customisation. The image builds fine and the Pi boots
 > without errors or warnings, but the network interface eth0 (configured
 > with DHCP) will not have an IPv4 address and I manually have to
 > perform an `ifup -a` or `ifup eth0` to get it going. So I went through
 > all the initialisation files that have a part in bringing up the
 > network interface and soon came to conclude that the script in
 > /etc/network/if-pre-up.d/wait_iface has the right intentions, but
 > still doesn?t solve the slow-to-appear interface issue.

 > It looks as if that particular script still needs to wait just that
 > bit longer when it sees that /sys/class/net/eth0 exists before
 > actually exiting. Using the file as is will not make eth0 retrieve an
 > address from the DHCP server, but if I add a bit of extra sleep in the
 > if-statement where it detects the presence of /sys/class/net/eth0 then
 > it *will* work.

The script simply waits for the interface to become available (E.G. for
the USB connected interface to be detected by the kernel).

I guess your problem is that the device (and/or whatever you have
connected at the other end of the cable) is too slow to negotiate a
link, so the DHCP client send the DHCP requests too soo and gives up
before the link comes up.

You should normally see output in the kernel log and the serial console
when a link is detected.


 > Now my problem with this approach is that it feels like a
 > hack. Moreover, I?d like to know if something else could be the issue
 > here. So:

 > 1) Did anyone of you experience similar issues with a Raspberry Pi?

I haven't personally.


 > 2) Are there any alternative, better solutions to this problem?

The classical way of waiting for the network link to come up is ifplugd:

http://0pointer.de/lennart/projects/ifplugd/

Notice: for a slow-to-be-detected-by-the-kernel interface like the USB
connected ethernet on the raspberrypi you will need to use the -M
option to be able to start ifplugd before the kernel has detected eth0:

http://0pointer.de/lennart/projects/ifplugd/ifplugd.8.xml

-- 
Bye, Peter Korsgaard

[Buildroot] Handling delays in network interface visibility in Raspberry Pi 3

[Ivo Grondman]

> On 15 Feb 2021, at 23:01, Peter Korsgaard <peter@korsgaard.com> wrote:
> 
> I guess your problem is that the device (and/or whatever you have
> connected at the other end of the cable) is too slow to negotiate a
> link, so the DHCP client send the DHCP requests too soo and gives up
> before the link comes up.

This comment, together with Cristopher's comment on the possibility of a switch-related issue, made me wonder what could all be causing a slow link negotiation. So I decided I?d put up the Raspberry Pi right next to the router, as before it was indeed behind an extra switch *and* needed to go over a power line adapter. Then again, I could not imagine that this was the cause as I never ever experienced this particular problem before with any other similar device, so if it would be the switch, the router, or the power line adapter then surely I would?ve seen this problem before. Therefore, I decided on a whim that I first would try swapping the Pi?s network cable with that of some other device and it worked immediately!

Turned out I was using an old cable only capable of 100BASE-TX (i.e. it only had 2 wire pairs). Replacing it with a cable with all 4 wire pairs in it did the trick, even when still behind the switch and the power line adapter. Never had problems with that cable before (didn?t care about the speed difference), but apparently in this case it was the culprit of slow link negotiation.

Thanks for the help.

Cheers,

Ivo

[Buildroot] Handling delays in network interface visibility in Raspberry Pi 3

[Peter Korsgaard]

>>>>> "Ivo" == Ivo Grondman <buildroot@grondman.net> writes:

Hi,

 > Turned out I was using an old cable only capable of 100BASE-TX
 > (i.e. it only had 2 wire pairs). Replacing it with a cable with all 4
 > wire pairs in it did the trick, even when still behind the switch and
 > the power line adapter. Never had problems with that cable before
 > (didn?t care about the speed difference), but apparently in this case
 > it was the culprit of slow link negotiation.

Ok, good to hear that you solved it!

-- 
Bye, Peter Korsgaard

[Buildroot] Handling delays in network interface visibility in Raspberry Pi 3

[Christopher McCrory]

On phone so please excuse typos and brevity.

This is probably due to spanning tree (stp) on your switch. STP will delay
forwarding traffic from 30 to 55 ish seconds. In this case the pi is just
booting too fast. PCs have enough delays that you probably wouldn't notice
this happening.  Add a sleep 56 before the ifup bit, if that solves
the problem, stp it is.  If so, solved and done. Or if possible enable
portfast on the swichport, this will speed up the stp convergence.

For buildroot, on my raspberry pi builds, I only did wifi. But IIRC, the
dhcp client should have some options to adjust the number of requests
and/or the time between requests. If so the time it waits to receive a
response should be over 55 ish seconds in total to account for stp.
Preferably, the last request should happen after stp has converged.



On Mon, Feb 15, 2021, 9:08 AM Ivo Grondman <buildroot@grondman.net> wrote:

> Hi all,
>
> Today I?ve been trying to debug the following issue. Using Buildroot
> 2020.11.2 and the Vagrantfile that comes with it, I tried to build a
> ?vanilla? Buildroot image in the vagrant VM for my Raspberry Pi 3 Model B+
> in the following way:
>
> $ make raspberrypi3_defconfig
> $ make
>
> i.e. without any customisation. The image builds fine and the Pi boots
> without errors or warnings, but the network interface eth0 (configured with
> DHCP) will not have an IPv4 address and I manually have to perform an `ifup
> -a` or `ifup eth0` to get it going. So I went through all the
> initialisation files that have a part in bringing up the network interface
> and soon came to conclude that the script in
> /etc/network/if-pre-up.d/wait_iface has the right intentions, but still
> doesn?t solve the slow-to-appear interface issue.
>
> It looks as if that particular script still needs to wait just that bit
> longer when it sees that /sys/class/net/eth0 exists before actually
> exiting. Using the file as is will not make eth0 retrieve an address from
> the DHCP server, but if I add a bit of extra sleep in the if-statement
> where it detects the presence of /sys/class/net/eth0 then it *will* work.
>
> Now my problem with this approach is that it feels like a hack. Moreover,
> I?d like to know if something else could be the issue here. So:
>
> 1) Did anyone of you experience similar issues with a Raspberry Pi?
> 2) Are there any alternative, better solutions to this problem?
> 3) If the answer to 2) is a ?no?, should I submit a patch for this
> particular script? I?m a bit hesitant to put in a patch with an extra delay
> on detection of the interface, as this is obviously not something you want
> to do for *all* possible platforms?
>
> Any help is greatly appreciated.
>
> Best regards,
>
> Ivo Grondman
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20210215/df6ca83c/attachment.html>

[Buildroot] Handling delays in network interface visibility in Raspberry Pi 3

[Peter Seiderer]

Hello *,

On Mon, 15 Feb 2021 15:33:57 -0800, Christopher McCrory <chrismcc@gmail.com> wrote:

> On phone so please excuse typos and brevity.
> 
> This is probably due to spanning tree (stp) on your switch. STP will delay
> forwarding traffic from 30 to 55 ish seconds. In this case the pi is just
> booting too fast. PCs have enough delays that you probably wouldn't notice
> this happening.  Add a sleep 56 before the ifup bit, if that solves
> the problem, stp it is.  If so, solved and done. Or if possible enable
> portfast on the swichport, this will speed up the stp convergence.
> 
> For buildroot, on my raspberry pi builds, I only did wifi. But IIRC, the
> dhcp client should have some options to adjust the number of requests
> and/or the time between requests. If so the time it waits to receive a
> response should be over 55 ish seconds in total to account for stp.
> Preferably, the last request should happen after stp has converged.

Reminds me of patch already lingering around some time in my private
repo to adjust the udhcpc start with the '-b' ('background if lease is not
obtained') option, see [1]...

Fixed dhcp lease for me in case the RPi is connected to a switch at startup
with no dhcp server available but connected later...

Regards,
Peter

[1] https://patchwork.ozlabs.org/project/buildroot/patch/20210216202022.27264-1-ps.report at gmx.net/

> 
> 
> 
> On Mon, Feb 15, 2021, 9:08 AM Ivo Grondman <buildroot@grondman.net> wrote:
> 
> > Hi all,
> >
> > Today I?ve been trying to debug the following issue. Using Buildroot
> > 2020.11.2 and the Vagrantfile that comes with it, I tried to build a
> > ?vanilla? Buildroot image in the vagrant VM for my Raspberry Pi 3 Model B+
> > in the following way:
> >
> > $ make raspberrypi3_defconfig
> > $ make
> >
> > i.e. without any customisation. The image builds fine and the Pi boots
> > without errors or warnings, but the network interface eth0 (configured with
> > DHCP) will not have an IPv4 address and I manually have to perform an `ifup
> > -a` or `ifup eth0` to get it going. So I went through all the
> > initialisation files that have a part in bringing up the network interface
> > and soon came to conclude that the script in
> > /etc/network/if-pre-up.d/wait_iface has the right intentions, but still
> > doesn?t solve the slow-to-appear interface issue.
> >
> > It looks as if that particular script still needs to wait just that bit
> > longer when it sees that /sys/class/net/eth0 exists before actually
> > exiting. Using the file as is will not make eth0 retrieve an address from
> > the DHCP server, but if I add a bit of extra sleep in the if-statement
> > where it detects the presence of /sys/class/net/eth0 then it *will* work.
> >
> > Now my problem with this approach is that it feels like a hack. Moreover,
> > I?d like to know if something else could be the issue here. So:
> >
> > 1) Did anyone of you experience similar issues with a Raspberry Pi?
> > 2) Are there any alternative, better solutions to this problem?
> > 3) If the answer to 2) is a ?no?, should I submit a patch for this
> > particular script? I?m a bit hesitant to put in a patch with an extra delay
> > on detection of the interface, as this is obviously not something you want
> > to do for *all* possible platforms?
> >
> > Any help is greatly appreciated.
> >
> > Best regards,
> >
> > Ivo Grondman
> >
> > _______________________________________________
> > buildroot mailing list
> > buildroot at busybox.net
> > http://lists.busybox.net/mailman/listinfo/buildroot
> >