[Buildroot] SSH server starts too late

[Buildroot] SSH server starts too late

[Hammami Omar]

Hello,

I am using the buildroot version "2019.11.1" and I have noticed that my SSH
server starts too late.
I was using an old version "2017.08" and the SSH server was starting
normally.
Are you aware of this issue please ?
And do you propose a workaround for this ?
And thank you very much.

Best regards,
omar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200221/778063a9/attachment.html>

[Buildroot] SSH server starts too late

[Peter Seiderer]

Hello Hammami,

On Fri, 21 Feb 2020 10:13:17 +0100, Hammami Omar <omar18hammami@gmail.com> wrote:

> Hello,
>
> I am using the buildroot version "2019.11.1" and I have noticed that my SSH
> server starts too late.
> I was using an old version "2017.08" and the SSH server was starting
> normally.
> Are you aware of this issue please ?
> And do you propose a workaround for this ?

What exactly do you mean by 'to late'? Which init system do you use?

On first boot the ssh keys must be generated (which can use some time
depending on the hardware and the random source)...., random generation
can be speed up by e.g. using/enabling haveged...

Regards,
Peter

> And thank you very much.
>
> Best regards,
> omar

[Buildroot] SSH server starts too late

[Hammami Omar]

Hello Peter,

Thank you very much for your response.

By too late, I mean that my ssh server starts ~20 seconds later comparing
to the normal  (Old buildroot version in my case).
Also, I am using the busybox init system. And I am thinking to use the
systemd init system, so that I can see the dependencies of the ssh process
and their execution time.

Now I have one question about haveged:
Is haveged used only with systemd init system ? Or it can be used also with
busybox init system ?
Is it possible to enable havged from menuconfig of buildroot ?

Best regards,
omar

Le ven. 21 f?vr. 2020 ? 11:19, Peter Seiderer <ps.report@gmx.net> a ?crit :

> Hello Hammami,
>
> On Fri, 21 Feb 2020 10:13:17 +0100, Hammami Omar <omar18hammami@gmail.com>
> wrote:
>
> > Hello,
> >
> > I am using the buildroot version "2019.11.1" and I have noticed that my
> SSH
> > server starts too late.
> > I was using an old version "2017.08" and the SSH server was starting
> > normally.
> > Are you aware of this issue please ?
> > And do you propose a workaround for this ?
>
> What exactly do you mean by 'to late'? Which init system do you use?
>
> On first boot the ssh keys must be generated (which can use some time
> depending on the hardware and the random source)...., random generation
> can be speed up by e.g. using/enabling haveged...
>
> Regards,
> Peter
>
> > And thank you very much.
> >
> > Best regards,
> > omar
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200221/10edb879/attachment.html>

[Buildroot] SSH server starts too late

[Hammami Omar]

Hello,

I have added the package haveged to my Linux image. However, the problem is
not resolved. In fact, the ssh server have started too late.

Regards,
omar

Le ven. 21 f?vr. 2020 ? 15:50, Hammami Omar <omar18hammami@gmail.com> a
?crit :

> Hello Peter,
>
> Thank you very much for your response.
>
> By too late, I mean that my ssh server starts ~20 seconds later comparing
> to the normal  (Old buildroot version in my case).
> Also, I am using the busybox init system. And I am thinking to use the
> systemd init system, so that I can see the dependencies of the ssh process
> and their execution time.
>
> Now I have one question about haveged:
> Is haveged used only with systemd init system ? Or it can be used also
> with busybox init system ?
> Is it possible to enable havged from menuconfig of buildroot ?
>
> Best regards,
> omar
>
> Le ven. 21 f?vr. 2020 ? 11:19, Peter Seiderer <ps.report@gmx.net> a
> ?crit :
>
>> Hello Hammami,
>>
>> On Fri, 21 Feb 2020 10:13:17 +0100, Hammami Omar <omar18hammami@gmail.com>
>> wrote:
>>
>> > Hello,
>> >
>> > I am using the buildroot version "2019.11.1" and I have noticed that my
>> SSH
>> > server starts too late.
>> > I was using an old version "2017.08" and the SSH server was starting
>> > normally.
>> > Are you aware of this issue please ?
>> > And do you propose a workaround for this ?
>>
>> What exactly do you mean by 'to late'? Which init system do you use?
>>
>> On first boot the ssh keys must be generated (which can use some time
>> depending on the hardware and the random source)...., random generation
>> can be speed up by e.g. using/enabling haveged...
>>
>> Regards,
>> Peter
>>
>> > And thank you very much.
>> >
>> > Best regards,
>> > omar
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200221/deabe4cc/attachment.html>

[Buildroot] SSH server starts too late

[Grant Edwards]

On 2020-02-21, Hammami Omar <omar18hammami@gmail.com> wrote:

> By too late, I mean that my ssh server starts ~20 seconds later comparing
> to the normal  (Old buildroot version in my case).

Does it generate a new host key each time it starts?

-- 
Grant Edwards               grant.b.edwards        Yow! Is a tattoo real, like
                                  at               a curb or a battleship?
                              gmail.com            Or are we suffering in
                                                   Safeway?

[Buildroot] SSH server starts too late

[Hammami Omar]

hello Grant,

No, I have always the same "/var/ssh_host_rsa_key"
The haveged script was started, I have very high value of entropy (2332)
and the sshd process started too late.


Le ven. 21 f?vr. 2020 ? 19:10, Grant Edwards <grant.b.edwards@gmail.com> a
?crit :

> On 2020-02-21, Hammami Omar <omar18hammami@gmail.com> wrote:
>
> > By too late, I mean that my ssh server starts ~20 seconds later comparing
> > to the normal  (Old buildroot version in my case).
>
> Does it generate a new host key each time it starts?
>
> --
> Grant Edwards               grant.b.edwards        Yow! Is a tattoo real,
> like
>                                   at               a curb or a battleship?
>                               gmail.com            Or are we suffering in
>                                                    Safeway?
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200221/b550efcd/attachment.html>

[Buildroot] SSH server starts too late

[Hammami Omar]

Hello,

My problem is not resolved yet.
In fact, I have known that the libopenssl *1.1.1d* use getrandom function
in blocking mode.
So to disable this I have applied the modification below in
*crypto/rand/rand_unix.c* . But my problem is not resolved. In fact, my ssh
server starts too late (after ~2 minutes).

patch :
*# if defined(__linux) && defined(__NR_getrandom) *

* - return syscall(__NR_getrandom, buf, buflen, 0); *
*  +   return syscall(__NR_getrandom, buf, buflen, * *GRND_NONBLOCK* *);*

Did I miss something ?

In fact, I applied this modification because I saw that my ssh server
started only if the* nonblocking pool was initialized*.

Is it possible that  "*OPENSSL_RAND_SEED_GETRANDOM"* is not defined ?


*Note :*
I am using the openssh version* 8.1p1*

Kind regards,
Omar

Le ven. 21 f?vr. 2020 ? 19:30, Hammami Omar <omar18hammami@gmail.com> a
?crit :

> hello Grant,
>
> No, I have always the same "/var/ssh_host_rsa_key"
> The haveged script was started, I have very high value of entropy (2332)
> and the sshd process started too late.
>
>
> Le ven. 21 f?vr. 2020 ? 19:10, Grant Edwards <grant.b.edwards@gmail.com>
> a ?crit :
>
>> On 2020-02-21, Hammami Omar <omar18hammami@gmail.com> wrote:
>>
>> > By too late, I mean that my ssh server starts ~20 seconds later
>> comparing
>> > to the normal  (Old buildroot version in my case).
>>
>> Does it generate a new host key each time it starts?
>>
>> --
>> Grant Edwards               grant.b.edwards        Yow! Is a tattoo real,
>> like
>>                                   at               a curb or a battleship?
>>                               gmail.com            Or are we suffering in
>>                                                    Safeway?
>>
>> _______________________________________________
>> buildroot mailing list
>> buildroot at busybox.net
>> http://lists.busybox.net/mailman/listinfo/buildroot
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200303/67072457/attachment.html>

[Buildroot] SSH server starts too late

[Hammami Omar]

Hello,

Does anyone know the modification or the patch that was applied to resolve
the problem in the kernel version 5.4.y ?

Regards,
Omar

Le lun. 24 f?vr. 2020 ? 15:07, Hammami Omar <omar18hammami@gmail.com> a
?crit :

> Hi,
> In fact, my problem is not seen in my PC. Rather, it is seen in my
> embedded board which has a Linux OS.
> That is why I cannot use a keyboard.
>
>
> Le lun. 24 f?vr. 2020 ? 14:22, Andreas Ziegler <br015@umbiko.net> a
> ?crit :
>
>> On 2020-02-24 13:33, Hammami Omar wrote:
>>
>> Hi Omar,
>>
>> > The problem is that, in kernel logs, I cannot see that my ssh server
>> > was blocked.
>> > That is why, I am thinking if my problem is really du to entropy or
>> > no.
>>
>> You should experience a time lag in kernel log time stamps when that
>> happens. If this lag is caused by an entropy issue, starting to type
>> randomly on the keyboard should resolve it quite fast. This will speed
>> up the time until you see the final random output:
>>
>> > [   77.847665] random: nonblocking pool is initialized
>>
>> The SSH server will definitely not be operative until this pool is
>> initialized.
>>
>> Kind regards,
>> Andreas
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200224/3782cf87/attachment.html>

[Buildroot] SSH server starts too late

[Hammami Omar]

Hi,
In fact, my problem is not seen in my PC. Rather, it is seen in my embedded
board which has a Linux OS.
That is why I cannot use a keyboard.


Le lun. 24 f?vr. 2020 ? 14:22, Andreas Ziegler <br015@umbiko.net> a ?crit :

> On 2020-02-24 13:33, Hammami Omar wrote:
>
> Hi Omar,
>
> > The problem is that, in kernel logs, I cannot see that my ssh server
> > was blocked.
> > That is why, I am thinking if my problem is really du to entropy or
> > no.
>
> You should experience a time lag in kernel log time stamps when that
> happens. If this lag is caused by an entropy issue, starting to type
> randomly on the keyboard should resolve it quite fast. This will speed
> up the time until you see the final random output:
>
> > [   77.847665] random: nonblocking pool is initialized
>
> The SSH server will definitely not be operative until this pool is
> initialized.
>
> Kind regards,
> Andreas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200224/32f5e76c/attachment.html>

[Buildroot] SSH server starts too late

[Andreas Ziegler]

On 2020-02-24 13:33, Hammami Omar wrote:

Hi Omar,

> The problem is that, in kernel logs, I cannot see that my ssh server
> was blocked.
> That is why, I am thinking if my problem is really du to entropy or
> no.

You should experience a time lag in kernel log time stamps when that 
happens. If this lag is caused by an entropy issue, starting to type 
randomly on the keyboard should resolve it quite fast. This will speed 
up the time until you see the final random output:

> [   77.847665] random: nonblocking pool is initialized

The SSH server will definitely not be operative until this pool is 
initialized.

Kind regards,
Andreas

[Buildroot] SSH server starts too late

[Andreas Ziegler]

> Message: 31
> Date: Mon, 24 Feb 2020 10:29:42 +0100
> From: Hammami Omar <omar18hammami@gmail.com>
> To: Andreas Ziegler <br015@umbiko.net>
> Cc: buildroot at busybox.net, Peter Seiderer <ps.report@gmx.net>
> Subject: Re: [Buildroot] SSH server starts too late

> Hello Andreas,
> 
> Thank you for your response.
> In fact, I have tried to add extra randomnes by enabling "haveged". The
> entropy has increased but the problem is the same.
> Is it possible that the entropy value is always less than the wanted 
> one ?
> 
> Kind regards,
> Omar

Hi Omar,

The kernel entropy pool needs to be "seeded"; after that it works as 
intended. Until the seeding is finished, calls to getentropy() or reads 
from /dev/random block; reads from /dev/urandom work, but print warnings 
in the kernel log. Thus it seems that there is always less entropy than 
needed, because successful reads do not produce warning messages:

# dmesg | grep random
[    0.070842] 000: random: get_random_u32 called from 0x8b299601 with 
crng_init=0
[    9.553915] 000: random: fast init done
[   11.522913] 000: random: dd: uninitialized urandom read (512 bytes 
read)
[   14.271888] 000: random: wpa_supplicant: uninitialized urandom read 
(32 bytes read)
[   14.307673] 000: random: mktemp: uninitialized urandom read (6 bytes 
read)
[   23.668125] 000: random: mktemp: uninitialized urandom read (6 bytes 
read)
[   23.863680] 000: random: mktemp: uninitialized urandom read (6 bytes 
read)
[   24.003307] 000: random: sshd: uninitialized urandom read (32 bytes 
read)
[  221.067499] 000: random: sshd: uninitialized urandom read (32 bytes 
read)
[  221.092863] 000: random: sshd: uninitialized urandom read (32 bytes 
read)
[  221.405090] 000: random: sshd: uninitialized urandom read (32 bytes 
read)
[  327.117294] 000: random: crng init done
[  327.117305] 000: random: 1 urandom warning(s) missed due to 
ratelimiting

Changes in libopenssh, starting with version 1.1.1c, try to enforce a 
blocking behaviour (regardless of the device used) until the kernel pool 
is ready.

Increasing entropy can be achieved by typing on the keyboard, generating 
I/O from physical disks, or by using hardware devices (RNG). Another 
source of randomness is the patch that was introduced by Linus Torvalds 
in kernel 5.4.y:

   https://lkml.org/lkml/2019/9/18/1078

Kind regards,
Andreas

[Buildroot] SSH server starts too late

[Hammami Omar]

Hello Andreas,

Thank you for your response.
In fact, I have tried to add extra randomnes by enabling "haveged". The
entropy has increased but the problem is the same.
Is it possible that the entropy value is always less than the wanted one ?

Kind regards,
Omar

Le ven. 21 f?vr. 2020 ? 13:51, Andreas Ziegler <br015@umbiko.net> a ?crit :

> Hi Omar,
>
> On 2020-02-21 13:00, Hammami Omar <omar18hammami@gmail.com> wrote
> > I am using the buildroot version "2019.11.1" and I have noticed that my
> > SSH
> > server starts too late.
> > I was using an old version "2017.08" and the SSH server was starting
> > normally.
>
> This issue is not related to Buildroot, but caused by changes in
> libopenssl (which in turn is used by OpenSSH), starting around the
> middle of last year. You will find a lot of background information when
> you search the www for 'boot-time entropy starvation'; the cause is,
> reading random data from /dev/urandom blocks until the kernel entropy
> pool has been initialized.
>
> Workarounds:
>
> (a) Patch libopenssl (not recommended for nodes directly connected to
> the WAN).
> (b) Use a more recent kernel; version 5.4.y introduced a temporary fix
> for this issue.
> (c) Use other sources of extra randomness, as proposed by Peter
> Seiderer.
>
> Kind regards,
> Andreas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200224/28027cac/attachment.html>

[Buildroot] SSH server starts too late

[Andreas Ziegler]

Hi Omar,

On 2020-02-21 13:00, Hammami Omar <omar18hammami@gmail.com> wrote
> I am using the buildroot version "2019.11.1" and I have noticed that my 
> SSH
> server starts too late.
> I was using an old version "2017.08" and the SSH server was starting
> normally.

This issue is not related to Buildroot, but caused by changes in 
libopenssl (which in turn is used by OpenSSH), starting around the 
middle of last year. You will find a lot of background information when 
you search the www for 'boot-time entropy starvation'; the cause is, 
reading random data from /dev/urandom blocks until the kernel entropy 
pool has been initialized.

Workarounds:

(a) Patch libopenssl (not recommended for nodes directly connected to 
the WAN).
(b) Use a more recent kernel; version 5.4.y introduced a temporary fix 
for this issue.
(c) Use other sources of extra randomness, as proposed by Peter 
Seiderer.

Kind regards,
Andreas