[PATCH 1/2] dropbear/openssh: Lower priority of key generation

[PATCH 1/2] dropbear/openssh: Lower priority of key generation

[Richard Purdie]

Where we have images with PAM+systemd, serial login can be extremely
slow. The load generated by key generation does slow down the rest
of the boot process.

Lower the priority level of these systemd services, since we'd
prefer to have the rest of the system boot more effectively.

This doesn't "solve" the slow systemd boot issues but does help.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-connectivity/openssh/openssh/sshdgenkeys.service | 1 +
 meta/recipes-core/dropbear/dropbear/dropbearkey.service       | 1 +
 2 files changed, 2 insertions(+)

diff --git a/meta/recipes-connectivity/openssh/openssh/sshdgenkeys.service b/meta/recipes-connectivity/openssh/openssh/sshdgenkeys.service
index 603c33787f1..fd81793d511 100644
--- a/meta/recipes-connectivity/openssh/openssh/sshdgenkeys.service
+++ b/meta/recipes-connectivity/openssh/openssh/sshdgenkeys.service
@@ -6,3 +6,4 @@ RequiresMountsFor=/var /run
 ExecStart=@LIBEXECDIR@/sshd_check_keys
 Type=oneshot
 RemainAfterExit=yes
+Nice=10
diff --git a/meta/recipes-core/dropbear/dropbear/dropbearkey.service b/meta/recipes-core/dropbear/dropbear/dropbearkey.service
index c49053d57c7..71a12a6110c 100644
--- a/meta/recipes-core/dropbear/dropbear/dropbearkey.service
+++ b/meta/recipes-core/dropbear/dropbear/dropbearkey.service
@@ -11,3 +11,4 @@ Type=oneshot
 ExecStart=@BASE_BINDIR@/mkdir -p ${DROPBEAR_RSAKEY_DIR}
 ExecStart=@SBINDIR@/dropbearkey -t rsa -f ${DROPBEAR_RSAKEY_DIR}/dropbear_rsa_host_key
 RemainAfterExit=yes
+Nice=10
-- 
2.25.1

[PATCH 2/2] oeqa/qemurunner: Increase serial timeout

[Richard Purdie]

Increase the serial login timeout from 60 to 120s. This seems like a
long time, however for a qemumips image with systemd+PAM and openssh,
(e.g. core-image-sato-sdk + DISTRO=poky-altcfg), the getty connects
to systemd's pam module which waits on logind and 45s for all this
to happen at the same time as things like ssh key generation happens
is not unknown.

Increase the timeout to match the longer times we know these things
can take in the worst case scenarios since we're tired of intermittent
issues related to the serial login affecting the autobuilder.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/utils/qemurunner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index 519aa9aa1e5..77ec939ad72 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -393,7 +393,7 @@ class QemuRunner:
 
         # If we are not able to login the tests can continue
         try:
-            (status, output) = self.run_serial(self.boot_patterns['send_login_user'], raw=True)
+            (status, output) = self.run_serial(self.boot_patterns['send_login_user'], raw=True, timeout=120)
             if re.search(self.boot_patterns['search_login_succeeded'], output):
                 self.logged = True
                 self.logger.debug("Logged as root in serial console")
-- 
2.25.1

Re: [OE-core] [PATCH 1/2] dropbear/openssh: Lower priority of key generation

[Khem Raj]

On 9/17/20 11:00 PM, Richard Purdie wrote:
> Where we have images with PAM+systemd, serial login can be extremely
> slow. The load generated by key generation does slow down the rest
> of the boot process.
> 

this is a good find. So I wonder if its because we have entropy 
starvation. I see that we do enable virtio-rng while launching qemu so 
something to look in this area perhaps.


> Lower the priority level of these systemd services, since we'd
> prefer to have the rest of the system boot more effectively.
> 
> This doesn't "solve" the slow systemd boot issues but does help.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>   meta/recipes-connectivity/openssh/openssh/sshdgenkeys.service | 1 +
>   meta/recipes-core/dropbear/dropbear/dropbearkey.service       | 1 +
>   2 files changed, 2 insertions(+)
> 
> diff --git a/meta/recipes-connectivity/openssh/openssh/sshdgenkeys.service b/meta/recipes-connectivity/openssh/openssh/sshdgenkeys.service
> index 603c33787f1..fd81793d511 100644
> --- a/meta/recipes-connectivity/openssh/openssh/sshdgenkeys.service
> +++ b/meta/recipes-connectivity/openssh/openssh/sshdgenkeys.service
> @@ -6,3 +6,4 @@ RequiresMountsFor=/var /run
>   ExecStart=@LIBEXECDIR@/sshd_check_keys
>   Type=oneshot
>   RemainAfterExit=yes
> +Nice=10
> diff --git a/meta/recipes-core/dropbear/dropbear/dropbearkey.service b/meta/recipes-core/dropbear/dropbear/dropbearkey.service
> index c49053d57c7..71a12a6110c 100644
> --- a/meta/recipes-core/dropbear/dropbear/dropbearkey.service
> +++ b/meta/recipes-core/dropbear/dropbear/dropbearkey.service
> @@ -11,3 +11,4 @@ Type=oneshot
>   ExecStart=@BASE_BINDIR@/mkdir -p ${DROPBEAR_RSAKEY_DIR}
>   ExecStart=@SBINDIR@/dropbearkey -t rsa -f ${DROPBEAR_RSAKEY_DIR}/dropbear_rsa_host_key
>   RemainAfterExit=yes
> +Nice=10
> 
> 
> 
> 
> 

Re: [OE-core] [PATCH 1/2] dropbear/openssh: Lower priority of key generation

[Richard Purdie]

On Fri, 2020-09-18 at 10:45 -0700, Khem Raj wrote:
> 
> On 9/17/20 11:00 PM, Richard Purdie wrote:
> > Where we have images with PAM+systemd, serial login can be
> > extremely
> > slow. The load generated by key generation does slow down the rest
> > of the boot process.
> > 
> 
> this is a good find. So I wonder if its because we have entropy 
> starvation. I see that we do enable virtio-rng while launching qemu
> so something to look in this area perhaps.

I did check and the virtio rng is alive and well in the image. Its the
CPU usage of the key generation that partly causes the issue.

Cheers,

Richard

Re: [OE-core] [PATCH 1/2] dropbear/openssh: Lower priority of key generation

[Mike Looijmans]

Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands

T: +31 (0) 499 33 69 69
E: mike.looijmans@topicproducts.com
W: www.topicproducts.com

Please consider the environment before printing this e-mail
On 18-09-2020 19:55, Richard Purdie via lists.openembedded.org wrote:
> On Fri, 2020-09-18 at 10:45 -0700, Khem Raj wrote:
>> On 9/17/20 11:00 PM, Richard Purdie wrote:
>>> Where we have images with PAM+systemd, serial login can be
>>> extremely
>>> slow. The load generated by key generation does slow down the rest
>>> of the boot process.
>>>
>> this is a good find. So I wonder if its because we have entropy
>> starvation. I see that we do enable virtio-rng while launching qemu
>> so something to look in this area perhaps.
> I did check and the virtio rng is alive and well in the image. Its the
> CPU usage of the key generation that partly causes the issue.
>
>
Another solution is to just have it create smaller keys. The computation 
time seems to grow exponential with the bit count, a 2048 size key can 
take 10 times longer than a 1024 bit key to generate, and a 4096 may 
take 10x that again.

There's been a move to 4096 sizes keys lately, pushing the CPU use even 
further. Unless you're controlling a nuclear power plant or so, that's a 
bit excessive.

-- 
Mike Looijmans