[V3][PATCH] ruby: workaround ptest hang problem

[V3][PATCH] ruby: workaround ptest hang problem

[changqing.li]

From: Changqing Li <changqing.li@windriver.com>

since openssl 3 not compatible problem, ruby have disable openssl
extention. But disable openssl extention make test_smtp.rs hang at
test case "test_start".

Net::TestSMTP#test_start:
NameError: uninitialized constant Net::SMTP::OpenSSL
Did you mean? Open3
/usr/lib64/ruby/3.0.0/net/smtp.rb:195:in `default_ssl_context'
/usr/lib64/ruby/3.0.0/net/smtp.rb:552:in `start'
/usr/lib64/ruby/3.0.0/net/smtp.rb:475:in `start'
/usr/lib64/ruby/ptest/test/net/smtp/test_smtp.rb:199:in `test_start'

temporarily remove the hang case to make other testcases can be run.

Meantime, move ruby-ptest out of the PTESTS_PROBLEMS list.
On 48 core host, run ruby ptest in qemux86-64:
root@qemux86-64:/usr/lib64/ruby/ptest# time ./run-ptest
PASS: test/test_set.rb
PASS: test/stringio/test_stringio.rb
...
PASS: test/did_you_mean/test_tree_spell_checker.rb
PASS: test/test_mutex_m.rb

real	5m42.872s
user	3m50.923s
sys	0m44.136s

Signed-off-by: Changqing Li <changqing.li@windriver.com>
---
 meta/conf/distro/include/ptest-packagelists.inc | 3 +--
 meta/recipes-devtools/ruby/ruby.inc             | 4 ++++
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/conf/distro/include/ptest-packagelists.inc b/meta/conf/distro/include/ptest-packagelists.inc
index 2e324f8da4..f1c40ac813 100644
--- a/meta/conf/distro/include/ptest-packagelists.inc
+++ b/meta/conf/distro/include/ptest-packagelists.inc
@@ -81,6 +81,7 @@ PTESTS_SLOW = "\
     openssl-ptest \
     perl-ptest \
     python3-ptest \
+    ruby-ptest \
     strace-ptest \
     tcl-ptest \
     util-linux-ptest \
@@ -90,7 +91,6 @@ PTESTS_SLOW = "\
 PTESTS_SLOW:remove:riscv64 = "valgrind-ptest"
 PTESTS_PROBLEMS:append:riscv64 = "valgrind-ptest"
 
-#    ruby-ptest \ # Timeout
 #    lz4-ptest \ # Needs a rewrite
 #    rt-tests-ptest \ # Needs to be checked whether it runs at all
 #    bash-ptest \ # Test outcomes are non-deterministic by design
@@ -103,7 +103,6 @@ PTESTS_PROBLEMS:append:riscv64 = "valgrind-ptest"
 #    python3-numpy-ptest \ # requires even more RAM and (possibly) disk space; multiple failures
 
 PTESTS_PROBLEMS = "\
-    ruby-ptest \
     lz4-ptest \
     rt-tests-ptest \
     bash-ptest \
diff --git a/meta/recipes-devtools/ruby/ruby.inc b/meta/recipes-devtools/ruby/ruby.inc
index 2b5caf35f5..776c409f6c 100644
--- a/meta/recipes-devtools/ruby/ruby.inc
+++ b/meta/recipes-devtools/ruby/ruby.inc
@@ -42,3 +42,7 @@ do_configure:prepend() {
 do_configure:prepend() {
     rm -rf ${S}/ext/openssl/extconf.rb
 }
+# Remove this test temporarily until it compatible with openssl 3
+do_install_ptest:append() {
+    rm -rf ${D}${PTEST_PATH}/test/net/smtp/test_smtp.rb
+}
-- 
2.32.0

Re: [OE-core] [V3][PATCH] ruby: workaround ptest hang problem

[Richard Purdie]

On Fri, 2021-11-05 at 10:18 +0800, Changqing Li wrote:
> From: Changqing Li <changqing.li@windriver.com>
> 
> since openssl 3 not compatible problem, ruby have disable openssl
> extention. But disable openssl extention make test_smtp.rs hang at
> test case "test_start".
> 
> Net::TestSMTP#test_start:
> NameError: uninitialized constant Net::SMTP::OpenSSL
> Did you mean? Open3
> /usr/lib64/ruby/3.0.0/net/smtp.rb:195:in `default_ssl_context'
> /usr/lib64/ruby/3.0.0/net/smtp.rb:552:in `start'
> /usr/lib64/ruby/3.0.0/net/smtp.rb:475:in `start'
> /usr/lib64/ruby/ptest/test/net/smtp/test_smtp.rb:199:in `test_start'
> 
> temporarily remove the hang case to make other testcases can be run.
> 
> Meantime, move ruby-ptest out of the PTESTS_PROBLEMS list.
> On 48 core host, run ruby ptest in qemux86-64:
> root@qemux86-64:/usr/lib64/ruby/ptest# time ./run-ptest
> PASS: test/test_set.rb
> PASS: test/stringio/test_stringio.rb
> ...
> PASS: test/did_you_mean/test_tree_spell_checker.rb
> PASS: test/test_mutex_m.rb
> 
> real	5m42.872s
> user	3m50.923s
> sys	0m44.136s
> 
> Signed-off-by: Changqing Li <changqing.li@windriver.com>
> ---
>  meta/conf/distro/include/ptest-packagelists.inc | 3 +--
>  meta/recipes-devtools/ruby/ruby.inc             | 4 ++++
>  2 files changed, 5 insertions(+), 2 deletions(-)

Unfortunately the success rate isn't like that when we tried this on the
autobuilder, it failed on both arm64 and x86_64 with multiple failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2791
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2507


Cheers,

Richard

Re: [OE-core] [V3][PATCH] ruby: workaround ptest hang problem

[Changqing Li]

On 11/6/21 5:27 PM, Richard Purdie wrote:
> [Please note: This e-mail is from an EXTERNAL e-mail address]
>
> On Fri, 2021-11-05 at 10:18 +0800, Changqing Li wrote:
>> From: Changqing Li <changqing.li@windriver.com>
>>
>> since openssl 3 not compatible problem, ruby have disable openssl
>> extention. But disable openssl extention make test_smtp.rs hang at
>> test case "test_start".
>>
>> Net::TestSMTP#test_start:
>> NameError: uninitialized constant Net::SMTP::OpenSSL
>> Did you mean? Open3
>> /usr/lib64/ruby/3.0.0/net/smtp.rb:195:in `default_ssl_context'
>> /usr/lib64/ruby/3.0.0/net/smtp.rb:552:in `start'
>> /usr/lib64/ruby/3.0.0/net/smtp.rb:475:in `start'
>> /usr/lib64/ruby/ptest/test/net/smtp/test_smtp.rb:199:in `test_start'
>>
>> temporarily remove the hang case to make other testcases can be run.
>>
>> Meantime, move ruby-ptest out of the PTESTS_PROBLEMS list.
>> On 48 core host, run ruby ptest in qemux86-64:
>> root@qemux86-64:/usr/lib64/ruby/ptest# time ./run-ptest
>> PASS: test/test_set.rb
>> PASS: test/stringio/test_stringio.rb
>> ...
>> PASS: test/did_you_mean/test_tree_spell_checker.rb
>> PASS: test/test_mutex_m.rb
>>
>> real  5m42.872s
>> user  3m50.923s
>> sys   0m44.136s
>>
>> Signed-off-by: Changqing Li <changqing.li@windriver.com>
>> ---
>>   meta/conf/distro/include/ptest-packagelists.inc | 3 +--
>>   meta/recipes-devtools/ruby/ruby.inc             | 4 ++++
>>   2 files changed, 5 insertions(+), 2 deletions(-)
> Unfortunately the success rate isn't like that when we tried this on the
> autobuilder, it failed on both arm64 and x86_64 with multiple failures:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2791
> https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2507

Sorry,  I thought this is expected.  Ruby have 800+ test cases, and 
failed about

30+ all the time.  so I think we still need to put ruby ptest in the  
PTESTS_PROBLEMS list until

we address all the failed cases.

>
>
> Cheers,
>
> Richard
>

Re: [OE-core] [V3][PATCH] ruby: workaround ptest hang problem

[Richard Purdie]

On Mon, 2021-11-08 at 12:02 +0800, Changqing Li wrote:
> On 11/6/21 5:27 PM, Richard Purdie wrote:
> > [Please note: This e-mail is from an EXTERNAL e-mail address]
> > 
> > On Fri, 2021-11-05 at 10:18 +0800, Changqing Li wrote:
> > > From: Changqing Li <changqing.li@windriver.com>
> > > 
> > > since openssl 3 not compatible problem, ruby have disable openssl
> > > extention. But disable openssl extention make test_smtp.rs hang at
> > > test case "test_start".
> > > 
> > > Net::TestSMTP#test_start:
> > > NameError: uninitialized constant Net::SMTP::OpenSSL
> > > Did you mean? Open3
> > > /usr/lib64/ruby/3.0.0/net/smtp.rb:195:in `default_ssl_context'
> > > /usr/lib64/ruby/3.0.0/net/smtp.rb:552:in `start'
> > > /usr/lib64/ruby/3.0.0/net/smtp.rb:475:in `start'
> > > /usr/lib64/ruby/ptest/test/net/smtp/test_smtp.rb:199:in `test_start'
> > > 
> > > temporarily remove the hang case to make other testcases can be run.
> > > 
> > > Meantime, move ruby-ptest out of the PTESTS_PROBLEMS list.
> > > On 48 core host, run ruby ptest in qemux86-64:
> > > root@qemux86-64:/usr/lib64/ruby/ptest# time ./run-ptest
> > > PASS: test/test_set.rb
> > > PASS: test/stringio/test_stringio.rb
> > > ...
> > > PASS: test/did_you_mean/test_tree_spell_checker.rb
> > > PASS: test/test_mutex_m.rb
> > > 
> > > real  5m42.872s
> > > user  3m50.923s
> > > sys   0m44.136s
> > > 
> > > Signed-off-by: Changqing Li <changqing.li@windriver.com>
> > > ---
> > >   meta/conf/distro/include/ptest-packagelists.inc | 3 +--
> > >   meta/recipes-devtools/ruby/ruby.inc             | 4 ++++
> > >   2 files changed, 5 insertions(+), 2 deletions(-)
> > Unfortunately the success rate isn't like that when we tried this on the
> > autobuilder, it failed on both arm64 and x86_64 with multiple failures:
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/2791
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/2507
> 
> Sorry,  I thought this is expected.  Ruby have 800+ test cases, and 
> failed about
> 
> 30+ all the time.  so I think we still need to put ruby ptest in the  
> PTESTS_PROBLEMS list until
> 
> we address all the failed cases.

What we could do is mask out the failing cases. The issue is that we'll see
warnings on the autobuilder for cases that fail so we need to remove any "known
to fail" cases before we can enable by default there.

Cheers,

Richard