Re: any remaining for-6.0 issues?

From: Cornelia Huck <cohuck@redhat.com>
To: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Cc: "Laurent Vivier" <lvivier@redhat.com>,
	"Peter Maydell" <peter.maydell@linaro.org>,
	"Thomas Huth" <thuth@redhat.com>,
	"Philippe Mathieu-Daudé" <philmd@redhat.com>,
	"Richard Henderson" <richard.henderson@linaro.org>,
	"QEMU Developers" <qemu-devel@nongnu.org>,
	"Alex Bennée" <alex.bennee@linaro.org>
Subject: Re: any remaining for-6.0 issues?
Date: Mon, 19 Apr 2021 19:02:24 +0200	[thread overview]
Message-ID: <20210419190224.33408d0e.cohuck@redhat.com> (raw)
In-Reply-To: <20210419090551.4a3ae4eb.cohuck@redhat.com>

On Mon, 19 Apr 2021 09:05:51 +0200
Cornelia Huck <cohuck@redhat.com> wrote:

> On Sun, 18 Apr 2021 11:38:09 +0100
> Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk> wrote:
> 
> > On 17/04/2021 20:46, Peter Maydell wrote:
> >   
> > >> On 14/04/2021 13:48, Thomas Huth wrote:    
> > >>> I've seen some intermittend, non-reproducible crashes with usermode QEMU in some of
> > >>> my CI runs, e.g.:
> > >>>
> > >>> https://gitlab.com/thuth/qemu/-/jobs/1178256498#L3876
> > >>>
> > >>> https://gitlab.com/thuth/qemu/-/jobs/1146276208#L3241
> > >>>
> > >>> This was only with ccache enabled, so I thought that it might be related to my
> > >>> work-in-progress ccache patches...
> > >>>
> > >>> ... but now Cornelia reported that she has seen such a crash in one of her branches,
> > >>> too (which is completely unrelated to my ccache patches):
> > >>>
> > >>> https://gitlab.com/cohuck/qemu/-/jobs/1178860927#L3867
> > >>>
> > >>> That makes me wonder whether we currently have a real problem with user-mode in the
> > >>> master branch? Did anybody else see such problems?    
> > >>
> > >> I've definitely seen the same issue as Cornelia in my Gitlab CI builds for the ESP
> > >> security fixes (first version of which appeared just before rc0). The user builds
> > >> always fail on "run-tcg-tests-s390x-linux-user" for me.    
> > > 
> > > Do we have any better understanding yet of the cause here?
> > > (I ask because I think we're going to need an rc4 for other reasons,
> > > so if there's a ready-to-go fix then we could consider it.)    
> > 
> > I don't think so. I tried a run with a possible candidate patch reverted (see 
> > https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg02345.html) but Cornelia's 
> > response indicates that the result is still inconclusive :(  
> 
> That patch seems to be our best candidate so far, but the intermittent
> nature of the failures make it hard to pin down... I don't see anything
> obviously wrong with the patch, maybe some linux-user experts have a
> better idea?

FWIW, I tried reproducing the issue on some local systems (no luck),
and on code pushed out to gitlab (where it works most of the time, and
the user builds where it fails are unpredictable.)

I fear the best we can do right now is stare at the code and try to
figure out what might be wrong :(