From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1752988AbbDAV7o (ORCPT <rfc822;w@1wt.eu>);
	Wed, 1 Apr 2015 17:59:44 -0400
Received: from youngberry.canonical.com ([91.189.89.112]:46784 "EHLO
	youngberry.canonical.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1752427AbbDAV7l (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Wed, 1 Apr 2015 17:59:41 -0400
Message-ID: <551C6A48.9060805@canonical.com>
Date: Wed, 01 Apr 2015 16:59:36 -0500
From: Chris J Arges <chris.j.arges@canonical.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0
MIME-Version: 1.0
To: Linus Torvalds <torvalds@linux-foundation.org>
CC: Ingo Molnar <mingo@kernel.org>, Rafael David Tinoco <inaddy@ubuntu.com>,
        Peter Anvin <hpa@zytor.com>, Jiang Liu <jiang.liu@linux.intel.com>,
        Peter Zijlstra <peterz@infradead.org>,
        LKML <linux-kernel@vger.kernel.org>, Jens Axboe <axboe@kernel.dk>,
        Frederic Weisbecker <fweisbec@gmail.com>,
        Gema Gomez <gema.gomez-solano@canonical.com>,
        the arch/x86 maintainers <x86@kernel.org>
Subject: Re: smp_call_function_single lockups
References: <CAMiJ5CU+rvQr-_Ejd3m3ha3HsiSKu0Sq_fTaE2Ws_c_01=qbLQ@mail.gmail.com>	<CA+55aFxWBKHth7x3FJ+dpGfy0ZT7SUhHnX7tDfgDo-wXTeX5Lg@mail.gmail.com>	<CA+55aFx2n9zsqwuW=p6KJF62rXp+9_M-HF3wbeJRA-MeT0XLLw@mail.gmail.com>	<CA+55aFyv1pJod7bhetc0ikmuCKzE=uhmT14KMju_fTbP93gLWA@mail.gmail.com>	<CA+55aFxd1WGNBzSHeOGiXXdUD1GqDYv9PUNGdrdiGFwaX7HYJQ@mail.gmail.com>	<CA+55aFxFkw7cKu6R8-v9z=c+yG+jsPHyQKW5-yyn3+M0BuyvxA@mail.gmail.com>	<20150331031536.GA9303@canonical.com>	<CA+55aFykg3SAO16=NRiC+tP1gGj5hgbu+Y93ss4Qg30+qyZ=+w@mail.gmail.com>	<20150331222327.GA12512@canonical.com>	<20150401124336.GB12841@gmail.com>	<20150401161047.GD12730@canonical.com> <CA+55aFxQ6q7MNS+4XWZ3=Xa0Hz6kumd84v_aEw3M4gBpXszTkQ@mail.gmail.com>
In-Reply-To: <CA+55aFxQ6q7MNS+4XWZ3=Xa0Hz6kumd84v_aEw3M4gBpXszTkQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org



On 04/01/2015 11:14 AM, Linus Torvalds wrote:
> On Wed, Apr 1, 2015 at 9:10 AM, Chris J Arges
> <chris.j.arges@canonical.com> wrote:
>>
>> Even with irqbalance removed from the L0/L1 machines the hang still occurs.
>>
>> This results in no 'apic: vector* or 'ack_APIC*' messages being displayed.
> 
> Ok. So the ack_APIC debug patch found *something*, but it seems to be
> unrelated to the hang.
> 
> Dang. Oh well.  Back to square one.
> 
>                            Linus
> 

With my L0 testing I've normally used a 3.13 series kernel since it
tends to reproduce the hang very quickly with the testcase. Note we have
reproduced an identical hang with newer kernels (3.19+patches) using the
openstack tempest on openstack reproducer, but the timing can vary
between hours and days.

Installing a v4.0-rc6+patch kernel on L0 makes the problem very slow to
reproduce, so I am running these tests now which may take day(s).

It is worthwhile to do a 'bisect' to see where on average it takes
longer to reproduce? Perhaps it will point to a relevant change, or it
may be completely useless.

--chris