From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06F85C433DB for ; Tue, 2 Mar 2021 17:27:01 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 967FC64F18 for ; Tue, 2 Mar 2021 17:27:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 967FC64F18 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id EBC836E226; Tue, 2 Mar 2021 17:26:59 +0000 (UTC) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5C2356E226 for ; Tue, 2 Mar 2021 17:26:59 +0000 (UTC) IronPort-SDR: 9l++vqzmtv3DepDbtR3exMNI9E8WaeLkf5nN0k9hc/NfTdgsV1MzaqWSceToRdbxVPYhGGDvxh 436pwwsjP67w== X-IronPort-AV: E=McAfee;i="6000,8403,9911"; a="166787212" X-IronPort-AV: E=Sophos;i="5.81,217,1610438400"; d="scan'208,217";a="166787212" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Mar 2021 09:26:58 -0800 IronPort-SDR: J1Z5QwVj55AXr4TvoiHeZ0BLdFmWU7gKGwtxL95vDnTSDkowusBYPVZDI0ZddYoCr4SWUEzYp0 zDY2hSOQYZtg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,217,1610438400"; d="scan'208,217";a="517926185" Received: from irsmsx602.ger.corp.intel.com ([163.33.146.8]) by orsmga004.jf.intel.com with ESMTP; 02 Mar 2021 09:26:57 -0800 Received: from irsmsx601.ger.corp.intel.com (163.33.146.7) by irsmsx602.ger.corp.intel.com (163.33.146.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2; Tue, 2 Mar 2021 17:26:57 +0000 Received: from irsmsx601.ger.corp.intel.com ([163.33.146.7]) by irsmsx601.ger.corp.intel.com ([163.33.146.7]) with mapi id 15.01.2106.002; Tue, 2 Mar 2021 17:26:56 +0000 From: "Sarvela, Tomi P" To: "intel-gfx@lists.freedesktop.org" Thread-Topic: Public i915 CI shardruns are disabled Thread-Index: AdcPWHkSgYTOIBqtStq84U3dnNSQvwAC8QNQAAhuRtA= Date: Tue, 2 Mar 2021 17:26:56 +0000 Message-ID: References: <51946a94b1154605bd7dda2c77ab12fc@intel.com> In-Reply-To: <51946a94b1154605bd7dda2c77ab12fc@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.6.0.76 x-originating-ip: [163.33.253.164] MIME-Version: 1.0 Subject: Re: [Intel-gfx] Public i915 CI shardruns are disabled X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0458258966==" Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" --===============0458258966== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_fb8a2d722d4b4c008eeb1ffae87233beintelcom_" --_000_fb8a2d722d4b4c008eeb1ffae87233beintelcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable The regression has been identified; Chris Wilson found commits touching swapfile.c, and reverting them the issue couldn't be reproduced any more. https://patchwork.freedesktop.org/series/87549/ This revert will be applied to core-for-CI branch. When new CI_DRM has been built, shard-testing will be enabled again. Regards, Tomi Sarvela From: Sarvela, Tomi P More information (excuse my top-posting): - Issue happens in igt@gem_tiled_swapping@non-threaded Mlocking phase, before "starting subtest" appears. - Filesystem trashed is the one containing swapfile - If swap is partition, it seems that the swap signature is correct even after running the test, so for now I'm assuming that the issue has to do with swapfile - Bisection between 20210129 and 20210215 proved to be challenging, because the kernels have pre-init hang, don't leave dmesg and I don't have console on testing host. Petri's suggestion to bisect between CI_DRM_9817 and 9818 might work better Regards, Tomi Sarvela From: Sarvela, Tomi P Hello, The linux i915 CI shardruns have been disabled. This is due to the unfortun= ate filesystem-corrupting bug first seen in linux-next 20210215, which now has been merged to linus 5.12-rc1 and further on to DRM-Tip, first instance see= n in CI_DRM_9818. Last changes coming in were: fb3b93df7979 drm-tip: 2021y-03m-01d-09h-36m-57s UTC integration manifest 3b3c4086295b drm-tip: 2021y-03m-01d-08h-49m-06s UTC integration manifest fe07bfda2fb9 Linux 5.12-rc1 More information can be seen at: https://phoronix.com/scan.php?page=3Dnews_item&px=3DLinux-5.12-Early-Buggy-= Issue I've seen this bug happen regularly with (but not limited to) IGT test: igt@gem_tiled_swapping@non-threaded The range for bisection is linux-next 20210215 to 20210129 because the kern= els in-between taint the kernel and our i915 testing was not done. Hitting the = bug corrupts the underlying filesystem very thoroughly, wiping out large amount= of data from the beginning of the partition which leaves fsck sad with thousan= ds of items lost. Bisection of the IGT testlist was done with two root filesystem= s, where testable kernel booted from 2. partition, and copy of the 2. partition was = stored on 1. partition and could be restored at will. I'll continue bisecting this bug on the linux-next tree again. If someone h= as more information where this issue originates from, help would be appreciated. Regards, Tomi Sarvela -- Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo --_000_fb8a2d722d4b4c008eeb1ffae87233beintelcom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

The regression has been identif= ied; Chris Wilson found commits touching

swapfile.c, and reverting them = the issue couldn’t be reproduced any more.

 

https://patchwork.freedesktop.org/series/87549= /

 

This revert will be applied to = core-for-CI branch. When new CI_DRM has

been built, shard-testing will = be enabled again.

 

Regards,

 

Tomi Sarvela<= /p>

 

From: Sarvela, Tomi P

More information (excuse my top= -posting):

 

- Issue happens in igt@gem_tile= d_swapping@non-threaded Mlocking

phase, before “starting s= ubtest” appears.

 

- Filesystem trashed is the one= containing swapfile

 

- If swap is partition, it seem= s that the swap signature is correct even

after running the test, so for = now I’m assuming that the issue has to do

with swapfile=

 

- Bisection between 20210129 an= d 20210215 proved to be challenging,

because the kernels have pre-in= it hang, don’t leave dmesg and I don’t

have console on testing host. P= etri’s suggestion to bisect between

CI_DRM_9817 and 9818 might work= better

 

Regards,

 

Tomi Sarvela<= /p>

 

From: Sarvela, Tomi P

Hello,

 

The linux i915 CI shardruns hav= e been disabled. This is due to the unfortunate

filesystem-corrupting bug first= seen in linux-next 20210215, which now has

been merged to linus 5.12-rc1 a= nd further on to DRM-Tip, first instance seen

in CI_DRM_9818. Last changes co= ming in were:

 

fb3b93df7979 drm-tip: 2021y-03m= -01d-09h-36m-57s UTC integration manifest

3b3c4086295b drm-tip: 2021y-03m= -01d-08h-49m-06s UTC integration manifest

fe07bfda2fb9 Linux 5.12-rc1

 

More information can be seen at= :

https://p= horonix.com/scan.php?page=3Dnews_item&px=3DLinux-5.12-Early-Buggy-Issue=

 

I’ve seen this bug happen= regularly with (but not limited to) IGT test:

igt@gem_tiled_swapping@non-thre= aded

 

The range for bisection is linu= x-next 20210215 to 20210129 because the kernels

in-between taint the kernel and= our i915 testing was not done. Hitting the bug

corrupts the underlying filesys= tem very thoroughly, wiping out large amount of

data from the beginning of the = partition which leaves fsck sad with thousands of

items lost. Bisection of the IG= T testlist was done with two root filesystems, where

testable kernel booted from 2. = partition, and copy of the 2. partition was stored

on 1. partition and could be re= stored at will.

 

I’ll continue bisecting t= his bug on the linux-next tree again. If someone has more=

information where this issue or= iginates from, help would be appreciated.

 

Regards,

 

Tomi Sarvela<= /p>

 

--

Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 = Espoo=

 

--_000_fb8a2d722d4b4c008eeb1ffae87233beintelcom_-- --===============0458258966== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/intel-gfx --===============0458258966==--