From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6CACC432BE for ; Mon, 9 Aug 2021 18:52:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ADCF660EE7 for ; Mon, 9 Aug 2021 18:52:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235666AbhHISwd (ORCPT ); Mon, 9 Aug 2021 14:52:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235835AbhHISwc (ORCPT ); Mon, 9 Aug 2021 14:52:32 -0400 Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7DB15C06179E for ; Mon, 9 Aug 2021 11:52:08 -0700 (PDT) Received: by mail-ed1-x52a.google.com with SMTP id f13so26101952edq.13 for ; Mon, 09 Aug 2021 11:52:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=XsY1T+ZcNexFoxRjxiTUapeVjeXQ1jY55lc32SOmy1E=; b=DdAAN3VumBBn6g2xznpLrHsooL1sZzeg6QScu6NH4CXFzv2eaOE9K+sBwjGcFuFzMh vt8N7ihEmfYt2LKBeaEcXBKDKfjKwuoTpqndZPfxTs60klkoMq7vtpOnYxRO9LFOjOzW JyUBBhxtnlIDBzmUASeAOp4lxS5TjX1oqNgROGwWLp9VmpvKXH+dQT/gXVCf8HpCd8GJ 7VU5e++NkxBX8NvOJACyTyyTFSsjJKF8aW19tQkX06DNXn13UkSVUg1BEnKRO1w1OD2D 5W5twEYjTZwmIcvjx8ayrM1Slj7I+utfJX/UXKTTdU/S9eIL5PcZMDVFgzXVzd27UkOl 289Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=XsY1T+ZcNexFoxRjxiTUapeVjeXQ1jY55lc32SOmy1E=; b=PjOqVjcQbtzHCrftzZ2z0ZbpdGPIAZNmeethMTqHqpu2gXPqKr3aPAQxQqDw2zU29w JOMtjs9mNrVIK9EgGPm8jKaTnLLDsozcli+c7KzGoJflp7iihlST+xZ4Hp5VvOQeHBOS coFGCBODT5EE7fsIHRX4kV/DtTZZ+8I/oz+coIqesdgTqLRqbNY3KDY8+nN7IVpRx/+s Qm836orFn6ylgWrLIgjurZ+rqDN4hG+Tocmvpcnb9COy8ihUNgiSAXB7kzHgbo1/jAl4 ROm8xJbKjGy7rBNwS5ehlWmNAeioy1CUnVYUXXghkunmCKlKyJbMiRBLDwn69vmtJ029 WWfA== X-Gm-Message-State: AOAM531oA5AAFSazT4rhc9wOKRGZKv8KA7cQsv1Li0peMOwSWC+mAm35 popnBFWkL1vgpxHoy28VFkNPctHESTTMSMAgjW8= X-Google-Smtp-Source: ABdhPJx2zr1GH5VAhSUfe42K2jv5XXNr1qRQm1tv7/sKSCER42eOHllUo4/uKas0piLmD0z6G/KLMmTjn67j1rJReh4= X-Received: by 2002:a05:6402:19a:: with SMTP id r26mr31451311edv.230.1628535126890; Mon, 09 Aug 2021 11:52:06 -0700 (PDT) MIME-Version: 1.0 References: <20210716232916.3572966-1-l.stach@pengutronix.de> <20210721204703.1424034-1-l.stach@pengutronix.de> <818b52fe-8fa6-b47a-6dde-783ac378c603@kontron.de> <8de1cd0a-4d91-60e2-61e6-9f903bbf546b@kontron.de> <8ea33d97fb3f7abb2d80b11db28cce8c01932a09.camel@pengutronix.de> In-Reply-To: From: Adam Ford Date: Mon, 9 Aug 2021 13:51:55 -0500 Message-ID: Subject: Re: [PATCH v2 00/18] i.MX8MM GPC improvements and BLK_CTRL driver To: Frieder Schrempf Cc: Lucas Stach , Shawn Guo , Rob Herring , NXP Linux Team , Peng Fan , Marek Vasut , devicetree , arm-soc , Sascha Hauer , patchwork-lst@pengutronix.de Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org On Mon, Aug 9, 2021 at 6:50 AM Frieder Schrempf wrote: > > On 09.08.21 13:01, Lucas Stach wrote: > > Hi Frieder, > > > > Am Donnerstag, dem 05.08.2021 um 20:56 +0200 schrieb Frieder Schrempf: > >> On 05.08.21 12:18, Frieder Schrempf wrote: > >>> On 21.07.21 22:46, Lucas Stach wrote: > >>>> Hi all, > >>>> > >>>> second revision of the GPC improvements and BLK_CTRL driver to make = use > >>>> of all the power-domains on the i.MX8MM. I'm not going to repeat the= full > >>>> blurb from the v1 cover letter here, but if you are not familiar wit= h > >>>> i.MX8MM power domains, it may be worth a read. > >>>> > >>>> This 2nd revision fixes the DT bindings to be valid yaml, some small > >>>> failure path issues and most importantly the interaction with system > >>>> suspend/resume. With the previous version some of the power domains > >>>> would not come up correctly after a suspend/resume cycle. > >>>> > >>>> Updated testing git trees here, disclaimer still applies: > >>>> https://eur04.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2F= git.pengutronix.de%2Fcgit%2Flst%2Flinux%2Flog%2F%3Fh%3Dimx8m-power-domains&= amp;data=3D04%7C01%7Cfrieder.schrempf%40kontron.de%7C189884f9332e40cd566a08= d95b250a82%7C8c9d3c973fd941c8a2b1646f3942daf1%7C0%7C0%7C637641036912506485%= 7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWw= iLCJXVCI6Mn0%3D%7C1000&sdata=3DOlymcyF9VOt6nsb2E%2BpFLTBnmlpOIOxwzdBbgg= Pu%2FHo%3D&reserved=3D0 > >>>> https://eur04.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2F= git.pengutronix.de%2Fcgit%2Flst%2Flinux%2Flog%2F%3Fh%3Dimx8m-power-domains-= testing&data=3D04%7C01%7Cfrieder.schrempf%40kontron.de%7C189884f9332e40= cd566a08d95b250a82%7C8c9d3c973fd941c8a2b1646f3942daf1%7C0%7C0%7C63764103691= 2506485%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI= 6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3DXSHl3JDKPFX%2FifXK5fcMQFOXbQXuHO= JaNnJ3%2BtrMErk%3D&reserved=3D0 > >>> > >>> I finally did some tests on my side using USB, GPU and DSI (no PCIe, = VPU, CSI so far) and the results are promising. Thanks for the effort! > >>> > >>> I will try to run some more automated suspend/resume and reboot test = cycles over the weekend and report the results here afterwards. > >>> > >> > >> Unfortunately I got some results sooner than I had hoped. I set up a s= imple loop to suspend/resume every few seconds and on the first run it took= around 2-3 hours for the device to lock up on resume. On the second run it= took less than half an hour. I had glmark2-es2-drm running in the backgrou= nd, but it looks like it crashed at some point before the lockup occurred. > >> > >> Of course this could also be unrelated and caused by some peripheral d= river or something but the first suspicion is definitely the power domains. > >> > >> If you have any suggestions for which debug options to enable or where= to add some printks, please let me know. If I do another run I would like = to make sure that the resulting logs are helpful for debugging. > >> > >> And I would appreciate if someone else could try to reproduce this pro= blem on his/her side. I use this simple script for testing: > >> > >> #!/bin/sh > >> > >> glmark2-es2-drm & > >> > >> while true; > >> do > >> echo +10 > /sys/class/rtc/rtc0/wakealarm > >> echo mem > /sys/power/state > >> sleep 5 > >> done; > > > > Hm, that's unfortunate. > > > > I'm back from a two week vacation, but it looks like I won't have much > > time available to look into this issue soon. It would be very helpful > > if you could try to pinpoint the hang a bit more. If you can reproduce > > the hang with no_console_suspend you might be able to extract a bit > > more info in which stage the hang happens (suspend, resume, TF-A, etc.) > > If the hang is in the kernel you might be able to add some prints to > > the suspend/resume paths to be able to track down the exact point of > > the hang. > > > > I'm happy to look into the issue once it's better known where to look, > > but I fear that I won't have time to do the above investigation myself > > short term. Frieder, is this something you could help with over the > > next few days? > > I will see if I can find some time to track down the issue at least a lit= tle bit more. But I imagine it could get quite tedious if it takes up to se= veral hours to reproduce the issue and I don't have much time to spare. > > @Peng, @Adam and everyone else: Any chance you could setup a similar test= and try to reproduce this? right now i am on medical leave due to a broken wrist, and i wont be able to help until it heals. sorry adam > > On the other hand reboot cycle testing didn't show any lockup problems ov= er more than 24 hours, so it seems like the issue is limited to resume. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.7 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0173C4338F for ; Mon, 9 Aug 2021 18:54:47 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 88DB060EE7 for ; Mon, 9 Aug 2021 18:54:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 88DB060EE7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8eX9CjPuYZ21uWxVghbvwoRB/lUialsVHAzcBeZFwR0=; b=xlrVasF+jSfD2/ FEN+Fo5gVYn78Q8BNxbMFVnM6ND0l8fC1SdfStuFbi/3GGG92axzycbK9d0B/Hw30LLsSBMrYL+GR iWJnlrd5nClyVRU8R5HGLkfO66wQug8Hwpc2+ey0Er/XrKjbyKVKmShSiLb1sx5XQijAguuotback Gwyxo2sygcMioiHMRavLgLWrmsSBtA3GNbKU2zuNIZ4xGBkuQD1flTVIBembUzzqu2ZpKxasGdTCE KDGhCZEHaBicLZmNJHJFnvt5d9p39JyeTEHi/1JMrICgWu2W0WAzxKtEsnrQu8na45iIkAb803Qh2 eFQT6rChwDAXWCzfJkcQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mDANo-001lcg-9T; Mon, 09 Aug 2021 18:52:20 +0000 Received: from mail-ed1-x52e.google.com ([2a00:1450:4864:20::52e]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mDANd-001lam-II for linux-arm-kernel@lists.infradead.org; Mon, 09 Aug 2021 18:52:13 +0000 Received: by mail-ed1-x52e.google.com with SMTP id y7so26132625eda.5 for ; Mon, 09 Aug 2021 11:52:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=XsY1T+ZcNexFoxRjxiTUapeVjeXQ1jY55lc32SOmy1E=; b=DdAAN3VumBBn6g2xznpLrHsooL1sZzeg6QScu6NH4CXFzv2eaOE9K+sBwjGcFuFzMh vt8N7ihEmfYt2LKBeaEcXBKDKfjKwuoTpqndZPfxTs60klkoMq7vtpOnYxRO9LFOjOzW JyUBBhxtnlIDBzmUASeAOp4lxS5TjX1oqNgROGwWLp9VmpvKXH+dQT/gXVCf8HpCd8GJ 7VU5e++NkxBX8NvOJACyTyyTFSsjJKF8aW19tQkX06DNXn13UkSVUg1BEnKRO1w1OD2D 5W5twEYjTZwmIcvjx8ayrM1Slj7I+utfJX/UXKTTdU/S9eIL5PcZMDVFgzXVzd27UkOl 289Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=XsY1T+ZcNexFoxRjxiTUapeVjeXQ1jY55lc32SOmy1E=; b=hezMOAEx2VKZoZIqmMzVprHDiAwSCLUOT42P3Ew2ayvW3SfrlRDwAFsKEyKitltibf Aellpk84TddF4U6y6IrrwWxKTs8MbiJec9JgFXr7dwUge9h3Np6b+JdeTPzs2gN6RV3x 4DHTPtAfN7jWm+KVY5T5GQKfGRbL0d05g4I5WbIf+ZDN7Q11cKPkOuV5B7LyuhkaV7Ch kipV1XuOPI40o7mKotfHqW2HuImaoK7G/lw2lYXyDE87t+4eMZNN3iMCHOQPfklCHo/D iWDnx5LN3f0T5v4tHdzNz/NraEDufQF8zrRSK4HFEm3UsXOz3VEv5UVABkiK+dSVTZCl JF+Q== X-Gm-Message-State: AOAM5312ghYDxeCqR6xNjU2EL7rQaCcWSTspN+wAVuNSxMcfrGgFC3Kt 3nv5LTmjJ11A0tkf6zgUPrqNVQ6AcEncu6gvAuI= X-Google-Smtp-Source: ABdhPJx2zr1GH5VAhSUfe42K2jv5XXNr1qRQm1tv7/sKSCER42eOHllUo4/uKas0piLmD0z6G/KLMmTjn67j1rJReh4= X-Received: by 2002:a05:6402:19a:: with SMTP id r26mr31451311edv.230.1628535126890; Mon, 09 Aug 2021 11:52:06 -0700 (PDT) MIME-Version: 1.0 References: <20210716232916.3572966-1-l.stach@pengutronix.de> <20210721204703.1424034-1-l.stach@pengutronix.de> <818b52fe-8fa6-b47a-6dde-783ac378c603@kontron.de> <8de1cd0a-4d91-60e2-61e6-9f903bbf546b@kontron.de> <8ea33d97fb3f7abb2d80b11db28cce8c01932a09.camel@pengutronix.de> In-Reply-To: From: Adam Ford Date: Mon, 9 Aug 2021 13:51:55 -0500 Message-ID: Subject: Re: [PATCH v2 00/18] i.MX8MM GPC improvements and BLK_CTRL driver To: Frieder Schrempf Cc: Lucas Stach , Shawn Guo , Rob Herring , NXP Linux Team , Peng Fan , Marek Vasut , devicetree , arm-soc , Sascha Hauer , patchwork-lst@pengutronix.de X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210809_115209_699944_4DB41B6C X-CRM114-Status: GOOD ( 42.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Aug 9, 2021 at 6:50 AM Frieder Schrempf wrote: > > On 09.08.21 13:01, Lucas Stach wrote: > > Hi Frieder, > > > > Am Donnerstag, dem 05.08.2021 um 20:56 +0200 schrieb Frieder Schrempf: > >> On 05.08.21 12:18, Frieder Schrempf wrote: > >>> On 21.07.21 22:46, Lucas Stach wrote: > >>>> Hi all, > >>>> > >>>> second revision of the GPC improvements and BLK_CTRL driver to make use > >>>> of all the power-domains on the i.MX8MM. I'm not going to repeat the full > >>>> blurb from the v1 cover letter here, but if you are not familiar with > >>>> i.MX8MM power domains, it may be worth a read. > >>>> > >>>> This 2nd revision fixes the DT bindings to be valid yaml, some small > >>>> failure path issues and most importantly the interaction with system > >>>> suspend/resume. With the previous version some of the power domains > >>>> would not come up correctly after a suspend/resume cycle. > >>>> > >>>> Updated testing git trees here, disclaimer still applies: > >>>> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.pengutronix.de%2Fcgit%2Flst%2Flinux%2Flog%2F%3Fh%3Dimx8m-power-domains&data=04%7C01%7Cfrieder.schrempf%40kontron.de%7C189884f9332e40cd566a08d95b250a82%7C8c9d3c973fd941c8a2b1646f3942daf1%7C0%7C0%7C637641036912506485%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=OlymcyF9VOt6nsb2E%2BpFLTBnmlpOIOxwzdBbggPu%2FHo%3D&reserved=0 > >>>> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.pengutronix.de%2Fcgit%2Flst%2Flinux%2Flog%2F%3Fh%3Dimx8m-power-domains-testing&data=04%7C01%7Cfrieder.schrempf%40kontron.de%7C189884f9332e40cd566a08d95b250a82%7C8c9d3c973fd941c8a2b1646f3942daf1%7C0%7C0%7C637641036912506485%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XSHl3JDKPFX%2FifXK5fcMQFOXbQXuHOJaNnJ3%2BtrMErk%3D&reserved=0 > >>> > >>> I finally did some tests on my side using USB, GPU and DSI (no PCIe, VPU, CSI so far) and the results are promising. Thanks for the effort! > >>> > >>> I will try to run some more automated suspend/resume and reboot test cycles over the weekend and report the results here afterwards. > >>> > >> > >> Unfortunately I got some results sooner than I had hoped. I set up a simple loop to suspend/resume every few seconds and on the first run it took around 2-3 hours for the device to lock up on resume. On the second run it took less than half an hour. I had glmark2-es2-drm running in the background, but it looks like it crashed at some point before the lockup occurred. > >> > >> Of course this could also be unrelated and caused by some peripheral driver or something but the first suspicion is definitely the power domains. > >> > >> If you have any suggestions for which debug options to enable or where to add some printks, please let me know. If I do another run I would like to make sure that the resulting logs are helpful for debugging. > >> > >> And I would appreciate if someone else could try to reproduce this problem on his/her side. I use this simple script for testing: > >> > >> #!/bin/sh > >> > >> glmark2-es2-drm & > >> > >> while true; > >> do > >> echo +10 > /sys/class/rtc/rtc0/wakealarm > >> echo mem > /sys/power/state > >> sleep 5 > >> done; > > > > Hm, that's unfortunate. > > > > I'm back from a two week vacation, but it looks like I won't have much > > time available to look into this issue soon. It would be very helpful > > if you could try to pinpoint the hang a bit more. If you can reproduce > > the hang with no_console_suspend you might be able to extract a bit > > more info in which stage the hang happens (suspend, resume, TF-A, etc.) > > If the hang is in the kernel you might be able to add some prints to > > the suspend/resume paths to be able to track down the exact point of > > the hang. > > > > I'm happy to look into the issue once it's better known where to look, > > but I fear that I won't have time to do the above investigation myself > > short term. Frieder, is this something you could help with over the > > next few days? > > I will see if I can find some time to track down the issue at least a little bit more. But I imagine it could get quite tedious if it takes up to several hours to reproduce the issue and I don't have much time to spare. > > @Peng, @Adam and everyone else: Any chance you could setup a similar test and try to reproduce this? right now i am on medical leave due to a broken wrist, and i wont be able to help until it heals. sorry adam > > On the other hand reboot cycle testing didn't show any lockup problems over more than 24 hours, so it seems like the issue is limited to resume. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel