From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE1E2C433F5 for ; Sat, 12 Feb 2022 08:33:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232650AbiBLIeB (ORCPT ); Sat, 12 Feb 2022 03:34:01 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:39656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232647AbiBLIeA (ORCPT ); Sat, 12 Feb 2022 03:34:00 -0500 Received: from alexa-out-sd-02.qualcomm.com (alexa-out-sd-02.qualcomm.com [199.106.114.39]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EEE91212; Sat, 12 Feb 2022 00:33:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; i=@quicinc.com; q=dns/txt; s=qcdkim; t=1644654837; x=1676190837; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=lmiueAkY/np8phTT6QzUC4DsJH0cEhvOy4i26Lo9XNw=; b=oArrRJCylCh0mkM8AQdyMKXUbPL41MeiDtCCnIoMEeICaSMm+uZuL18+ 5mizBCAjcjd/w6nSmb/XmHPjpB26939GRVizO2+qhFcGR/RIeLQbGPEGs anMC1KMQWDf3iHBbhF9dvT0w6Qk1dTwBdK3GJaZJNa5MvkY4TMj4anKqu 0=; Received: from unknown (HELO ironmsg04-sd.qualcomm.com) ([10.53.140.144]) by alexa-out-sd-02.qualcomm.com with ESMTP; 12 Feb 2022 00:33:57 -0800 X-QCInternal: smtphost Received: from nasanex01c.na.qualcomm.com ([10.47.97.222]) by ironmsg04-sd.qualcomm.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Feb 2022 00:33:56 -0800 Received: from nalasex01a.na.qualcomm.com (10.47.209.196) by nasanex01c.na.qualcomm.com (10.47.97.222) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.15; Sat, 12 Feb 2022 00:33:56 -0800 Received: from [10.38.246.233] (10.80.80.8) by nalasex01a.na.qualcomm.com (10.47.209.196) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.922.19; Sat, 12 Feb 2022 00:33:53 -0800 Message-ID: <588fbad9-6f50-5d85-9a33-c1206f5815a6@quicinc.com> Date: Sat, 12 Feb 2022 00:33:51 -0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.5.1 Subject: Re: [PATCH] devcoredump: increase the device delete timeout to 10 mins Content-Language: en-US To: Greg KH CC: , , , , , , , , , , , , , References: <1644349472-31077-1-git-send-email-quic_abhinavk@quicinc.com> <654d620b-9e14-c47f-b48c-762dc0bd32a1@quicinc.com> From: Abhinav Kumar In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.80.80.8] X-ClientProxiedBy: nasanex01b.na.qualcomm.com (10.46.141.250) To nalasex01a.na.qualcomm.com (10.47.209.196) Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org Hi Greg On 2/12/2022 12:29 AM, Greg KH wrote: > On Fri, Feb 11, 2022 at 11:52:41PM -0800, Abhinav Kumar wrote: >> Hi Greg >> >> On 2/11/2022 11:04 PM, Greg KH wrote: >>> On Fri, Feb 11, 2022 at 10:59:39AM -0800, Abhinav Kumar wrote: >>>> Hi Greg >>>> >>>> Thanks for the response. >>>> >>>> On 2/11/2022 3:09 AM, Greg KH wrote: >>>>> On Tue, Feb 08, 2022 at 11:44:32AM -0800, Abhinav Kumar wrote: >>>>>> There are cases where depending on the size of the devcoredump and the speed >>>>>> at which the usermode reads the dump, it can take longer than the current 5 mins >>>>>> timeout. >>>>>> >>>>>> This can lead to incomplete dumps as the device is deleted once the timeout expires. >>>>>> >>>>>> One example is below where it took 6 mins for the devcoredump to be completely read. >>>>>> >>>>>> 04:22:24.668 23916 23994 I HWDeviceDRM::DumpDebugData: Opening /sys/class/devcoredump/devcd6/data >>>>>> 04:28:35.377 23916 23994 W HWDeviceDRM::DumpDebugData: Freeing devcoredump node >>>>> >>>>> What makes this so slow? Reading from the kernel shouldn't be the >>>>> limit, is it where the data is being sent to? >>>> >>>> We are still checking this. We are seeing better read times when we bump up >>>> the thread priority of the thread which was reading this. >>> >>> Where is the thread sending the data to? >> >> The thread is writing the data to a file in local storage. From our >> profiling, the read is the one taking the time not the write. > > The read is coming directly from memory, there should not be any > slowdown at all here. How can that be the delay? Have a trace > somewhere? > > thanks, > > greg k-h Yes, like I mentioned in my previous comment we are still checking why its taking so long. We will update with our findings if we have any. Alright, we will try to capture trace to share and will update this thread if we find something as well. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 60A21C433EF for ; Sat, 12 Feb 2022 08:34:00 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7C7B110E137; Sat, 12 Feb 2022 08:33:59 +0000 (UTC) Received: from alexa-out-sd-01.qualcomm.com (alexa-out-sd-01.qualcomm.com [199.106.114.38]) by gabe.freedesktop.org (Postfix) with ESMTPS id 30DA410E137; Sat, 12 Feb 2022 08:33:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; i=@quicinc.com; q=dns/txt; s=qcdkim; t=1644654838; x=1676190838; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=lmiueAkY/np8phTT6QzUC4DsJH0cEhvOy4i26Lo9XNw=; b=SCFfgdJxbKIC1p69YNSgomEcVV0HTgo65TyWSJFLptlyZNAVG3cZ9E19 5TN+v42P1+5z7SJgRzI6MNw9tAxVEjdfPWqGucJjM2XpaNU9TclzTna5F ye/P64ZHJP1iTLzFSSakPnIT/ia5O9CA5xOSVVr6ETQsg0frPuM0D48dE 4=; Received: from unknown (HELO ironmsg04-sd.qualcomm.com) ([10.53.140.144]) by alexa-out-sd-01.qualcomm.com with ESMTP; 12 Feb 2022 00:33:57 -0800 X-QCInternal: smtphost Received: from nasanex01c.na.qualcomm.com ([10.47.97.222]) by ironmsg04-sd.qualcomm.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Feb 2022 00:33:56 -0800 Received: from nalasex01a.na.qualcomm.com (10.47.209.196) by nasanex01c.na.qualcomm.com (10.47.97.222) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.15; Sat, 12 Feb 2022 00:33:56 -0800 Received: from [10.38.246.233] (10.80.80.8) by nalasex01a.na.qualcomm.com (10.47.209.196) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.922.19; Sat, 12 Feb 2022 00:33:53 -0800 Message-ID: <588fbad9-6f50-5d85-9a33-c1206f5815a6@quicinc.com> Date: Sat, 12 Feb 2022 00:33:51 -0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.5.1 Subject: Re: [PATCH] devcoredump: increase the device delete timeout to 10 mins Content-Language: en-US To: Greg KH References: <1644349472-31077-1-git-send-email-quic_abhinavk@quicinc.com> <654d620b-9e14-c47f-b48c-762dc0bd32a1@quicinc.com> From: Abhinav Kumar In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.80.80.8] X-ClientProxiedBy: nasanex01b.na.qualcomm.com (10.46.141.250) To nalasex01a.na.qualcomm.com (10.47.209.196) X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: rafael@kernel.org, linux-arm-msm@vger.kernel.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, swboyd@chromium.org, khsieh@codeaurora.org, nganji@codeaurora.org, seanpaul@chromium.org, dmitry.baryshkov@linaro.org, johannes@sipsolutions.net, aravindh@codeaurora.org, freedreno@lists.freedesktop.org Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Hi Greg On 2/12/2022 12:29 AM, Greg KH wrote: > On Fri, Feb 11, 2022 at 11:52:41PM -0800, Abhinav Kumar wrote: >> Hi Greg >> >> On 2/11/2022 11:04 PM, Greg KH wrote: >>> On Fri, Feb 11, 2022 at 10:59:39AM -0800, Abhinav Kumar wrote: >>>> Hi Greg >>>> >>>> Thanks for the response. >>>> >>>> On 2/11/2022 3:09 AM, Greg KH wrote: >>>>> On Tue, Feb 08, 2022 at 11:44:32AM -0800, Abhinav Kumar wrote: >>>>>> There are cases where depending on the size of the devcoredump and the speed >>>>>> at which the usermode reads the dump, it can take longer than the current 5 mins >>>>>> timeout. >>>>>> >>>>>> This can lead to incomplete dumps as the device is deleted once the timeout expires. >>>>>> >>>>>> One example is below where it took 6 mins for the devcoredump to be completely read. >>>>>> >>>>>> 04:22:24.668 23916 23994 I HWDeviceDRM::DumpDebugData: Opening /sys/class/devcoredump/devcd6/data >>>>>> 04:28:35.377 23916 23994 W HWDeviceDRM::DumpDebugData: Freeing devcoredump node >>>>> >>>>> What makes this so slow? Reading from the kernel shouldn't be the >>>>> limit, is it where the data is being sent to? >>>> >>>> We are still checking this. We are seeing better read times when we bump up >>>> the thread priority of the thread which was reading this. >>> >>> Where is the thread sending the data to? >> >> The thread is writing the data to a file in local storage. From our >> profiling, the read is the one taking the time not the write. > > The read is coming directly from memory, there should not be any > slowdown at all here. How can that be the delay? Have a trace > somewhere? > > thanks, > > greg k-h Yes, like I mentioned in my previous comment we are still checking why its taking so long. We will update with our findings if we have any. Alright, we will try to capture trace to share and will update this thread if we find something as well.