From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B06A7C2D0C0 for ; Mon, 23 Dec 2019 16:26:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8D18820663 for ; Mon, 23 Dec 2019 16:26:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726911AbfLWQ0r (ORCPT ); Mon, 23 Dec 2019 11:26:47 -0500 Received: from outgoing-auth-1.mit.edu ([18.9.28.11]:56982 "EHLO outgoing.mit.edu" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726744AbfLWQ0r (ORCPT ); Mon, 23 Dec 2019 11:26:47 -0500 Received: from callcc.thunk.org (guestnat-104-133-0-111.corp.google.com [104.133.0.111] (may be forged)) (authenticated bits=0) (User authenticated as tytso@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id xBNGQJ8k010341 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Dec 2019 11:26:20 -0500 Received: by callcc.thunk.org (Postfix, from userid 15806) id 4BCF9420822; Mon, 23 Dec 2019 11:26:19 -0500 (EST) Date: Mon, 23 Dec 2019 11:26:19 -0500 From: "Theodore Y. Ts'o" To: Ming Lei Cc: Andrea Vai , "Schmid, Carsten" , Finn Thain , Damien Le Moal , Alan Stern , Jens Axboe , Johannes Thumshirn , USB list , SCSI development list , Himanshu Madhani , Hannes Reinecke , Omar Sandoval , "Martin K. Petersen" , Greg KH , Hans Holmberg , Kernel development list , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Message-ID: <20191223162619.GA3282@mit.edu> References: <8196b014b1a4d91169bf3b0d68905109aeaf2191.camel@unipv.it> <20191210080550.GA5699@ming.t460p> <20191211024137.GB61323@mit.edu> <20191211040058.GC6864@ming.t460p> <20191211160745.GA129186@mit.edu> <20191211213316.GA14983@ming.t460p> <20191218094830.GB30602@ming.t460p> <20191223130828.GA25948@ming.t460p> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20191223130828.GA25948@ming.t460p> User-Agent: Mutt/1.12.2 (2019-09-21) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Dec 23, 2019 at 09:08:28PM +0800, Ming Lei wrote: > > From the above trace: > > b'blk_mq_sched_request_inserted' > b'blk_mq_sched_request_inserted' > b'dd_insert_requests' > b'blk_mq_sched_insert_requests' > b'blk_mq_flush_plug_list' > b'blk_flush_plug_list' > b'io_schedule_prepare' > b'io_schedule' > b'rq_qos_wait' > b'wbt_wait' > b'__rq_qos_throttle' > b'blk_mq_make_request' > b'generic_make_request' > b'submit_bio' > b'ext4_io_submit' > b'ext4_writepages' > b'do_writepages' > b'__filemap_fdatawrite_range' > b'ext4_release_file' > b'__fput' > b'task_work_run' > b'exit_to_usermode_loop' > b'do_syscall_64' > b'entry_SYSCALL_64_after_hwframe' > b'cp' [19863] > 4400 > > So this write is clearly from 'cp' process, and it should be one > ext4 fs issue. We need a system call trace of the cp process, to understand what system call is resulting in fput, (eg., I assume it's close(2) but let's be sure), and often it's calling that system call. What cp process is it? Is it from shellutils? Is it from busybox? - Ted