From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FA21C2D0D2 for ; Mon, 23 Dec 2019 19:53:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F3321206B7 for ; Mon, 23 Dec 2019 19:53:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726903AbfLWTx2 (ORCPT ); Mon, 23 Dec 2019 14:53:28 -0500 Received: from outgoing-auth-1.mit.edu ([18.9.28.11]:59299 "EHLO outgoing.mit.edu" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726766AbfLWTx2 (ORCPT ); Mon, 23 Dec 2019 14:53:28 -0500 Received: from callcc.thunk.org (guestnat-104-133-0-111.corp.google.com [104.133.0.111] (may be forged)) (authenticated bits=0) (User authenticated as tytso@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id xBNJr19e001916 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Dec 2019 14:53:02 -0500 Received: by callcc.thunk.org (Postfix, from userid 15806) id 1C82A420822; Mon, 23 Dec 2019 14:53:01 -0500 (EST) Date: Mon, 23 Dec 2019 14:53:01 -0500 From: "Theodore Y. Ts'o" To: Andrea Vai Cc: Ming Lei , "Schmid, Carsten" , Finn Thain , Damien Le Moal , Alan Stern , Jens Axboe , Johannes Thumshirn , USB list , SCSI development list , Himanshu Madhani , Hannes Reinecke , Omar Sandoval , "Martin K. Petersen" , Greg KH , Hans Holmberg , Kernel development list , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Message-ID: <20191223195301.GC3282@mit.edu> References: <20191211160745.GA129186@mit.edu> <20191211213316.GA14983@ming.t460p> <20191218094830.GB30602@ming.t460p> <20191223130828.GA25948@ming.t460p> <20191223162619.GA3282@mit.edu> <4c85fd3f2ec58694cc1ff7ab5c88d6e11ab6efec.camel@unipv.it> <20191223172257.GB3282@mit.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.2 (2019-09-21) Sender: linux-usb-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-usb@vger.kernel.org On Mon, Dec 23, 2019 at 07:45:57PM +0100, Andrea Vai wrote: > basically, it's: > > mount UUID=$uuid /mnt/pendrive > SECONDS=0 > cp $testfile /mnt/pendrive > umount /mnt/pendrive > tempo=$SECONDS > > and it copies one file only. Anyway, you can find the whole script > attached. OK, so whether we are doing the writeback at the end of cp, or when you do the umount, it's probably not going to make any difference. We can get rid of the stack trace in question by changing the script to be basically: mount UUID=$uuid /mnt/pendrive SECONDS=0 rm -f /mnt/pendrive/$testfile cp $testfile /mnt/pendrive umount /mnt/pendrive tempo=$SECONDS I predict if you do that, you'll see that all of the time is spent in the umount, when we are trying to write back the file. I really don't think then this is a file system problem at all. It's just that USB I/O is slow, for whatever reason. We'll see a stack trace in the writeback code waiting for the I/O to be completed, but that doesn't mean that the root cause is in the writeback code or in the file system which is triggering the writeback. I suspect the next step is use a blktrace, to see what kind of I/O is being sent to the USB drive, and how long it takes for the I/O to complete. You might also try to capture the output of "iostat -x 1" while the script is running, and see what the difference might be between a kernel version that has the problem and one that doesn't, and see if that gives us a clue. > > And then send me > btw, please tell me if "me" means only you or I cc: all the > recipients, as usual Well, I don't think we know what the root cause is. Ming is focusing on that stack trace, but I think it's a red herring..... And if it's not a file system problem, then other people will be best suited to debug the issue. - Ted