From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Vitaly Kuznetsov To: Ming Lei Cc: linux-block , linux-mm , Linux FS Devel , "open list\:XFS FILESYSTEM" , Dave Chinner , Linux Kernel Mailing List , Christoph Hellwig , Jens Axboe , Ming Lei Subject: Re: block: DMA alignment of IO buffer allocated from slab References: Date: Wed, 19 Sep 2018 11:41:07 +0200 In-Reply-To: (Ming Lei's message of "Wed, 19 Sep 2018 17:15:43 +0800") Message-ID: <877ejh3jv0.fsf@vitty.brq.redhat.com> MIME-Version: 1.0 Content-Type: text/plain List-ID: Ming Lei writes: > Hi Guys, > > Some storage controllers have DMA alignment limit, which is often set via > blk_queue_dma_alignment(), such as 512-byte alignment for IO buffer. While mostly drivers use 512-byte alignment it is not a rule of thumb, 'git grep' tell me we have: ide-cd.c with 32-byte alignment ps3disk.c and rsxx/dev.c with variable alignment. What if our block configuration consists of several devices (in raid array, for example) with different requirements, e.g. one requiring 512-byte alignment and the other requiring 256? > > Block layer now only checks if this limit is respected for buffer of > pass-through request, > see blk_rq_map_user_iov(), bio_map_user_iov(). > > The userspace buffer for direct IO is checked in dio path, see > do_blockdev_direct_IO(). > IO buffer from page cache should be fine wrt. this limit too. > > However, some file systems, such as XFS, may allocate single sector IO buffer > via slab. Usually I guess kmalloc-512 should be fine to return > 512-aligned buffer. > But once KASAN or other slab debug options are enabled, looks this > isn't true any > more, kmalloc-512 may not return 512-aligned buffer. Then data corruption > can be observed because the IO buffer from fs layer doesn't respect the DMA > alignment limit any more. > > Follows several related questions: > > 1) does kmalloc-N slab guarantee to return N-byte aligned buffer? If > yes, is it a stable rule? > > 2) If it is a rule for kmalloc-N slab to return N-byte aligned buffer, > seems KASAN violates this > rule? (as I was kinda involved in debugging): the issue was observed with SLUB allocator KASAN is not to blame, everything wich requires aditional metadata space will break this, see e.g. calculate_sizes() in slub.c > > 3) If slab can't guarantee to return 512-aligned buffer, how to fix > this data corruption issue? I'm no expert in block layer but in case of complex block device configurations when bio submitter can't know all the requirements I see no other choice than bouncing. -- Vitaly From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CB53ECE562 for ; Wed, 19 Sep 2018 09:41:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 558B92098A for ; Wed, 19 Sep 2018 09:41:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 558B92098A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731204AbeISPSi (ORCPT ); Wed, 19 Sep 2018 11:18:38 -0400 Received: from mx1.redhat.com ([209.132.183.28]:58398 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727820AbeISPSi (ORCPT ); Wed, 19 Sep 2018 11:18:38 -0400 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.24]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E9DCD308ED72; Wed, 19 Sep 2018 09:41:31 +0000 (UTC) Received: from vitty.brq.redhat.com.redhat.com (unknown [10.43.2.155]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D638A30AD7B6; Wed, 19 Sep 2018 09:41:10 +0000 (UTC) From: Vitaly Kuznetsov To: Ming Lei Cc: linux-block , linux-mm , Linux FS Devel , "open list\:XFS FILESYSTEM" , Dave Chinner , Linux Kernel Mailing List , Christoph Hellwig , Jens Axboe , Ming Lei Subject: Re: block: DMA alignment of IO buffer allocated from slab References: Date: Wed, 19 Sep 2018 11:41:07 +0200 In-Reply-To: (Ming Lei's message of "Wed, 19 Sep 2018 17:15:43 +0800") Message-ID: <877ejh3jv0.fsf@vitty.brq.redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Scanned-By: MIMEDefang 2.84 on 10.5.11.24 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.44]); Wed, 19 Sep 2018 09:41:33 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Ming Lei writes: > Hi Guys, > > Some storage controllers have DMA alignment limit, which is often set via > blk_queue_dma_alignment(), such as 512-byte alignment for IO buffer. While mostly drivers use 512-byte alignment it is not a rule of thumb, 'git grep' tell me we have: ide-cd.c with 32-byte alignment ps3disk.c and rsxx/dev.c with variable alignment. What if our block configuration consists of several devices (in raid array, for example) with different requirements, e.g. one requiring 512-byte alignment and the other requiring 256? > > Block layer now only checks if this limit is respected for buffer of > pass-through request, > see blk_rq_map_user_iov(), bio_map_user_iov(). > > The userspace buffer for direct IO is checked in dio path, see > do_blockdev_direct_IO(). > IO buffer from page cache should be fine wrt. this limit too. > > However, some file systems, such as XFS, may allocate single sector IO buffer > via slab. Usually I guess kmalloc-512 should be fine to return > 512-aligned buffer. > But once KASAN or other slab debug options are enabled, looks this > isn't true any > more, kmalloc-512 may not return 512-aligned buffer. Then data corruption > can be observed because the IO buffer from fs layer doesn't respect the DMA > alignment limit any more. > > Follows several related questions: > > 1) does kmalloc-N slab guarantee to return N-byte aligned buffer? If > yes, is it a stable rule? > > 2) If it is a rule for kmalloc-N slab to return N-byte aligned buffer, > seems KASAN violates this > rule? (as I was kinda involved in debugging): the issue was observed with SLUB allocator KASAN is not to blame, everything wich requires aditional metadata space will break this, see e.g. calculate_sizes() in slub.c > > 3) If slab can't guarantee to return 512-aligned buffer, how to fix > this data corruption issue? I'm no expert in block layer but in case of complex block device configurations when bio submitter can't know all the requirements I see no other choice than bouncing. -- Vitaly From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com ([209.132.183.28]:58398 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727820AbeISPSi (ORCPT ); Wed, 19 Sep 2018 11:18:38 -0400 From: Vitaly Kuznetsov Subject: Re: block: DMA alignment of IO buffer allocated from slab References: Date: Wed, 19 Sep 2018 11:41:07 +0200 In-Reply-To: (Ming Lei's message of "Wed, 19 Sep 2018 17:15:43 +0800") Message-ID: <877ejh3jv0.fsf@vitty.brq.redhat.com> MIME-Version: 1.0 Content-Type: text/plain Sender: linux-xfs-owner@vger.kernel.org List-ID: List-Id: xfs To: Ming Lei Cc: linux-block , linux-mm , Linux FS Devel , "open list:XFS FILESYSTEM" , Dave Chinner , Linux Kernel Mailing List , Christoph Hellwig , Jens Axboe , Ming Lei Ming Lei writes: > Hi Guys, > > Some storage controllers have DMA alignment limit, which is often set via > blk_queue_dma_alignment(), such as 512-byte alignment for IO buffer. While mostly drivers use 512-byte alignment it is not a rule of thumb, 'git grep' tell me we have: ide-cd.c with 32-byte alignment ps3disk.c and rsxx/dev.c with variable alignment. What if our block configuration consists of several devices (in raid array, for example) with different requirements, e.g. one requiring 512-byte alignment and the other requiring 256? > > Block layer now only checks if this limit is respected for buffer of > pass-through request, > see blk_rq_map_user_iov(), bio_map_user_iov(). > > The userspace buffer for direct IO is checked in dio path, see > do_blockdev_direct_IO(). > IO buffer from page cache should be fine wrt. this limit too. > > However, some file systems, such as XFS, may allocate single sector IO buffer > via slab. Usually I guess kmalloc-512 should be fine to return > 512-aligned buffer. > But once KASAN or other slab debug options are enabled, looks this > isn't true any > more, kmalloc-512 may not return 512-aligned buffer. Then data corruption > can be observed because the IO buffer from fs layer doesn't respect the DMA > alignment limit any more. > > Follows several related questions: > > 1) does kmalloc-N slab guarantee to return N-byte aligned buffer? If > yes, is it a stable rule? > > 2) If it is a rule for kmalloc-N slab to return N-byte aligned buffer, > seems KASAN violates this > rule? (as I was kinda involved in debugging): the issue was observed with SLUB allocator KASAN is not to blame, everything wich requires aditional metadata space will break this, see e.g. calculate_sizes() in slub.c > > 3) If slab can't guarantee to return 512-aligned buffer, how to fix > this data corruption issue? I'm no expert in block layer but in case of complex block device configurations when bio submitter can't know all the requirements I see no other choice than bouncing. -- Vitaly