From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1753358AbbDAH0Q (ORCPT <rfc822;w@1wt.eu>);
	Wed, 1 Apr 2015 03:26:16 -0400
Received: from verein.lst.de ([213.95.11.211]:58501 "EHLO newverein.lst.de"
	rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
	id S1751474AbbDAH0L (ORCPT <rfc822;linux-kernel@vger.kernel.org>);
	Wed, 1 Apr 2015 03:26:11 -0400
Date: Wed, 1 Apr 2015 09:26:08 +0200
From: Christoph Hellwig <hch@lst.de>
To: "Elliott, Robert (Server Storage)" <Elliott@hp.com>
Cc: Christoph Hellwig <hch@lst.de>,
        "linux-nvdimm@ml01.01.org" <linux-nvdimm@ml01.01.org>,
        "linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "x86@kernel.org" <x86@kernel.org>,
        "ross.zwisler@linux.intel.com" <ross.zwisler@linux.intel.com>,
        "axboe@kernel.dk" <axboe@kernel.dk>,
        "boaz@plexistor.com" <boaz@plexistor.com>,
        "Kani, Toshimitsu" <toshi.kani@hp.com>
Subject: Re: another pmem variant V2
Message-ID: <20150401072608.GA24312@lst.de>
References: <1427358764-6126-1-git-send-email-hch@lst.de> <94D0CD8314A33A4D9D801C0FE68B40295A853392@G9W0745.americas.hpqcorp.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <94D0CD8314A33A4D9D801C0FE68B40295A853392@G9W0745.americas.hpqcorp.net>
User-Agent: Mutt/1.5.17 (2007-11-01)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Mar 31, 2015 at 10:11:29PM +0000, Elliott, Robert (Server Storage) wrote:
> I used fio to test 4 KiB random read and write IOPS 
> on a 2-socket x86 DDR4 system.  With various cache attributes:
> 
> attr	read		write		notes
> ----	----		-----		-----
> UC	37 K		21 K		ioremap_nocache
> WB	3.6 M		2.5 M		ioremap
> WC	764 K		3.7 M		ioremap_wc
> WT	<not tested yet>		ioremap_wt
> 
> So, although UC and WT are the only modes certain to be safe,
> the V1 default of UC provides abysmal performance - worse than
> a consumer-class SATA SSD.

It doesn't look quite as bad on my setup, but performance is fairly
bad here as well.

> A solution for x86 is to use the MOVNTI instruction in WB
> mode. This non-temporal hint uses a buffer like the write
> combining buffer, not filling the cache and not stopping
> everything in the CPU.  The kernel function __copy_from_user() 
> uses that instruction (with SFENCE at the end) - see
> arch/x86/lib/copy_user_nocache_64.S.
> 
> If I made the change from memcpy() to __copy_from_user()
> correctly, that results in:
> 
> attr		read		write		notes
> ----		----		-----		-----
> WB w/NTI	2.4 M		2.6 M		__copy_from_user()
> WC w/NTI	3.2 M		2.1 M		__copy_from_user()

That looks a lot better.  It doesn't help us with a pmem device
mapped directly into userspace using mmap with the DAX infrastructure,
though.

Note when we want to move to non-temporal copies we'll need to add
a new prototype, as __copy_from_user isn't guaranteed to use these,
and it is defined to only work on user addresses.  That doesn't matter
on x86 but would blow up on say sparc or s390.