RE: REPORT: FIO random read performance degradation without "norandommap" option

From: "Elliott, Robert (Server Storage)" <Elliott@hp.com>
To: "Kudryavtsev, Andrey O" <andrey.o.kudryavtsev@intel.com>,
	"fio@vger.kernel.org" <fio@vger.kernel.org>
Subject: RE: REPORT: FIO random read performance degradation without "norandommap" option
Date: Fri, 9 Jan 2015 22:49:08 +0000	[thread overview]
Message-ID: <94D0CD8314A33A4D9D801C0FE68B40295A72F566@G9W0745.americas.hpqcorp.net> (raw)
In-Reply-To: <D0D596A1.FF42%andrey.o.kudryavtsev@intel.com>

> -----Original Message-----
> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On
> Behalf Of Kudryavtsev, Andrey O
> Sent: Friday, 09 January, 2015 4:42 PM
> To: fio@vger.kernel.org
> Subject: REPORT: FIO random read performance degradation without
> "norandommap" option
> 
> Colleagues,
> I executed 2-hour runs of 4KRR to understand performance changes across
> the time on the specific very fast NVMe SSD with 1.6TB capacity.
> I noticed the side effect of "norandommap" parameter performing full span
> test on the block device.
> Here is the example of the result with random map (I.e. without
> "norandommap" option) within 120 minutes windows.
> [cid:E6872B64-35D1-4447-A0CF-32E6411D9BDB]
> (IOPS in blue)
> 
> As soon as I enabled "norandommap" option the curve has changed into the
> straight line as expected.

It takes resources to maintain the random map table.  I always run with
norandommap unless using verify, which has to remember which ones have
been accessed.

Here's a description I gave someone a while back:
With a huge device (e.g., 5.8 TB from RAID-0 made from 16 SSDs),
if you do not use "norandommap", fio allocates a bitmap for all
the disk blocks to keep track of where it has read or written.
It uses this to avoid accessing the same blocks until all
the blocks have been accessed, and to know which blocks it
needs to verify if verify=<something> is enabled.

For 5.8 TB, that is 1562714136 = 1.5 GB.  Not many of those
huge allocations work, so it 
* hangs the system for a while
* generates estimates like
	[eta 1158050440d:06h:50m:22s]
* and eventually reports
	smalloc: failed adding pool
	fio: failed allocating random map. If running a large number of jobs, try the 'norandommap' option or set 'softrandommap'. Or give a larger --alloc-size to fio.

fio continues to run after that; I think it verifies only
the devices for which the allocation worked and ignores
the rest.

---
Rob Elliott    HP Server Storage