Unaligned AIO/DIO

Unaligned AIO/DIO

[Troels Arvin]

Hello,

Running fio 1.55 on RHEL 6.1 x86_64 with a multipath FC storage backend 
on an EXT4 file system.

Some minutes after starting fio, I see the following in syslog:
kernel: EXT4-fs (dm-28): Unaligned AIO/DIO on inode 4063238 by fio; 
performance will be poor.

Should I be worried? Is there something wrong with my storage layout?
Or am I using fio in strange way?

My job file:

[changed-from-iometer-file-access-server]
bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10
rw=randrw
rwmixread=70
direct=1
size=10g
ioengine=libaio
iodepth=256
write_bw_log
write_lat_log
numjobs=6

-- 
Regards,
Troels Arvin <troels@arvin.dk>
http://troels.arvin.dk/

Re: Unaligned AIO/DIO

[Jens Axboe]

On 2011-05-31 20:22, Troels Arvin wrote:
> Hello,
> 
> Running fio 1.55 on RHEL 6.1 x86_64 with a multipath FC storage backend on an EXT4 file system.
> 
> Some minutes after starting fio, I see the following in syslog:
> kernel: EXT4-fs (dm-28): Unaligned AIO/DIO on inode 4063238 by fio; performance will be poor.
> 
> Should I be worried? Is there something wrong with my storage layout?
> Or am I using fio in strange way?
> 
> My job file:
> 
> [changed-from-iometer-file-access-server]
> bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10
> rw=randrw
> rwmixread=70
> direct=1
> size=10g
> ioengine=libaio
> iodepth=256
> write_bw_log
> write_lat_log
> numjobs=6

Do you have the RHEL6.1 kernel sources handy? That message is not in the
upstream kernel. Unaligned could be memory, or it could be offset/size.
In general O_DIRECT does not work for unaligned IO. Perhaps it's
referring to writes that aren't stripe aligned, but that seems fairly
expected and something dm should handle.

So to render an opinion on this, I'd need to see a copy of the file that
prints that message :-)

-- 
Jens Axboe

Re: Unaligned AIO/DIO

[Troels Arvin]

Hello,

Jens Axboe wrote:
> Do you have the RHEL6.1 kernel sources handy?

I just grabbed the particular kernel package'es source-package, 
installed it into ~/rpmbuild and ran "rpmbuild -bp" on its spec-file 
which should result in all RHs patches to be applied.

 From the resulting code in ~/rpmbuild/BUILD/..., I grabbed file.c and 
placed it here:
http://troels.arvin.dk/misc/fio/unaligned/

Lines close to the error message (probably wrapped by my mailer):

	/* Unaligned direct AIO must be serialized; see comment above */
	if (unaligned_aio) {
		static unsigned long unaligned_warn_time;

		/* Warn about this once per day */
		if (printk_timed_ratelimit(&unaligned_warn_time, 60*60*24*HZ))
			ext4_msg(inode->i_sb, KERN_WARNING,
				 "Unaligned AIO/DIO on inode %ld by %s; "
				 "performance will be poor.",
				 inode->i_ino, current->comm);

		mutex_lock(&EXT4_I(inode)->i_aio_mutex);
		ext4_aiodio_wait(inode);
  	}

-- 
Regards,
Troels Arvin <troels@arvin.dk>
http://troels.arvin.dk/

Re: Unaligned AIO/DIO

[Jens Axboe]

On 2011-06-01 14:51, Troels Arvin wrote:
> Hello,
> 
> Jens Axboe wrote:
>> Do you have the RHEL6.1 kernel sources handy?
> 
> I just grabbed the particular kernel package'es source-package, installed it into ~/rpmbuild and ran "rpmbuild -bp" on its spec-file which should result in all RHs patches to be applied.
> 
> From the resulting code in ~/rpmbuild/BUILD/..., I grabbed file.c and placed it here:
> http://troels.arvin.dk/misc/fio/unaligned/
> 
> Lines close to the error message (probably wrapped by my mailer):
> 
>     /* Unaligned direct AIO must be serialized; see comment above */
>     if (unaligned_aio) {
>         static unsigned long unaligned_warn_time;
> 
>         /* Warn about this once per day */
>         if (printk_timed_ratelimit(&unaligned_warn_time, 60*60*24*HZ))
>             ext4_msg(inode->i_sb, KERN_WARNING,
>                  "Unaligned AIO/DIO on inode %ld by %s; "
>                  "performance will be poor.",
>                  inode->i_ino, current->comm);
> 
>         mutex_lock(&EXT4_I(inode)->i_aio_mutex);
>         ext4_aiodio_wait(inode);
>      }

Ah, so this is indeed the < 4KB O_DIRECT writes. The iometer like test
case you are using is meant to be used on a raw block device, not on a
file system. For a file system, doing less than the block size of IO
will incur RMW cycles and thus be very slow compared to full block size
(or more) O_DIRECT writes.

If you want to run it on the fs, either change the block size mappings
to avoid anything less than 4K. Or change the fs to be formatted with
512b blocks.

Or just run the test on the raw device :-)

-- 
Jens Axboe

Re: Unaligned AIO/DIO

[Troels Arvin]

Hello,

Jens Axboe wrote:
> The iometer like test
> case you are using is meant to be used on a raw block device, not on a
> file system.

OK, but how does one use fio with a raw device? Can you give an example 
of a job specification file for working with a raw block device?

And can that be done in a way which simulates parallel I/O (several 
processes/threads working simultaneously)?

-- 
Regards,
Troels Arvin <troels@arvin.dk>
http://troels.arvin.dk/

Re: Unaligned AIO/DIO

[Jens Axboe]

On 2011-06-01 16:33, Troels Arvin wrote:
> Hello,
> 
> Jens Axboe wrote:
>> The iometer like test
>> case you are using is meant to be used on a raw block device, not on a
>> file system.
> 
> OK, but how does one use fio with a raw device? Can you give an example of a job specification file for working with a raw block device?

To take your example:

[global]
bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10
rw=randrw
rwmixread=70
direct=1
size=10g
ioengine=libaio
iodepth=256
write_bw_log
write_lat_log
numjobs=6

[sda]
filename=/dev/sda

would run the workload in question directly on the raw sda device.

> And can that be done in a way which simulates parallel I/O (several
> processes/threads working simultaneously)?

For the above example, you have 6 processes doing random read/write
patterns with the 70/30 read/write mix and with the block sizes given.

-- 
Jens Axboe