RFC swap over raid1

RFC swap over raid1

[Roberto Spadim]

hi guys, should i use raid1 with swap partitions? why?

example:
swapon /dev/sda1
mdadm --create /dev/md-swap --level=1 --raid-devices=2 /dev/sda /dev/sdb
mkswap /dev/md-swap
swapon /dev/md-swap

what's the advantages and disvantages?
should i use raid0, linear, raid1, raid10, raid5 or raid6?

or just two swap partitions and run this way:
swapon /dev/sda
swapon /dev/sdb
?

thanks

-- 
Roberto Spadim
SPAEmpresarial

Re: RFC swap over raid1

[Doug Ledford]

On 08/01/2013 06:11 PM, Roberto Spadim wrote:
> hi guys, should i use raid1 with swap partitions? why?
> 
> example:
> swapon /dev/sda1
> mdadm --create /dev/md-swap --level=1 --raid-devices=2 /dev/sda /dev/sdb

You can't do this.  You've just corrupted your hard drives.  The reason
is that you've set /dev/sda1 to be used for swap, then you've made a
raid1 array from the entirety of /dev/sda and /dev/sdb, which of course
includes the space in /dev/sda1, which is already in use.

> mkswap /dev/md-swap

Here you've just turned all of /dev/sda into a swap device (as well as
/dev/sdb).

> swapon /dev/md-swap

And here you've turned on a swap device that is /dev/sda and which will
overlap with the swap you already enabled on /dev/sda1.  You're using
the same space twice.  It's definitely corrupted and broken.
Fortunately, I'm pretty sure the kernel won't even allow you to do all
of this.  You would have been stopped at the mdadm --create because
/dev/sda1 was in use so /dev/sda couldn't be allocated to the newly
created md raid array.

> what's the advantages and disvantages?
> should i use raid0, linear, raid1, raid10, raid5 or raid6?

These are all the same basic questions as if you were running something
other than swap.  The same tradeoffs exists.  Although I would never use
raid4/5/6 for swap as you don't want to have to use parity on your swap,
and raid10 is overkill.  If you use raid0 or linear and loose a drive,
you can crash your machine when the swap is needed.  I only use raid1
(if I use swap at all, which is not that often any more).

> or just two swap partitions and run this way:
> swapon /dev/sda
> swapon /dev/sdb
> ?

What exactly do you think swapon does?  Because you've just allocated
all of /dev/sda and /dev/sdb to swap and they can't be used for anything
else.

Re: RFC swap over raid1

[Brad Campbell]

On 02/08/13 06:11, Roberto Spadim wrote:
> hi guys, should i use raid1 with swap partitions? why?

> what's the advantages and disvantages?


In parallel to Doug's comments. A few years ago, I spent literally 
*days* trying to figure out why random processes were dying on one of my 
servers. It turned out there was a bad block in the swap partition (this 
was pre-SMART). Ever since then, if I've used swap on a machine that was 
important enough, I used RAID-1.

Re: RFC swap over raid1

[Roberto Spadim]

ops, sorry the first swapon was a typo (control v) sorry, i know using
the swapon lock device, only swapoff and after a mdadm --create could
work in this case, but remove the first swapon... just mdadm create,
and swap over md device is the point here...

the /dev/sda and /dev/sdb was just examples, they can be partitions
without problems

the point is: using swap at two partitions / disks, is "better" than
using a swap over a md raid1? (or any other level?)
other point is... swap have a badblock feature? i think it's not
linux-raid but linux-vm or something like it...
for example if i'm using a disk and swap find a badblock, it will use
it? does swap handle bad blocks? it remove the device? continue using
it? or change the device priority?

Re: RFC swap over raid1

[Roberto Spadim]

hi brad, what kernel version you was using? anyone know if the new 3.9
and 3.10 have a badblock feature at swap?
anyone know who control the swap? or what source files should i look
to understand swap better?

Re: RFC swap over raid1

[Brad Campbell]

On 02/08/13 10:02, Roberto Spadim wrote:
> hi brad, what kernel version you was using? anyone know if the new 3.9
> and 3.10 have a badblock feature at swap?
> anyone know who control the swap? or what source files should i look
> to understand swap better?

Back then it would have been 2.2 or possibly early 2.4.
Can't really help with the other questions. I still have large amounts 
of swap on all my machines for historical reasons, however it rarely 
gets touched and certainly not intentionally.

I find it much nicer to be able to deal with performance issues caused 
by a runaway processes forcing stuff into swap than it is to deal with 
"random" processes being shot by the OOM killer.

Re: RFC swap over raid1

[Roberto Spadim]

nice, i used 2.2 and 2.4, they was very nice kernels, now we have
newers features, and i read some emails (i don't remember when) about
swap over raid1 isn't important, since newer kernels swap feature
could handle disks better now, and raid1 could be forget... but i
don't remember when, why, where :/

anyone who have more information please help =)
thanks guys

Re: RFC swap over raid1

[Stan Hoeppner]

On 8/1/2013 9:01 PM, Roberto Spadim wrote:

> other point is... swap have a badblock feature? i think it's not
> linux-raid but linux-vm or something like it...
> for example if i'm using a disk and swap find a badblock, it will use
> it? does swap handle bad blocks?

I believe you're thinking of "mkswap -c":

"Check the device (if it is a block device) for bad blocks before
creating the swap area. If any are found, the count is printed."

This simply tells you if bad blocks are found during mkswap, and how
many.  It doesn't tell you the locations of the blocks nor attempt to
remap them.  It's informational only.  Remedial action is left to the user.

I'm not aware of any code in the mm or block layer that transparently
handles bad block management, nor code that simply tracks bad blocks to
avoid using them.  If there were such a patch, it would not apply simply
to swap, but to the entire block layer.  I've not heard of any such
thing in recent development.  If this was included in the block layer
you'd surely have seen emails about it on the linux-raid list, as the
current md code that deals with bad blocks would have needed rewriting
to interface with any new generic interface in the block layer.

So if you're worried about your swap partition sitting on potentially
bad blocks you'll want to have one form or another of redundant md
device sitting under that swap partition.

However, you stated you're using enterprise class drives.  These are
usually pretty good about remapping bad blocks on the fly, and have much
larger reserved block pools than consumer drives for remapping use, so
this may simply be a non-issue.

-- 
Stan

Re: RFC swap over raid1

[Roberto Spadim]

hummm, wow very new informations to me...

today linux don't have a generic badblock remap?! that's what i understood?
for example... ext2,3,4,xfs,reiserfs,zfs, and others fs, they handle
badblock by their self? right?

it's a nice information i never thinked about a layer only for
badblock reallocation, i read/write in this list of linux-raid when
the started the badblock development, in some time near to raid1 write
multithread

today the badblock of raid1 is embedded in the source? or it's easy to
implement a new layer just to badblock realloc logic?

about "mkswap -c" it just show information like you told, i'm a bit
surprised about no badblock at swap that's information is new to me i
will read about others os (freebsd, reactos, etc) to check how they
handle this there

I'm rethinking now about the swap as a file in a filesystem, this
could increase security or another solution is better?

Re: RFC swap over raid1

[Doug Ledford]

On 08/01/2013 10:01 PM, Roberto Spadim wrote:
> the point is: using swap at two partitions / disks, is "better" than
> using a swap over a md raid1? (or any other level?)

That depends on your goals.  If your goal is for your system to be
resilient to disk failure, and you've put your filesystems on a raid
device to be tolerant of failure, then your swap needs to be on one too
or else you are undermining the work you did on your filesystems.  If
that's not the case, then two swap partitions gets you twice the
capacity, but slightly lower performance, than a raid1 swap device over
the same partitions due to the fact that when you are swapping out,
performance is about the same, but when swapping in, we can load balance
reads and increase performance.  Of course, you only have half as much
swap this way.

> other point is... swap have a badblock feature? i think it's not
> linux-raid but linux-vm or something like it...
> for example if i'm using a disk and swap find a badblock, it will use
> it? does swap handle bad blocks? it remove the device? continue using
> it? or change the device priority?

The swap layer does not do bad blocks *while in use*.  You can pass in a
list of badblocks when creating the device, in which case it won't ever
use those blocks.  However, if a block *goes* bad while in use, and we
get a read error, then whatever application was trying to page in that
page of swap is going to get killed due to an unhandled page fault.

Re: RFC swap over raid1

[Doug Ledford]

On 08/02/2013 10:21 AM, Roberto Spadim wrote:
> hummm, wow very new informations to me...
> 
> today linux don't have a generic badblock remap?! that's what i understood?
> for example... ext2,3,4,xfs,reiserfs,zfs, and others fs, they handle
> badblock by their self? right?

No, not really.  I think back in the day, badblock support mattered, but
it was really just limited to having a list of known bad sectors that
the filesystem would never use because once the sector failed, it was
toast.  But that was when the firmware on the disk controllers didn't
have a pool of spare sectors that were available to remap bad blocks.

Now a days, the drives automatically remap bad sectors into their own
internal spare sector pool.  The only time the OS sees a bad block is
when it went bad by surprise to the drive and so the data couldn't be
read and remapped before it went away.

In that case, you just rewrite something to the bad sector, and
generally the drive firmware will have remap-on-write-error enabled and
generally the failed sector will not only fail to be read but will also
fail to be written, and so the drive will remap the bad sector to a
spare as long as it has spares available.

It's for this reason that, with modern drives, a failed read is somewhat
acceptable as it will likely be fixed simply by writing back to the same
sector, but if that sector persists in being bad even after a write,
then you know that the drive's internal pool of spare sectors are all
allocated and so all future failures on the drive will be permanent
failures.  It's at that point that you need to replace the drive ASAP.

However, filesystems don't keep two copies of their data laying around
in order to rewrite bad sectors.  The md raid layer does (when using a
reliable level of course).  Basically, badblock management by
filesystems has always just been to mark a sector as bad and work around
it (with a possibly corrupted file as a result).  Badblock management by
the raid subsystem is to try and get the drive to reallocate the sector
by rewriting the correct data to that sector.

> it's a nice information i never thinked about a layer only for
> badblock reallocation, i read/write in this list of linux-raid when
> the started the badblock development, in some time near to raid1 write
> multithread
> 
> today the badblock of raid1 is embedded in the source? or it's easy to
> implement a new layer just to badblock realloc logic?
> 
> about "mkswap -c" it just show information like you told, i'm a bit
> surprised about no badblock at swap that's information is new to me i
> will read about others os (freebsd, reactos, etc) to check how they
> handle this there
> 
> I'm rethinking now about the swap as a file in a filesystem, this
> could increase security or another solution is better?

It used to be that a swapfile on a filesystem was slower than swap on
its own partition.  I think they cleared that up some time ago so that
the speed difference is mostly negligible.  But having it as a file on
the filesystem makes management of partitions easier, so that's
something in the swapfile's favor.

Re: RFC swap over raid1

[Roberto Spadim]

nice good explanations thanks

one more doubt....
filesystem handle bad blocks
md handle bad blocks

what happen when i have a filesystem over md?
does md report badblocks to filesystem?

Re: RFC swap over raid1

[Doug Ledford]

On 08/02/2013 11:59 AM, Roberto Spadim wrote:
> nice good explanations thanks
> 
> one more doubt....
> filesystem handle bad blocks
> md handle bad blocks
> 
> what happen when i have a filesystem over md?
> does md report badblocks to filesystem?
> 

If md can not repair the block and it is unable to get the data from
some other device in the array, then and only then will it report the
error back up to the filesystem layer on top of it.  In that case, the
filesystem will see the error and execute whatever the default behavior
on error is, which is different from filesystem to filesystem, but in
the past has commonly been to force the filesystem immediately into a
read-only mode so no further damage can happen and attempts at repair
can be made in rescue mode.

Re: RFC swap over raid1

[Roberto Spadim]

humm interesting, a last doubt, what error is reported to filesystem
in this case today? badblock, read error, write error?

Re: RFC swap over raid1

[Doug Ledford]

On 08/02/2013 12:40 PM, Roberto Spadim wrote:
> humm interesting, a last doubt, what error is reported to filesystem
> in this case today? badblock, read error, write error?
> 

Just an error on the request.  If it was a read request, it is a read
error.  If it was a write request, then it is a write error.

Re: RFC swap over raid1

[Roberto Spadim]

nice, well, considering everything...

1)partitions + raid1 is prefered for read speed and security
2)fileswap only if i can't add partitions
3)only use swap over single drive if don't have two devices (in this
case use raid1)
4)there's no badblock layer in linux, each layer / filesystem must
implement it by yourself

that's all right?

Re: RFC swap over raid1

[Doug Ledford]

On 08/02/2013 01:29 PM, Roberto Spadim wrote:
> nice, well, considering everything...
> 
> 1)partitions + raid1 is prefered for read speed and security
> 2)fileswap only if i can't add partitions
> 3)only use swap over single drive if don't have two devices (in this
> case use raid1)
> 4)there's no badblock layer in linux, each layer / filesystem must
> implement it by yourself
> 
> that's all right?

Yep.

Re: RFC swap over raid1

[Roberto Spadim]

ok thanks guys =]

2013/8/2 Doug Ledford <dledford@redhat.com>:
> On 08/02/2013 01:29 PM, Roberto Spadim wrote:
>> nice, well, considering everything...
>>
>> 1)partitions + raid1 is prefered for read speed and security
>> 2)fileswap only if i can't add partitions
>> 3)only use swap over single drive if don't have two devices (in this
>> case use raid1)
>> 4)there's no badblock layer in linux, each layer / filesystem must
>> implement it by yourself
>>
>> that's all right?
>
> Yep.
>



-- 
Roberto Spadim
SPAEmpresarial

Re: RFC swap over raid1

[keld]

On Fri, Aug 02, 2013 at 02:29:50PM -0300, Roberto Spadim wrote:
> nice, well, considering everything...
> 
> 1)partitions + raid1 is prefered for read speed and security
> 2)fileswap only if i can't add partitions
> 3)only use swap over single drive if don't have two devices (in this
> case use raid1)
> 4)there's no badblock layer in linux, each layer / filesystem must
> implement it by yourself
> 
> that's all right?

I would use raid10,far for its faster read vs raid1.
There is badblock handling in the raid layer.

Best regards
keld

Re: RFC swap over raid1

[Roberto Spadim]

hum, swap "use" more sequencial writes or random writes?
i tested raid1 vs raid10,far for random writes/read and i'm using
raid1 for this kind of workload, for sequencial reads/writes the
raid10,far is faster (at least in my tests)

Re: RFC swap over raid1

[Keld Jørn Simonsen]

On Fri, Aug 02, 2013 at 03:39:22PM -0300, Roberto Spadim wrote:
> hum, swap "use" more sequencial writes or random writes?
> i tested raid1 vs raid10,far for random writes/read and i'm using
> raid1 for this kind of workload, for sequencial reads/writes the
> raid10,far is faster (at least in my tests)

swap use sequential writes, I think, when a process is swapped out.
There is not much difference between the different mirrorred 
raid layouts when it comes to random or sequential writes.
anyway the elevator for the whole  disk orders writes.


It is more when you need to swap in a process, that you need the 
read speed of raid10,far. Especially for big processes like firefox
or libreoffice.

Best regards
Keld