Reasoning of exposing queue/rotational=0

Reasoning of exposing queue/rotational=0

[Kai Krakow]

Hello!

What's the reasoning for exposing bcache devices as being
non-rotational? Currently, it fools btrfs into using ssd allocation
scheme on the underlying harddisks which isn't really what I expected
to get. So I used a udev rule to change this:

ACTION=="add|change", KERNEL=="bcache*", ATTR{queue/rotational}="1"

Wouldn't it make more sense to set this to the same value as the
underlying backing device by default?

Because in reality, the bcache is still what the backing device is: A
rotational medium. A cache doesn't make this non-rotational.

Thoughts?

-- 
Regards,
Kai

Replies to list-only preferred.

Re: Reasoning of exposing queue/rotational=0

[Coly Li]

On 2017/5/5 上午5:24, Kai Krakow wrote:
> Hello!
> 
> What's the reasoning for exposing bcache devices as being
> non-rotational? Currently, it fools btrfs into using ssd allocation
> scheme on the underlying harddisks which isn't really what I expected
> to get. So I used a udev rule to change this:
> 
> ACTION=="add|change", KERNEL=="bcache*", ATTR{queue/rotational}="1"
> 
> Wouldn't it make more sense to set this to the same value as the
> underlying backing device by default?
> 
> Because in reality, the bcache is still what the backing device is: A
> rotational medium. A cache doesn't make this non-rotational.
> 
> Thoughts?

It depends on hit ration. If a non-rotational device used as cache, and
hit ration is high enough, the cached device just responses as
non-rotational device.

But yes, I feel your opinion makes sense, in the btrfs case. How about a
policy like this:


cache-device-rotational   backing-device-rotational   export-rotational
         Y                            Y                      Y
         Y                            N                      N
         N                            Y                      N
         N                            N                      N

That is, a bcache device is exposed as non-rotational device only when
all devices of cache devices and backing devices are all rotational.

Thanks.

-- 
Coly Li

Re: Reasoning of exposing queue/rotational=0

[Vojtech Pavlik]

On Sat, May 06, 2017 at 12:11:13AM +0800, Coly Li wrote:
> On 2017/5/5 上午5:24, Kai Krakow wrote:
> > Hello!
> > 
> > What's the reasoning for exposing bcache devices as being
> > non-rotational? Currently, it fools btrfs into using ssd allocation
> > scheme on the underlying harddisks which isn't really what I expected
> > to get. So I used a udev rule to change this:
> > 
> > ACTION=="add|change", KERNEL=="bcache*", ATTR{queue/rotational}="1"
> > 
> > Wouldn't it make more sense to set this to the same value as the
> > underlying backing device by default?
> > 
> > Because in reality, the bcache is still what the backing device is: A
> > rotational medium. A cache doesn't make this non-rotational.
> > 
> > Thoughts?
> 
> It depends on hit ration. If a non-rotational device used as cache, and
> hit ration is high enough, the cached device just responses as
> non-rotational device.
> 
> But yes, I feel your opinion makes sense, in the btrfs case. How about a
> policy like this:
> 
> 
> cache-device-rotational   backing-device-rotational   export-rotational
>          Y                            Y                      Y
>          Y                            N                      N
>          N                            Y                      N
>          N                            N                      N
> 
> That is, a bcache device is exposed as non-rotational device only when
> all devices of cache devices and backing devices are all rotational.

I don't think that makes much sense either - the cache device will not
be used in the pattern that the exposed bcache device is, so any choice
of access patterns by a higher level based on rotational/non-rotational
will be messed up anyway.

I think the current behavior (rotational=0) is correct in most cases.

-- 
Vojtech Pavlik
Director SuSE Labs

Re: Reasoning of exposing queue/rotational=0

[Kai Krakow]

Am Sat, 6 May 2017 00:11:13 +0800
schrieb Coly Li <i@coly.li>:

> On 2017/5/5 上午5:24, Kai Krakow wrote:
> > Hello!
> > 
> > What's the reasoning for exposing bcache devices as being
> > non-rotational? Currently, it fools btrfs into using ssd allocation
> > scheme on the underlying harddisks which isn't really what I
> > expected to get. So I used a udev rule to change this:
> > 
> > ACTION=="add|change", KERNEL=="bcache*", ATTR{queue/rotational}="1"
> > 
> > Wouldn't it make more sense to set this to the same value as the
> > underlying backing device by default?
> > 
> > Because in reality, the bcache is still what the backing device is:
> > A rotational medium. A cache doesn't make this non-rotational.
> > 
> > Thoughts?  
> 
> It depends on hit ration. If a non-rotational device used as cache,
> and hit ration is high enough, the cached device just responses as
> non-rotational device.
> 
> But yes, I feel your opinion makes sense, in the btrfs case. How
> about a policy like this:
> 
> 
> cache-device-rotational   backing-device-rotational export-rotational
> Y                            Y                      Y
> Y                            N                      N
> N                            Y                      N
> N                            N                      N

This probably makes most sense, although it won't fix my particular
situation... Because I have:

 cdev    bdev    bcache
  N   &&  Y   ==  N
 
But I'd like to have bcache == Y

Hit rate is around 70-85% for me (500GB cache on 2TB data). So your
particular reasoning makes sense, too: 80% of accesses hit the cache
which makes it behave like non-rotational in 80% of all accesses.

But the bcache device itself is only a transition layer, especially we
cannot set any IO scheduler for it, this is left to the lower layers.
And these correctly expose the rotational flag, and that is where I set
deadline for SSD, and cfq for HDD. I also experimented with slice_idle
= 0 on SSD with cfq but deadline gave better results.

Given that, what could the rotational flag also be used for? Currently
it's used by btrfs to select an allocation scheme. I can imagine that
other filesystems do that, too. Does the kernel depend any decision on
this flag? Or anything else other then allocation decision?

Given the case of allocation decision: It makes no sense to pretend SSD
allocation through bcache as bcache block allocation is translated to
the real device and has nothing to do with the actual physical layout
of the backing device. So why pretend it is non-rotational?

Also think of discarding the cache: Now it would be clearly rotational
until cache hit rate builds up again.

Also I don't think applications should mis-interpret the bcache as
non-rotational to optimize workloads for it, because bcache is a
caching layer. It operates exactly for the purpose of optimizing those
workloads itself. Doing otherwise could work against what bcache tries
to achieve, e.g. doing lots of random IO because we pretend to be
non-rotational would push my precious cache data out of the cache for
no reason. Bcache is there to turn random IO into sequential IO - but
not for the sake of "because it can". Applications should still
optimize for rotational media even when running through bcache.

Without further clues it makes most sense to me to set
bcache.rotational = bdev.rotational.

I almost think that bcache does not explicitly set this flag, so it
stays 0. I think the same applies to iscsi and other network block
devices which pretend to be also non-rotational although in reality
they probably aren't. Only, they should probably explicitly not use an
IO scheduler as that is best left to the host system - as in virtual
guests, and as with enterprise RAID controllers, which do their own IO
scheduling. But "rotational" is totally not a decision we would
automatically select a default IO scheduler by. This should be left to
layers that more exactly know what a device is, e.g. udev or the
administrator.

> That is, a bcache device is exposed as non-rotational device only when
> all devices of cache devices and backing devices are all rotational.

I didn't really get that sentence... Either appearance of rotational
seems to be wrong in your sentence. ;-)


-- 
Regards,
Kai

Replies to list-only preferred.

Re: Reasoning of exposing queue/rotational=0

[Kai Krakow]

Am Fri, 5 May 2017 19:44:39 +0200
schrieb Vojtech Pavlik <vojtech@suse.com>:

> On Sat, May 06, 2017 at 12:11:13AM +0800, Coly Li wrote:
> > On 2017/5/5 上午5:24, Kai Krakow wrote:  
> > > Hello!
> > > 
> > > What's the reasoning for exposing bcache devices as being
> > > non-rotational? Currently, it fools btrfs into using ssd
> > > allocation scheme on the underlying harddisks which isn't really
> > > what I expected to get. So I used a udev rule to change this:
> > > 
> > > ACTION=="add|change", KERNEL=="bcache*",
> > > ATTR{queue/rotational}="1"
> > > 
> > > Wouldn't it make more sense to set this to the same value as the
> > > underlying backing device by default?
> > > 
> > > Because in reality, the bcache is still what the backing device
> > > is: A rotational medium. A cache doesn't make this non-rotational.
> > > 
> > > Thoughts?  
> > 
> > It depends on hit ration. If a non-rotational device used as cache,
> > and hit ration is high enough, the cached device just responses as
> > non-rotational device.
> > 
> > But yes, I feel your opinion makes sense, in the btrfs case. How
> > about a policy like this:
> > 
> > 
> > cache-device-rotational   backing-device-rotational
> > export-rotational Y
> > Y                      Y Y
> > N                      N N
> > Y                      N N
> > N                      N
> > 
> > That is, a bcache device is exposed as non-rotational device only
> > when all devices of cache devices and backing devices are all
> > rotational.  
> 
> I don't think that makes much sense either - the cache device will not
> be used in the pattern that the exposed bcache device is, so any
> choice of access patterns by a higher level based on
> rotational/non-rotational will be messed up anyway.
> 
> I think the current behavior (rotational=0) is correct in most cases.

Well, I don't want to do bikeshedding... But both didn't answer my
original question of what's the reasoning. Did anyone put thoughts into
this? Was it arbitrarily chosen? Is rotational=0 just a default that
bcache didn't bother to explicitly set?

Answering the last two questions with "yes" would suggest that it should
be rethought...

Answering the first with "yes" means I'd like to know more. ;-)


-- 
Regards,
Kai

Replies to list-only preferred.

Re: Reasoning of exposing queue/rotational=0

[Kai Krakow]

Am Fri, 5 May 2017 19:44:39 +0200
schrieb Vojtech Pavlik <vojtech@suse.com>:

> On Sat, May 06, 2017 at 12:11:13AM +0800, Coly Li wrote:
> > On 2017/5/5 上午5:24, Kai Krakow wrote:  
> > > Hello!
> > > 
> > > What's the reasoning for exposing bcache devices as being
> > > non-rotational? Currently, it fools btrfs into using ssd
> > > allocation scheme on the underlying harddisks which isn't really
> > > what I expected to get. So I used a udev rule to change this:
> > > 
> > > ACTION=="add|change", KERNEL=="bcache*",
> > > ATTR{queue/rotational}="1"
> > > 
> > > Wouldn't it make more sense to set this to the same value as the
> > > underlying backing device by default?
> > > 
> > > Because in reality, the bcache is still what the backing device
> > > is: A rotational medium. A cache doesn't make this non-rotational.
> > > 
> > > Thoughts?  
> > 
> > It depends on hit ration. If a non-rotational device used as cache,
> > and hit ration is high enough, the cached device just responses as
> > non-rotational device.
> > 
> > But yes, I feel your opinion makes sense, in the btrfs case. How
> > about a policy like this:
> > 
> > 
> > cache-device-rotational   backing-device-rotational
> > export-rotational Y
> > Y                      Y Y
> > N                      N N
> > Y                      N N
> > N                      N
> > 
> > That is, a bcache device is exposed as non-rotational device only
> > when all devices of cache devices and backing devices are all
> > rotational.  
> 
> I don't think that makes much sense either - the cache device will not
> be used in the pattern that the exposed bcache device is, so any
> choice of access patterns by a higher level based on
> rotational/non-rotational will be messed up anyway.

BTW: Exactly that would be the reasoning for me to not set it
statically to 0, but instead to the value of the backing device. For
example, turning it from 1 into 0 up the layers already messes up with
decisions btrfs takes.

In the end, bcache doesn't magically turn my storage into
non-rotational. It is more about turning random IO into sequential IO.
That you get higher throughput also and almost 0 seek time for cache
hits, is just a by-product (tho, a very welcome one).

Given the case that write-caching is set to write-around, or
write-through, your application would still see rotational behavior but
the flag tells it "non-rotational". That seems wrong. Only write-back
caching gives you non-rotational write behavior as seen from the
application. And when bcache passes the sequential cutoff, it doesn't
matter anyway. But now a wrong assumption about revolution can come
into play: An application could try to do sequential IO if seeing
rotational media - but now it doesn't care: It will waste wear-leveling
and discard data that really should belong into the cache.

And when reading, it behaves more like a big, permanent block cache: If
the cache is hit, that's more comparable to a hit in the page/block
cache of the kernel. If it's a miss, it still looks like rotational
access to the application.

So what's the deal?

> I think the current behavior (rotational=0) is correct in most cases.

Currently I don't see why. What defines "most cases"?


-- 
Regards,
Kai

Replies to list-only preferred.

Re: Reasoning of exposing queue/rotational=0

[Vojtech Pavlik]

On Fri, May 05, 2017 at 08:23:17PM +0200, Kai Krakow wrote:
> > I don't think that makes much sense either - the cache device will not
> > be used in the pattern that the exposed bcache device is, so any
> > choice of access patterns by a higher level based on
> > rotational/non-rotational will be messed up anyway.
> > 
> > I think the current behavior (rotational=0) is correct in most cases.
> 
> Well, I don't want to do bikeshedding... But both didn't answer my
> original question of what's the reasoning. Did anyone put thoughts into
> this? 

Originally, rotational=1 is just a flag coming from the
IDE/SCSI/SATA/etc. layers to the OS telling it whether the device is
spinning or not. Without any specific implications as to the behavior of
the device.

It is writable for a reason - not even all flash based devices report
the flag correctly at the hardware level.

Linux uses the flag on the block device (queue) to tell whether seeks
are very expensive compared to linear reads and whether it makes sense
to spend large amounts CPU cycles and memory on reordering.

Btrfs is one user that tries to change the allocation policy and thus
the likelihood of fragmentation and/or long seeks based on whether the
device reports 'rotational'.

However, it actually has three modes at the fs level: 'nossd',
'ssd' and 'ssd_spread', with the last being faster on cheaper SSDs.
There are large differences even between individual SSD profiles. Again,
for a good reason, btrfs has these as mount options that override any
'rotational' hint.

All in all, if you want all the performance available, you need to see
what works best for your workload.

The same applies to i/o schedulers. They're much less dependent on the
underlying device than the workload put on them.

This is not the first time the question comes up.

> Was it arbitrarily chosen? Is rotational=0 just a default that
> bcache didn't bother to explicitly set?

A bcache device performance profile is neither one of a rotational
device, nor one of a SSD.

Sequential reads may be bypassed or not. If not, some parts of it may
be cached, in which case there will be seeks on the backing device even
when there should be none on a real rotational device.

Random reads may be fast if they're hitting cached locations.

Random and sequential writes will be always cached if writeback is
enabled and so there is no point in spending CPU cycles on optimizing
writes.

How much the bcache device will behave like the backing device and how
much like the caching device does depend mainly on the workload and the
size of its working set compared to the size of the cache.

I do not believe that the choice of rotational=0 was arbitrary or a
default. It's simply that bcache changes the access pattern to both the
caching and backing device so much that it no longer resembles a
rotational device's performance profile in any case.

> Answering the last two questions with "yes" would suggest that it should
> be rethought...
> 
> Answering the first with "yes" means I'd like to know more. ;-)

-- 
Vojtech Pavlik
Director SuSE Labs

Re: Reasoning of exposing queue/rotational=0

[Kai Krakow]

Am Fri, 5 May 2017 21:02:31 +0200
schrieb Vojtech Pavlik <vojtech@suse.com>:

> On Fri, May 05, 2017 at 08:23:17PM +0200, Kai Krakow wrote:
> > > I don't think that makes much sense either - the cache device
> > > will not be used in the pattern that the exposed bcache device
> > > is, so any choice of access patterns by a higher level based on
> > > rotational/non-rotational will be messed up anyway.
> > > 
> > > I think the current behavior (rotational=0) is correct in most
> > > cases.  
> > 
> > Well, I don't want to do bikeshedding... But both didn't answer my
> > original question of what's the reasoning. Did anyone put thoughts
> > into this?   
> 
> Originally, rotational=1 is just a flag coming from the
> IDE/SCSI/SATA/etc. layers to the OS telling it whether the device is
> spinning or not. Without any specific implications as to the behavior
> of the device.
> 
> It is writable for a reason - not even all flash based devices report
> the flag correctly at the hardware level.
> 
> Linux uses the flag on the block device (queue) to tell whether seeks
> are very expensive compared to linear reads and whether it makes sense
> to spend large amounts CPU cycles and memory on reordering.
> 
> Btrfs is one user that tries to change the allocation policy and thus
> the likelihood of fragmentation and/or long seeks based on whether the
> device reports 'rotational'.
> 
> However, it actually has three modes at the fs level: 'nossd',
> 'ssd' and 'ssd_spread', with the last being faster on cheaper SSDs.
> There are large differences even between individual SSD profiles.
> Again, for a good reason, btrfs has these as mount options that
> override any 'rotational' hint.
> 
> All in all, if you want all the performance available, you need to see
> what works best for your workload.
> 
> The same applies to i/o schedulers. They're much less dependent on the
> underlying device than the workload put on them.
> 
> This is not the first time the question comes up.

I tried to look up information about it previously but didn't came up
with useful results.

> > Was it arbitrarily chosen? Is rotational=0 just a default that
> > bcache didn't bother to explicitly set?  
> 
> A bcache device performance profile is neither one of a rotational
> device, nor one of a SSD.
> 
> Sequential reads may be bypassed or not. If not, some parts of it may
> be cached, in which case there will be seeks on the backing device
> even when there should be none on a real rotational device.
> 
> Random reads may be fast if they're hitting cached locations.
> 
> Random and sequential writes will be always cached if writeback is
> enabled and so there is no point in spending CPU cycles on optimizing
> writes.
> 
> How much the bcache device will behave like the backing device and how
> much like the caching device does depend mainly on the workload and
> the size of its working set compared to the size of the cache.
> 
> I do not believe that the choice of rotational=0 was arbitrary or a
> default. It's simply that bcache changes the access pattern to both
> the caching and backing device so much that it no longer resembles a
> rotational device's performance profile in any case.
> 
> > Answering the last two questions with "yes" would suggest that it
> > should be rethought...
> > 
> > Answering the first with "yes" means I'd like to know more. ;-)  

Okay, that answers my questions. Thanks. :-)

But that only tells me that a "default" cannot be really chosen. Both
make sense.

I wonder if Linux chose to call the flag "non_rotational", would it
also default to 0 in bcache? I think nobody would know. ;-)

For me it looks like sticking that to rotational=1 gives overall better
long-time performance and btrfs filesystem layout.

Anyone who stumbles across this should judge on his own based on
Vojtech's good answer.


-- 
Regards,
Kai

Replies to list-only preferred.

Re: Reasoning of exposing queue/rotational=0

[Eric Wheeler]

On Fri, 5 May 2017, Kai Krakow wrote:

> Am Fri, 5 May 2017 21:02:31 +0200
> schrieb Vojtech Pavlik <vojtech@suse.com>:
> 
> > On Fri, May 05, 2017 at 08:23:17PM +0200, Kai Krakow wrote:
> > > > I don't think that makes much sense either - the cache device
> > > > will not be used in the pattern that the exposed bcache device
> > > > is, so any choice of access patterns by a higher level based on
> > > > rotational/non-rotational will be messed up anyway.
> > > > 
> > > > I think the current behavior (rotational=0) is correct in most
> > > > cases.  
> > > 
> > > Well, I don't want to do bikeshedding... But both didn't answer my
> > > original question of what's the reasoning. Did anyone put thoughts
> > > into this?   
> > 
> > Originally, rotational=1 is just a flag coming from the
> > IDE/SCSI/SATA/etc. layers to the OS telling it whether the device is
> > spinning or not. Without any specific implications as to the behavior
> > of the device.
> > 
> > It is writable for a reason - not even all flash based devices report
> > the flag correctly at the hardware level.
> > 
> > Linux uses the flag on the block device (queue) to tell whether seeks
> > are very expensive compared to linear reads and whether it makes sense
> > to spend large amounts CPU cycles and memory on reordering.
> > 
> > Btrfs is one user that tries to change the allocation policy and thus
> > the likelihood of fragmentation and/or long seeks based on whether the
> > device reports 'rotational'.
> > 
> > However, it actually has three modes at the fs level: 'nossd',
> > 'ssd' and 'ssd_spread', with the last being faster on cheaper SSDs.
> > There are large differences even between individual SSD profiles.
> > Again, for a good reason, btrfs has these as mount options that
> > override any 'rotational' hint.
> > 
> > All in all, if you want all the performance available, you need to see
> > what works best for your workload.
> > 
> > The same applies to i/o schedulers. They're much less dependent on the
> > underlying device than the workload put on them.
> > 
> > This is not the first time the question comes up.
> 
> I tried to look up information about it previously but didn't came up
> with useful results.
> 
> > > Was it arbitrarily chosen? Is rotational=0 just a default that
> > > bcache didn't bother to explicitly set?  
> > 
> > A bcache device performance profile is neither one of a rotational
> > device, nor one of a SSD.
> > 
> > Sequential reads may be bypassed or not. If not, some parts of it may
> > be cached, in which case there will be seeks on the backing device
> > even when there should be none on a real rotational device.
> > 
> > Random reads may be fast if they're hitting cached locations.
> > 
> > Random and sequential writes will be always cached if writeback is
> > enabled and so there is no point in spending CPU cycles on optimizing
> > writes.
> > 
> > How much the bcache device will behave like the backing device and how
> > much like the caching device does depend mainly on the workload and
> > the size of its working set compared to the size of the cache.
> > 
> > I do not believe that the choice of rotational=0 was arbitrary or a
> > default. It's simply that bcache changes the access pattern to both
> > the caching and backing device so much that it no longer resembles a
> > rotational device's performance profile in any case.
> > 
> > > Answering the last two questions with "yes" would suggest that it
> > > should be rethought...
> > > 
> > > Answering the first with "yes" means I'd like to know more. ;-)  
> 
> Okay, that answers my questions. Thanks. :-)
> 
> But that only tells me that a "default" cannot be really chosen. Both
> make sense.
> 
> I wonder if Linux chose to call the flag "non_rotational", would it
> also default to 0 in bcache? I think nobody would know. ;-)
> 
> For me it looks like sticking that to rotational=1 gives overall better
> long-time performance and btrfs filesystem layout.
> 
> Anyone who stumbles across this should judge on his own based on
> Vojtech's good answer.

Indeed!

Also note:

# cat /sys/block/bcache0/queue/scheduler 
none

There is no scheduler for bcache, so the bio's pass through whatever your 
backing (cache) device uses as a queue scheduler, which could differ 
between cache/backing.  If you use hardware RAID, your 'rotational' flag 
is probably wrong for SSDs so set it on boot somehow (udev, etc.)

--
Eric Wheeler



> 
> 
> -- 
> Regards,
> Kai
> 
> Replies to list-only preferred.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Reasoning of exposing queue/rotational=0

[Kai Krakow]

Am Tue, 9 May 2017 18:11:06 +0000 (UTC)
schrieb Eric Wheeler <bcache@lists.ewheeler.net>:

> On Fri, 5 May 2017, Kai Krakow wrote:
> 
> > Am Fri, 5 May 2017 21:02:31 +0200
> > schrieb Vojtech Pavlik <vojtech@suse.com>:
> >   
> > > On Fri, May 05, 2017 at 08:23:17PM +0200, Kai Krakow wrote:  
>  [...]  
>  [...]  
> > > 
> > > Originally, rotational=1 is just a flag coming from the
> > > IDE/SCSI/SATA/etc. layers to the OS telling it whether the device
> > > is spinning or not. Without any specific implications as to the
> > > behavior of the device.
> > > 
> > > It is writable for a reason - not even all flash based devices
> > > report the flag correctly at the hardware level.
> > > 
> > > Linux uses the flag on the block device (queue) to tell whether
> > > seeks are very expensive compared to linear reads and whether it
> > > makes sense to spend large amounts CPU cycles and memory on
> > > reordering.
> > > 
> > > Btrfs is one user that tries to change the allocation policy and
> > > thus the likelihood of fragmentation and/or long seeks based on
> > > whether the device reports 'rotational'.
> > > 
> > > However, it actually has three modes at the fs level: 'nossd',
> > > 'ssd' and 'ssd_spread', with the last being faster on cheaper
> > > SSDs. There are large differences even between individual SSD
> > > profiles. Again, for a good reason, btrfs has these as mount
> > > options that override any 'rotational' hint.
> > > 
> > > All in all, if you want all the performance available, you need
> > > to see what works best for your workload.
> > > 
> > > The same applies to i/o schedulers. They're much less dependent
> > > on the underlying device than the workload put on them.
> > > 
> > > This is not the first time the question comes up.  
> > 
> > I tried to look up information about it previously but didn't came
> > up with useful results.
> >   
>  [...]  
> > > 
> > > A bcache device performance profile is neither one of a rotational
> > > device, nor one of a SSD.
> > > 
> > > Sequential reads may be bypassed or not. If not, some parts of it
> > > may be cached, in which case there will be seeks on the backing
> > > device even when there should be none on a real rotational device.
> > > 
> > > Random reads may be fast if they're hitting cached locations.
> > > 
> > > Random and sequential writes will be always cached if writeback is
> > > enabled and so there is no point in spending CPU cycles on
> > > optimizing writes.
> > > 
> > > How much the bcache device will behave like the backing device
> > > and how much like the caching device does depend mainly on the
> > > workload and the size of its working set compared to the size of
> > > the cache.
> > > 
> > > I do not believe that the choice of rotational=0 was arbitrary or
> > > a default. It's simply that bcache changes the access pattern to
> > > both the caching and backing device so much that it no longer
> > > resembles a rotational device's performance profile in any case.
> > >   
>  [...]  
> > 
> > Okay, that answers my questions. Thanks. :-)
> > 
> > But that only tells me that a "default" cannot be really chosen.
> > Both make sense.
> > 
> > I wonder if Linux chose to call the flag "non_rotational", would it
> > also default to 0 in bcache? I think nobody would know. ;-)
> > 
> > For me it looks like sticking that to rotational=1 gives overall
> > better long-time performance and btrfs filesystem layout.
> > 
> > Anyone who stumbles across this should judge on his own based on
> > Vojtech's good answer.  
> 
> Indeed!
> 
> Also note:
> 
> # cat /sys/block/bcache0/queue/scheduler 
> none

Yes, I know that.

> There is no scheduler for bcache, so the bio's pass through whatever
> your backing (cache) device uses as a queue scheduler, which could
> differ between cache/backing.

What does this exactly mean? I understand that depending on where the
bio ends up, I'm using two different IO schedulers.

At least this is how I currently set things up: I use different
scheduler (or different scheduler settings) to exploit exactly that
behavior.

> If you use hardware RAID, your
> 'rotational' flag is probably wrong for SSDs so set it on boot
> somehow (udev, etc.)

No hardware RAID involved here... Just three plain disks and one SSD.

I'm currently using udev to force it "1" for the bcache compound device
(which is what I guess the filesystem is seeing). The underlying bdev
and cdev still have their original rotational flag set, I didn't touch
it.


-- 
Regards,
Kai

Replies to list-only preferred.