Feature Request - Full Bypass/Verify Mode

Feature Request - Full Bypass/Verify Mode

[Cobra_Fast]

[-- Attachment #1.1: Type: text/plain, Size: 1152 bytes --]

Greetings,

I am using bcache in conjunction with SnapRAID, which works on the 
FS-level, and I have noticed that parity syncs as well as scrubs read 
data from the cache rather than the backing device. This probably not a 
problem when creating parity for new files, but could be a problem when 
running scrubs, as the parity is never checked against data on disk 
since bcache hides it.

I would therefore very much like a cache_mode that would bypass any and 
all reads, that can be enabled for the duration of a SnapRAID sync or 
scrub. For writes I suppose this mode should act the same as "none".

This opportunity could be taken to verify data on cache as well; read 
from both backing and cache and invalidate the cache page if it differs 
from the backing data, while satisfying the actual read from backing in 
any case.

Perhaps something like this is already possible and I'm just not seeing it?
I know I can detach backing devices, but to my understanding that also 
invalidates all its cached pages and I would obviously like to keep them 
for this purpose.

Looking forward to your opinions,
Best regards,
Andy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: Feature Request - Full Bypass/Verify Mode

[Eric Wheeler]

On Wed, 5 Oct 2022, Cobra_Fast wrote:

> Greetings,
> 
> I am using bcache in conjunction with SnapRAID, which works on the FS-level,
> and I have noticed that parity syncs as well as scrubs read data from the
> cache rather than the backing device. This probably not a problem when
> creating parity for new files, but could be a problem when running scrubs, as
> the parity is never checked against data on disk since bcache hides it.

Interesting.
 
> I would therefore very much like a cache_mode that would bypass any and all
> reads, that can be enabled for the duration of a SnapRAID sync or scrub. For
> writes I suppose this mode should act the same as "none".
> 
> This opportunity could be taken to verify data on cache as well; read from
> both backing and cache and invalidate the cache page if it differs from the
> backing data, while satisfying the actual read from backing in any case.

assuming that one or the other is correct... I'm not sure bcache could 
tell which block is valid, and SnapRAID doesn't know about the lldevs.
 
> Perhaps something like this is already possible and I'm just not seeing it?
> I know I can detach backing devices, but to my understanding that also
> invalidates all its cached pages and I would obviously like to keep them for
> this purpose.

Well you can only read-validate pages that are not dirty...if its dirty 
you _must_ read from cache for consistency.

You could put it in write_around mode and wait for dirty_bytes to be 0, 
but I think it will still read from the cache if the page is hot.

Detach sounds like the only option at the moment to get what you're 
seeking.  Future work could include adding a `readaround` to 
/sys/block/bcache0/bcache/cache_mode, but it would still have to read from 
the cache if dirty.  Or maybe if `readaround` hits a dirty block it evicts 
it and re-reads the backing device?  But that sounds messy and slow.

Maybe you could assume that if the cache is correct then the backing 
device is correct and a verify atop of bcache means that bcache is 
consistent, independent of the lower layers.  This is certainly true for 
dirty blocks, and probably for clean (in-cache) blocks too.

-Eric

> 
> Looking forward to your opinions,
> Best regards,
> Andy
> 
> 

Re: Feature Request - Full Bypass/Verify Mode

[Coly Li]

> 2022年10月6日 09:39，Eric Wheeler <bcache@lists.ewheeler.net> 写道：
> 
> On Wed, 5 Oct 2022, Cobra_Fast wrote:
> 
>> Greetings,
>> 
>> I am using bcache in conjunction with SnapRAID, which works on the FS-level,
>> and I have noticed that parity syncs as well as scrubs read data from the
>> cache rather than the backing device. This probably not a problem when
>> creating parity for new files, but could be a problem when running scrubs, as
>> the parity is never checked against data on disk since bcache hides it.
> 
> Interesting.
> 
>> I would therefore very much like a cache_mode that would bypass any and all
>> reads, that can be enabled for the duration of a SnapRAID sync or scrub. For
>> writes I suppose this mode should act the same as "none".
>> 
>> This opportunity could be taken to verify data on cache as well; read from
>> both backing and cache and invalidate the cache page if it differs from the
>> backing data, while satisfying the actual read from backing in any case.
> 
> assuming that one or the other is correct... I'm not sure bcache could 
> tell which block is valid, and SnapRAID doesn't know about the lldevs.
> 
>> Perhaps something like this is already possible and I'm just not seeing it?
>> I know I can detach backing devices, but to my understanding that also
>> invalidates all its cached pages and I would obviously like to keep them for
>> this purpose.
> 
> Well you can only read-validate pages that are not dirty...if its dirty 
> you _must_ read from cache for consistency.
> 
> You could put it in write_around mode and wait for dirty_bytes to be 0, 
> but I think it will still read from the cache if the page is hot.

Or set the cache mode to none after dirty data to be 0, before the sync or scrub.


> 
> Detach sounds like the only option at the moment to get what you're 
> seeking.  Future work could include adding a `readaround` to 
> /sys/block/bcache0/bcache/cache_mode, but it would still have to read from 
> the cache if dirty.  Or maybe if `readaround` hits a dirty block it evicts 
> it and re-reads the backing device?  But that sounds messy and slow.


Indeed we already have a similar patch from Guoju Fang, see
	Message-Id: <1594610902-4428-1-git-send-email-fangguoju@gmail.com>

But we didn’t call it as readaround or writeonly because they are not any of them indeed. The behavior is to avoid refill cache if a read-miss happens. If this option is enabled from beginning, data will always be read from backing device expect for *dirty* data.

The reason for not having it in is naming, we don’t find a better name for the option yet.

Thanks.

Coly Li