Re: Bcache in writes direct with fsync. Are IOPS limited?

[Eric Wheeler <bcache@lists.ewheeler.net>]

[-- Attachment #1: Type: text/plain, Size: 3375 bytes --]

On Sat, 28 May 2022, colyli wrote:
> 在 2022-05-27 03:15，Eric Wheeler 写道：
> > On Mon, 23 May 2022, Coly Li wrote:
> >> On 5/18/22 9:22 AM, Eric Wheeler wrote:
> >> > Some time ago you ordered an an SSD to test the 4k cache issue, has that
> >> > been fixed?  I've kept an eye out for the patch but not sure if it was
> >> > released.
> >> 
> >> Yes, I got the Intel P3700 PCIe SSD to fix the 4Kn unaligned I/O issue
> >> (borrowed from a hardware vendor). The new situation is, current kernel
> >> does
> >> the sector size alignment checking quite earlier in bio layer, if the LBA
> >> is
> >> not sector size aligned, it is rejected in the bio code, and the underlying
> >> driver doesn't have chance to see the bio anymore. So for now, the
> >> unaligned
> >> LBA for 4Kn device cannot reach bcache code, that's to say, the original
> >> reported condition won't happen now.
> > 
> > The issue is not with unaligned 4k IOs hitting /dev/bcache0 because you
> > are right, the bio layer will reject those before even getting to
> > bcache:
> > 
> > The issue is that the bcache cache metadata sometimes makes metadata or
> > journal requests from _inside_ bcache that are not 4k aligned.  When
> > this happens the bio layer rejects the request from bcache (not from
> > whatever is above bcache).
> > 
> > Correct me if I misunderstood what you meant here, maybe it really was
> > fixed.  Here is your response from that old thread that pointed at
> > unaligned key access where you said "Wow, the above lines are very
> > informative, thanks!"
> > 
> 
> It was not fixed, at least I didn't do it on purpose. Maybe it was avoided
> by other fixes, e.g. the oversize bkey fix. But I don't have evidence the
> issue was fixed.
> 
> > bcache: check_4k_alignment() KEY_OFFSET(&w->key) is not 4KB aligned:
> > 15725385535
> >   https://www.spinics.net/lists/linux-bcache/msg06076.html
> > 
> > In that thread Kent sent a quick top-post asking "have you checked extent
> > merging?"
> >  https://www.spinics.net/lists/linux-bcache/msg06077.html
> > 
> 
> It embarrassed me that I received your informative debug information, and I
> glared very hard at the code for quite long time, but didn't have any clue
> that how such problem may happen in the extent related code.

You do great work on bcache, I appreciate everything you do.  No need to 
be embarrassed, this is just a hard bug to pin down!

> Since you reported the issue and I believe you, I will keep my eyes on the
> non-aligned 4Kn issue for bcache internal I/O. Hope someday I may have idea
> suddenly to point out where the problem is, and fix it.

You might try this for testing:

1. Format with -w 4096

2. Add some WARN_ONCE's around metadata and journal IO operations and run 
   it through your stress test to see what turns up.  The -w 4096 will 
   guarantee that all userspace IOs are 4k aligned, and then if any WARN's 
   trigger then they are suspect.  Even on 512-byte cache deployments we 
   should target 4k-aligned meta IOs hitting the SSD cache.  This would
   fix 2 things:

      a. It will guarantee that all journal/meta IOs are aligned to 4k for 
         4k cache users.

      b. Fix Adriano's performance issues since for at least his Hynix 
         SSD because 512b IOs are ~6x high latency than 4k IOs on his 
         system.

--
Eric Wheeler


> 
> Coly Li
> 
>