[QUESTION] jbd2: how metadata blocks are checkpointed by the page-cache?

[QUESTION] jbd2: how metadata blocks are checkpointed by the page-cache?

[Red Swaqz]

Hello all,

I am studying the ext4/JBD2 internals and there is one thing that is missing
in my understanding: the **Checkpoint** part for metadata blocks.

So far I understood that the JBD2 marks the metadata buffers as
_bufer_jbddirty(bh)_,
then during the commit, the buffers are shadowed, written to the
journal area, and later
they are marked as _buffer_dirty(bh)_ and left to the page-cache to
writeback them
to the original position at the disk. Later on, the JBD2 will check
its checkpoint list
and check if the _buffer_heads_ are clean, which indicates that they
were written to
the storage and the CP operation is complete, thus JBD2 can remove
those _buffer_heads_
from its control.

So, this is the part I didn't catch in the code: where/when/how the
page-cache writes
the metadata blocks exactly? So far I could understand (not 100% sure)
that the ext4
fills the _address_space_operations_ structure with the
_ext4_writepage()_, _ext4_writepages()_,
and so on for the page-cache operations. But still, it is not clear to
me when the metadata
blocks are written back to cause the CP completion.

The writeback part for data blocks is OK because it looks to me that
the page-cache uses
the _inodes_ to be able to write them to the storage (with the data
blocks). So, this is kinda
easy to figure out in the source code.

Anyone could help me with some insights into the source code or maybe theory?

Thank you all.

Re: [QUESTION] jbd2: how metadata blocks are checkpointed by the page-cache?

[Jan Kara]

Hello!

The reply is late but maybe it will be still useful.

On Mon 27-01-20 11:26:40, Red Swaqz wrote:
> I am studying the ext4/JBD2 internals and there is one thing that is missing
> in my understanding: the **Checkpoint** part for metadata blocks.
> 
> So far I understood that the JBD2 marks the metadata buffers as
> _bufer_jbddirty(bh)_, then during the commit, the buffers are shadowed,
> written to the journal area, and later they are marked as
> _buffer_dirty(bh)_ and left to the page-cache to writeback them to the
> original position at the disk. Later on, the JBD2 will check its
> checkpoint list and check if the _buffer_heads_ are clean, which
> indicates that they were written to the storage and the CP operation is
> complete, thus JBD2 can remove those _buffer_heads_ from its control.

Correct.

> So, this is the part I didn't catch in the code: where/when/how the
> page-cache writes the metadata blocks exactly? So far I could understand
> (not 100% sure) that the ext4 fills the _address_space_operations_
> structure with the _ext4_writepage()_, _ext4_writepages()_, and so on for
> the page-cache operations. But still, it is not clear to me when the
> metadata blocks are written back to cause the CP completion.

So metadata reside in "block device page cache" also sometimes called
"buffer cache" - that's page cache behind say /dev/sda. The kernel tracks
dirty block device inode like any other inodes and also schedules their
writeback using 'flush workers'. These flush workers will call ->writepage
/ ->writepages method for the inode - for block devices this is
blkdev_writepage(). And this sends dirty metadata to the disk.

Also there's another "synchronous" method how metadata can get
checkpointed. When we are running out of space in the journal, the process
starting transaction will call log_do_checkpoint() to writeback metadata
needing checkpoint.

Hope this helps.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR