Re: [RFC][PATCH v2 0/5] Experiments with overlayfs filemap

[Amir Goldstein <amir73il@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2102 bytes --]

> > ...Luckily, I have implemented ./run --ov --recycle way back, otherwise we
> > would have had very poor test coverage for persistency of overlayfs writes.
> >
> > So we currently have 6 xfstests failing on write persistency:
> > overlay/018 overlay/032 overlay/044 overlay/047 overlay/050 overlay/051
> >
> > and 2 unionmount tests (over /base xfs):
> > ./run --ov --samefs --recycle rename-new-pop-dir
> > ./run --ov --samefs --recycle rename-mass-dir
>
> Great progress.  Thanks!
>
> I don't think you need to mess with dcache flushing, since this is all
> kernel (page cache) pages without any possible dcache aliasing issues
> that plague user pages.  So a simple copy_page() should do.
>

Pardon my ignorance, but couldn't pages be mapped to user?
Anyway, it seems that generic_file_buffered_read() does
flush_dcache_page() if needed, so we need not worry about it.

> It would be even better to do it with O_DIRECT to remove cache
> duplication.  Something like the attached...
>

Yes, definitely better. Thanks for the patch!

You'll be happy to know that the two unionmount tests fails on tmpfs
as expected (or maybe you already checked that).

I do get one new lockdep splat from xfstest, something we need to
look into (attached 013.dmesg).

One questions about your patch:
Skipping the ovl_change_flags() for upper realfile, that's just a
quick hack, right?
I'll need to fix this up properly.

About the way forward, implementing writepage() should be quite straight forward
from this, so I will add write support re-work the series and re-post.

I think I can now see how to implement the LOWER -> LOWER_SHARED ->
UPPER_SHARED transitions, but that will require some more juggling of
realfile flags etc.

With the current implementation of UPPER_SHARED only we can gain:
- No copy up latency on open() with metacopy=on
- Very improved ovl_sync_fs() (no longer sync all containers on syncfs(2))

Do you think we should aim for merging just UPPER_SHARED in dev
cycle to hash out the bugs of overlay page cache or aim for merging
LOWER_SHARED+UPPER_SHARED when it is ready?

Thanks,
Amir.

[-- Attachment #2: 013.dmesg --]
[-- Type: application/octet-stream, Size: 9576 bytes --]

[  119.448268] run fstests overlay/013 at 2019-01-25 10:55:14
[  119.755174] XFS (vde): Mounting V5 Filesystem
[  119.774400] XFS (vde): Ending clean mount
[  119.925944] XFS (vdf): Unmounting Filesystem
[  119.981734] XFS (vdf): Mounting V5 Filesystem
[  120.008546] XFS (vdf): Ending clean mount

[  120.133970] ======================================================
[  120.135229] WARNING: possible circular locking dependency detected
[  120.136520] 5.0.0-rc1-xfstests-00011-g8c152042ba8e-dirty #4047 Not tainted
[  120.137917] ------------------------------------------------------
[  120.139182] test_upper/9289 is trying to acquire lock:
[  120.140250] 00000000c03c6afd (&inode->i_rwsem){++++}, at: xfs_ilock+0x18b/0x208
[  120.141745] 
               but task is already holding lock:
[  120.142929] 000000002e44e1ff (&mm->mmap_sem){++++}, at: __do_page_fault+0x1b4/0x3ff
[  120.144388] 
               which lock already depends on the new lock.

[  120.146072] 
               the existing dependency chain (in reverse order) is:
[  120.147652] 
               -> #1 (&mm->mmap_sem){++++}:
[  120.148847]        __might_fault+0x61/0x89
[  120.149727]        copy_page_to_iter+0x11f/0x2c1
[  120.150474]        generic_file_buffered_read+0x279/0x660
[  120.151544]        xfs_file_buffered_aio_read+0x117/0x135
[  120.152557]        xfs_file_read_iter+0x81/0xc8
[  120.153538]        __vfs_read+0x136/0x16e
[  120.154359]        vfs_read+0xb5/0x141
[  120.155137]        ksys_read+0x60/0xb1
[  120.155920]        do_syscall_64+0x57/0x14e
[  120.156783]        entry_SYSCALL_64_after_hwframe+0x49/0xbe
[  120.157920] 
               -> #0 (&inode->i_rwsem){++++}:
[  120.159055]        lock_acquire+0x141/0x16c
[  120.159921]        down_read_nested+0x45/0x9c
[  120.160818]        xfs_ilock+0x18b/0x208
[  120.161522]        xfs_file_dio_aio_read+0x110/0x145
[  120.162489]        xfs_file_read_iter+0x7a/0xc8
[  120.163380]        do_iter_readv_writev+0x14b/0x182
[  120.164203]        do_iter_read+0x87/0xfb
[  120.164763]        ovl_do_readpage+0x9e/0xed
[  120.165341]        ovl_readpage+0xf/0x22
[  120.165915]        filemap_fault+0x311/0x425
[  120.166620]        __do_fault+0x1d/0x67
[  120.167343]        __handle_mm_fault+0x823/0xae5
[  120.168300]        __do_page_fault+0x2fb/0x3ff
[  120.169366]        async_page_fault+0x1e/0x30
[  120.170361]        __clear_user+0x36/0x5f
[  120.171305]        padzero+0x1c/0x2b
[  120.172034]        load_elf_binary+0x795/0xdfe
[  120.172870]        search_binary_handler+0x72/0xdb
[  120.173812]        __do_execve_file.isra.11+0x708/0x9b2
[  120.174837]        do_execve+0x21/0x24
[  120.175588]        __x64_sys_execve+0x26/0x2b
[  120.176451]        do_syscall_64+0x57/0x14e
[  120.177274]        entry_SYSCALL_64_after_hwframe+0x49/0xbe
[  120.178364] 
               other info that might help us debug this:

[  120.179929]  Possible unsafe locking scenario:

[  120.181086]        CPU0                    CPU1
[  120.181974]        ----                    ----
[  120.182866]   lock(&mm->mmap_sem);
[  120.183549]                                lock(&inode->i_rwsem);
[  120.184735]                                lock(&mm->mmap_sem);
[  120.185746]   lock(&inode->i_rwsem);
[  120.186235] 
                *** DEADLOCK ***

[  120.187019] 1 lock held by test_upper/9289:
[  120.187728]  #0: 000000002e44e1ff (&mm->mmap_sem){++++}, at: __do_page_fault+0x1b4/0x3ff
[  120.189372] 
               stack backtrace:
[  120.190268] CPU: 0 PID: 9289 Comm: test_upper Not tainted 5.0.0-rc1-xfstests-00011-g8c152042ba8e-dirty #4047
[  120.192277] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[  120.194139] Call Trace:
[  120.194669]  dump_stack+0x67/0x8e
[  120.195539]  print_circular_bug.isra.18+0x26e/0x27b
[  120.196585]  check_prev_add.constprop.26+0x12c/0x5f7
[  120.197499]  ? __lock_acquire+0xd80/0xee7
[  120.198259]  __lock_acquire+0xd80/0xee7
[  120.199062]  lock_acquire+0x141/0x16c
[  120.199723]  ? xfs_ilock+0x18b/0x208
[  120.200352]  ? xfs_file_dio_aio_read+0x110/0x145
[  120.201158]  down_read_nested+0x45/0x9c
[  120.201829]  ? xfs_ilock+0x18b/0x208
[  120.202456]  xfs_ilock+0x18b/0x208
[  120.203057]  xfs_file_dio_aio_read+0x110/0x145
[  120.203933]  xfs_file_read_iter+0x7a/0xc8
[  120.204729]  do_iter_readv_writev+0x14b/0x182
[  120.205514]  do_iter_read+0x87/0xfb
[  120.206018]  ovl_do_readpage+0x9e/0xed
[  120.206676]  ovl_readpage+0xf/0x22
[  120.207177]  ? ovl_write_begin+0xab/0xab
[  120.208028]  filemap_fault+0x311/0x425
[  120.208802]  ? ovl_fault+0xc9/0x16d
[  120.209494]  __do_fault+0x1d/0x67
[  120.210152]  __handle_mm_fault+0x823/0xae5
[  120.210962]  __do_page_fault+0x2fb/0x3ff
[  120.211747]  async_page_fault+0x1e/0x30
[  120.212507] RIP: 0010:__clear_user+0x36/0x5f
[  120.213202] Code: 71 20 37 82 53 48 89 f3 be 13 00 00 00 e8 a2 58 77 ff 0f 01 cb 48 89 d8 48 c1 eb 03 48 89 ef 83 e0 07 48 89 d9 48 85 c9 74 0f <48> c7 07 00 00 00 00 48 83 c7 08 ff c9 75 f1 48 89 c1 85 c9 74 0a
[  120.217055] RSP: 0018:ffffc90002f37d88 EFLAGS: 00050206
[  120.218119] RAX: 0000000000000000 RBX: 00000000000001f5 RCX: 00000000000001f5
[  120.219217] RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000563302b8b058
[  120.220578] RBP: 0000563302b8b058 R08: 00000000df1fc5d3 R09: 0000000000000006
[  120.221953] R10: 0000000000000000 R11: 0000000000000001 R12: ffff888079794900
[  120.223331] R13: 0000000000000000 R14: 0000563302b8b070 R15: 0000563302b8b058
[  120.224675]  ? __clear_user+0x1e/0x5f
[  120.225192]  padzero+0x1c/0x2b
[  120.225614]  load_elf_binary+0x795/0xdfe
[  120.226172]  search_binary_handler+0x72/0xdb
[  120.226828]  __do_execve_file.isra.11+0x708/0x9b2
[  120.227685]  do_execve+0x21/0x24
[  120.228452]  __x64_sys_execve+0x26/0x2b
[  120.229447]  do_syscall_64+0x57/0x14e
[  120.230161]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[  120.231137] RIP: 0033:0x7fc14348a647
[  120.231846] Code: Bad RIP value.
[  120.232476] RSP: 002b:00007ffc28f4b638 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[  120.233529] RAX: ffffffffffffffda RBX: 00000000013cf568 RCX: 00007fc14348a647
[  120.234450] RDX: 000000000162f008 RSI: 00000000013d4f68 RDI: 00000000013cf568
[  120.235382] RBP: 0000000000000000 R08: 0000000000710710 R09: 0000000000000000
[  120.236840] R10: 000000000000059a R11: 0000000000000202 R12: 0000000000000000
[  120.238271] R13: 00000000013d4f68 R14: 000000000162f008 R15: 0000000000000000
[  120.246570] ------------[ cut here ]------------
[  120.247604] downgrading a read lock
[  120.247618] WARNING: CPU: 1 PID: 9289 at /home/amir/build/src/linux/kernel/locking/lockdep.c:3553 lock_downgrade+0xe3/0x17b
[  120.250592] CPU: 1 PID: 9289 Comm: test_upper Not tainted 5.0.0-rc1-xfstests-00011-g8c152042ba8e-dirty #4047
[  120.252718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[  120.254537] RIP: 0010:lock_downgrade+0xe3/0x17b
[  120.255537] Code: 00 00 00 8b 44 24 04 89 83 60 08 00 00 48 8b 45 00 48 89 83 58 08 00 00 f6 45 32 03 74 0e 48 c7 c7 79 7a 28 82 e8 77 d8 fa ff <0f> 0b 8a 45 32 4c 89 6d 08 44 89 e6 48 89 df 83 e0 fc 83 c8 01 88
[  120.259903] RSP: 0018:ffffc90002f37e28 EFLAGS: 00010082
[  120.261047] RAX: 0000000000000000 RBX: ffff88807a3e02c0 RCX: 0000000000000000
[  120.262509] RDX: ffffffff810fb24f RSI: 0000000000000001 RDI: ffffffff810fb261
[  120.263887] RBP: ffff88807a3e0b28 R08: 0000000000000017 R09: 0000000000000004
[  120.265338] R10: 0000000000000000 R11: ffffc90002f37c47 R12: 0000000000000001
[  120.266666] R13: ffffffff811e4bb3 R14: 0000000000000246 R15: ffff88805ca38168
[  120.268122] FS:  00007f371215ebc0(0000) GS:ffff88807dc00000(0000) knlGS:0000000000000000
[  120.269597] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  120.270361] CR2: 00007f37115fd1b0 CR3: 000000007c470006 CR4: 00000000003606e0
[  120.271299] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  120.272776] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  120.274323] Call Trace:
[  120.274863]  downgrade_write+0x17/0x81
[  120.275698]  __do_munmap+0x26f/0x2fe
[  120.276437]  __vm_munmap+0x74/0xbf
[  120.277138]  __x64_sys_munmap+0x27/0x2c
[  120.277931]  do_syscall_64+0x57/0x14e
[  120.278697]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[  120.279738] RIP: 0033:0x7f3711f5c447
[  120.280475] Code: b4 eb 91 f7 d8 89 05 f8 bc 20 00 48 c7 c0 ff ff ff ff eb a1 f7 d8 89 05 e7 bc 20 00 e9 73 ff ff ff 66 90 b8 0b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8d 0d c9 bc 20 00 f7 d8 89 01 48 83
[  120.284277] RSP: 002b:00007ffe35bb37c8 EFLAGS: 00000202 ORIG_RAX: 000000000000000b
[  120.285603] RAX: ffffffffffffffda RBX: 0000004060eef67d RCX: 00007f3711f5c447
[  120.286542] RDX: 0000004000000000 RSI: 0000000000002739 RDI: 00007f3712160000
[  120.287592] RBP: 00007ffe35bb39e0 R08: 0000000000000000 R09: 00007f3711583b98
[  120.288936] R10: 00007f3712167030 R11: 0000000000000202 R12: 0000000000000000
[  120.289887] R13: 00007f3712168170 R14: 00007f371215ebc0 R15: 00007f3712168170
[  120.290881] irq event stamp: 1547
[  120.291374] hardirqs last  enabled at (1547): [<ffffffff81a73a36>] _raw_spin_unlock_irq+0x29/0x33
[  120.293412] hardirqs last disabled at (1546): [<ffffffff81a737ba>] _raw_spin_lock_irq+0x10/0x67
[  120.295143] softirqs last  enabled at (1182): [<ffffffff81e0035e>] __do_softirq+0x35e/0x39a
[  120.296779] softirqs last disabled at (873): [<ffffffff810a7902>] irq_exit+0x62/0xb1
[  120.298291] ---[ end trace c1aa87c93f028edf ]---
[  120.320405] XFS (vde): Unmounting Filesystem