* wip-msgr2
@ 2020-12-14 13:43 Ilya Dryomov
2020-12-14 15:55 ` wip-msgr2 Luis Henriques
2020-12-16 15:31 ` wip-msgr2 Jeff Layton
0 siblings, 2 replies; 12+ messages in thread
From: Ilya Dryomov @ 2020-12-14 13:43 UTC (permalink / raw)
To: Ceph Development
Hello,
I've pushed wip-msgr2 and opened a dummy PR in ceph-client:
https://github.com/ceph/ceph-client/pull/22
This set has been through a over a dozen krbd test suite runs with no
issues other than those with the test suite itself. The diffstat is
rather big, so I didn't want to spam the list. If someone wants it
posted, let me know. Any comments are welcome!
drivers/block/rbd.c | 8 +-
fs/ceph/mds_client.c | 106 +-
fs/ceph/mdsmap.c | 21 +-
include/linux/ceph/auth.h | 68 +-
include/linux/ceph/ceph_features.h | 11 +-
include/linux/ceph/ceph_fs.h | 11 +
include/linux/ceph/decode.h | 8 +
include/linux/ceph/libceph.h | 10 +-
include/linux/ceph/mdsmap.h | 2 +-
include/linux/ceph/messenger.h | 285 ++-
include/linux/ceph/msgr.h | 57 +-
include/linux/ceph/osdmap.h | 4 +-
net/ceph/Kconfig | 3 +
net/ceph/Makefile | 3 +-
net/ceph/auth.c | 408 ++++-
net/ceph/auth_none.c | 5 +-
net/ceph/auth_x.c | 298 +++-
net/ceph/auth_x_protocol.h | 3 +-
net/ceph/ceph_common.c | 63 +
net/ceph/ceph_strings.c | 28 +
net/ceph/crypto.h | 3 +
net/ceph/decode.c | 101 ++
net/ceph/messenger.c | 2252 +++++------------------
net/ceph/messenger_v1.c | 1506 ++++++++++++++++
net/ceph/messenger_v2.c | 3443 ++++++++++++++++++++++++++++++++
net/ceph/mon_client.c | 320 +++-
net/ceph/osd_client.c | 111 +-
net/ceph/osdmap.c | 45 +-
28 files changed, 7027 insertions(+), 2156 deletions(-)
create mode 100644 net/ceph/messenger_v1.c
create mode 100644 net/ceph/messenger_v2.c
Thanks,
Ilya
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: wip-msgr2
2020-12-14 13:43 wip-msgr2 Ilya Dryomov
@ 2020-12-14 15:55 ` Luis Henriques
2020-12-14 16:59 ` wip-msgr2 Ilya Dryomov
2020-12-16 15:31 ` wip-msgr2 Jeff Layton
1 sibling, 1 reply; 12+ messages in thread
From: Luis Henriques @ 2020-12-14 15:55 UTC (permalink / raw)
To: Ilya Dryomov; +Cc: Ceph Development
Ilya Dryomov <idryomov@gmail.com> writes:
> Hello,
>
> I've pushed wip-msgr2 and opened a dummy PR in ceph-client:
>
> https://github.com/ceph/ceph-client/pull/22
>
> This set has been through a over a dozen krbd test suite runs with no
> issues other than those with the test suite itself. The diffstat is
> rather big, so I didn't want to spam the list. If someone wants it
> posted, let me know. Any comments are welcome!
That's *awesome*! Thanks for sharing, Ilya. Obviously this will need a
lot of time to digest but a quick attempt to do a mount using a v2 monitor
is just showing me a bunch of:
libceph: mon0 (1)192.168.155.1:40898 socket closed (con state V1_BANNER)
Note that this was just me giving it a try with a dummy vstart cluster
(octopus IIRC), so nothing that could be considered testing. I'll try to
find out what I'm doing wrong in the next couple of days or, worst case,
after EOY vacations.
Cheers,
--
Luis
>
> drivers/block/rbd.c | 8 +-
> fs/ceph/mds_client.c | 106 +-
> fs/ceph/mdsmap.c | 21 +-
> include/linux/ceph/auth.h | 68 +-
> include/linux/ceph/ceph_features.h | 11 +-
> include/linux/ceph/ceph_fs.h | 11 +
> include/linux/ceph/decode.h | 8 +
> include/linux/ceph/libceph.h | 10 +-
> include/linux/ceph/mdsmap.h | 2 +-
> include/linux/ceph/messenger.h | 285 ++-
> include/linux/ceph/msgr.h | 57 +-
> include/linux/ceph/osdmap.h | 4 +-
> net/ceph/Kconfig | 3 +
> net/ceph/Makefile | 3 +-
> net/ceph/auth.c | 408 ++++-
> net/ceph/auth_none.c | 5 +-
> net/ceph/auth_x.c | 298 +++-
> net/ceph/auth_x_protocol.h | 3 +-
> net/ceph/ceph_common.c | 63 +
> net/ceph/ceph_strings.c | 28 +
> net/ceph/crypto.h | 3 +
> net/ceph/decode.c | 101 ++
> net/ceph/messenger.c | 2252 +++++------------------
> net/ceph/messenger_v1.c | 1506 ++++++++++++++++
> net/ceph/messenger_v2.c | 3443 ++++++++++++++++++++++++++++++++
> net/ceph/mon_client.c | 320 +++-
> net/ceph/osd_client.c | 111 +-
> net/ceph/osdmap.c | 45 +-
> 28 files changed, 7027 insertions(+), 2156 deletions(-)
> create mode 100644 net/ceph/messenger_v1.c
> create mode 100644 net/ceph/messenger_v2.c
>
> Thanks,
>
> Ilya
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: wip-msgr2
2020-12-14 15:55 ` wip-msgr2 Luis Henriques
@ 2020-12-14 16:59 ` Ilya Dryomov
2020-12-14 17:26 ` wip-msgr2 Luis Henriques
0 siblings, 1 reply; 12+ messages in thread
From: Ilya Dryomov @ 2020-12-14 16:59 UTC (permalink / raw)
To: Luis Henriques; +Cc: Ceph Development
On Mon, Dec 14, 2020 at 4:55 PM Luis Henriques <lhenriques@suse.de> wrote:
>
> Ilya Dryomov <idryomov@gmail.com> writes:
>
> > Hello,
> >
> > I've pushed wip-msgr2 and opened a dummy PR in ceph-client:
> >
> > https://github.com/ceph/ceph-client/pull/22
> >
> > This set has been through a over a dozen krbd test suite runs with no
> > issues other than those with the test suite itself. The diffstat is
> > rather big, so I didn't want to spam the list. If someone wants it
> > posted, let me know. Any comments are welcome!
>
> That's *awesome*! Thanks for sharing, Ilya. Obviously this will need a
> lot of time to digest but a quick attempt to do a mount using a v2 monitor
> is just showing me a bunch of:
>
> libceph: mon0 (1)192.168.155.1:40898 socket closed (con state V1_BANNER)
>
> Note that this was just me giving it a try with a dummy vstart cluster
> (octopus IIRC), so nothing that could be considered testing. I'll try to
> find out what I'm doing wrong in the next couple of days or, worst case,
> after EOY vacations.
Hi Luis,
This is because the kernel continues to default to msgr1. The socket
gets closed by the mon right after it sees msgr1 banner and you should
see "peer ... is using msgr V1 protocol" error in the log.
For msgr2, you need to select a connection mode using the new ms_mode
option:
ms_mode=legacy - msgr1 (default)
ms_mode=crc - crc mode, if denied fail
ms_mode=secure - secure mode, if denied fail
ms_mode=prefer-crc - crc mode, if denied agree to secure mode
ms_mode=prefer-secure - secure mode, if denied agree to crc mode
Thanks,
Ilya
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: wip-msgr2
2020-12-14 16:59 ` wip-msgr2 Ilya Dryomov
@ 2020-12-14 17:26 ` Luis Henriques
2020-12-15 13:14 ` wip-msgr2 Luis Henriques
0 siblings, 1 reply; 12+ messages in thread
From: Luis Henriques @ 2020-12-14 17:26 UTC (permalink / raw)
To: Ilya Dryomov; +Cc: Ceph Development
Ilya Dryomov <idryomov@gmail.com> writes:
> On Mon, Dec 14, 2020 at 4:55 PM Luis Henriques <lhenriques@suse.de> wrote:
>>
>> Ilya Dryomov <idryomov@gmail.com> writes:
>>
>> > Hello,
>> >
>> > I've pushed wip-msgr2 and opened a dummy PR in ceph-client:
>> >
>> > https://github.com/ceph/ceph-client/pull/22
>> >
>> > This set has been through a over a dozen krbd test suite runs with no
>> > issues other than those with the test suite itself. The diffstat is
>> > rather big, so I didn't want to spam the list. If someone wants it
>> > posted, let me know. Any comments are welcome!
>>
>> That's *awesome*! Thanks for sharing, Ilya. Obviously this will need a
>> lot of time to digest but a quick attempt to do a mount using a v2 monitor
>> is just showing me a bunch of:
>>
>> libceph: mon0 (1)192.168.155.1:40898 socket closed (con state V1_BANNER)
>>
>> Note that this was just me giving it a try with a dummy vstart cluster
>> (octopus IIRC), so nothing that could be considered testing. I'll try to
>> find out what I'm doing wrong in the next couple of days or, worst case,
>> after EOY vacations.
>
> Hi Luis,
>
> This is because the kernel continues to default to msgr1. The socket
> gets closed by the mon right after it sees msgr1 banner and you should
> see "peer ... is using msgr V1 protocol" error in the log.
>
> For msgr2, you need to select a connection mode using the new ms_mode
> option:
>
> ms_mode=legacy - msgr1 (default)
> ms_mode=crc - crc mode, if denied fail
> ms_mode=secure - secure mode, if denied fail
> ms_mode=prefer-crc - crc mode, if denied agree to secure mode
> ms_mode=prefer-secure - secure mode, if denied agree to crc mode
Ah, right. I should have took a quick look at the patches first to check
for any new parameters. Thanks for pointing me at that, I'll retry my
test using ms_mode.
Cheers,
--
Luis
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: wip-msgr2
2020-12-14 17:26 ` wip-msgr2 Luis Henriques
@ 2020-12-15 13:14 ` Luis Henriques
2020-12-15 15:30 ` wip-msgr2 Ilya Dryomov
0 siblings, 1 reply; 12+ messages in thread
From: Luis Henriques @ 2020-12-15 13:14 UTC (permalink / raw)
To: Ilya Dryomov; +Cc: Ceph Development
Luis Henriques <lhenriques@suse.de> writes:
> Ilya Dryomov <idryomov@gmail.com> writes:
>
>> On Mon, Dec 14, 2020 at 4:55 PM Luis Henriques <lhenriques@suse.de> wrote:
>>>
>>> Ilya Dryomov <idryomov@gmail.com> writes:
>>>
>>> > Hello,
>>> >
>>> > I've pushed wip-msgr2 and opened a dummy PR in ceph-client:
>>> >
>>> > https://github.com/ceph/ceph-client/pull/22
>>> >
>>> > This set has been through a over a dozen krbd test suite runs with no
>>> > issues other than those with the test suite itself. The diffstat is
>>> > rather big, so I didn't want to spam the list. If someone wants it
>>> > posted, let me know. Any comments are welcome!
>>>
>>> That's *awesome*! Thanks for sharing, Ilya. Obviously this will need a
>>> lot of time to digest but a quick attempt to do a mount using a v2 monitor
>>> is just showing me a bunch of:
>>>
>>> libceph: mon0 (1)192.168.155.1:40898 socket closed (con state V1_BANNER)
>>>
>>> Note that this was just me giving it a try with a dummy vstart cluster
>>> (octopus IIRC), so nothing that could be considered testing. I'll try to
>>> find out what I'm doing wrong in the next couple of days or, worst case,
>>> after EOY vacations.
>>
>> Hi Luis,
>>
>> This is because the kernel continues to default to msgr1. The socket
>> gets closed by the mon right after it sees msgr1 banner and you should
>> see "peer ... is using msgr V1 protocol" error in the log.
>>
>> For msgr2, you need to select a connection mode using the new ms_mode
>> option:
>>
>> ms_mode=legacy - msgr1 (default)
>> ms_mode=crc - crc mode, if denied fail
>> ms_mode=secure - secure mode, if denied fail
>> ms_mode=prefer-crc - crc mode, if denied agree to secure mode
>> ms_mode=prefer-secure - secure mode, if denied agree to crc mode
>
> Ah, right. I should have took a quick look at the patches first to check
> for any new parameters. Thanks for pointing me at that, I'll retry my
> test using ms_mode.
Maybe you're already aware of this, but here it goes: I'm seeing the
following BUG when doing a mount using ms_mode=secure:
[ 159.648716] ==================================================================
[ 159.652328] BUG: KASAN: slab-out-of-bounds in prepare_head_secure_small+0x101/0x110
[ 159.656203] Write of size 16 at addr ffff8881014eb5c0 by task kworker/0:1/12
[ 159.659764]
[ 159.660600] CPU: 0 PID: 12 Comm: kworker/0:1 Not tainted 5.10.0-rc2+ #55
[ 159.663985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014
[ 159.669423] Workqueue: ceph-msgr ceph_con_workfn
[ 159.671559] Call Trace:
[ 159.672657] dump_stack+0x9a/0xcc
[ 159.674115] ? prepare_head_secure_small+0x101/0x110
[ 159.676127] print_address_description.constprop.0+0x1c/0x220
[ 159.678455] ? prepare_head_secure_small+0x101/0x110
[ 159.680334] ? prepare_head_secure_small+0x101/0x110
[ 159.682245] kasan_report.cold+0x1f/0x37
[ 159.683730] ? queue_zeros+0x141/0x1c0
[ 159.685069] ? prepare_head_secure_small+0x101/0x110
[ 159.686849] check_memory_region+0x145/0x1a0
[ 159.688306] memset+0x20/0x40
[ 159.689318] prepare_head_secure_small+0x101/0x110
[ 159.691034] ? gcm_crypt+0x190/0x190
[ 159.692266] ? memset+0x20/0x40
[ 159.693349] ? encode_preamble+0xc0/0xd0
[ 159.694591] __prepare_control+0x3f1/0x450
[ 159.695463] ? prepare_read_preamble+0x190/0x190
[ 159.696450] ? kasan_unpoison_shadow+0x33/0x40
[ 159.697439] ? ceph_kvmalloc+0x7e/0x110
[ 159.698369] __handle_control+0x1115/0x2120
[ 159.699294] ? process_hello+0x510/0x510
[ 159.700130] ? lock_chain_count+0x20/0x20
[ 159.700995] ? lock_acquire+0x1b1/0x580
[ 159.701780] ? create_object.isra.0+0x239/0x4e0
[ 159.702776] ? lockdep_enabled+0x39/0x50
[ 159.703667] ? find_held_lock+0x85/0xa0
[ 159.704448] ? __kmalloc+0x16d/0x330
[ 159.705231] ? mark_held_locks+0x65/0x90
[ 159.706020] ? lockdep_enabled+0x39/0x50
[ 159.706856] ? ceph_kvmalloc+0x7e/0x110
[ 159.707588] ceph_con_v2_try_read+0x13dc/0x1e00
[ 159.708456] ? populate_out_iter+0x14b0/0x14b0
[ 159.709306] ? __mutex_lock+0x2e9/0xb70
[ 159.710000] ? trace_hardirqs_on+0x3e/0x110
[ 159.710770] ? ceph_con_workfn+0x41/0xb00
[ 159.711503] ? lock_acquire+0x1b1/0x580
[ 159.712187] ? process_one_work+0x4b2/0xaa0
[ 159.712935] ? lock_release+0x410/0x410
[ 159.713617] ? lock_downgrade+0x380/0x380
[ 159.714353] ceph_con_workfn+0x2df/0xb00
[ 159.715048] process_one_work+0x599/0xaa0
[ 159.715729] ? pwq_dec_nr_in_flight+0x110/0x110
[ 159.716489] ? rwlock_bug.part.0+0x60/0x60
[ 159.717187] worker_thread+0x2bc/0x780
[ 159.717832] ? process_one_work+0xaa0/0xaa0
[ 159.718542] kthread+0x1de/0x200
[ 159.719087] ? kthread_create_worker_on_cpu+0xd0/0xd0
[ 159.719892] ret_from_fork+0x22/0x30
[ 159.720488]
[ 159.720744] Allocated by task 12:
[ 159.721285] kasan_save_stack+0x1b/0x40
[ 159.721901] __kasan_kmalloc.constprop.0+0xc2/0xd0
[ 159.722678] ceph_kvmalloc+0x68/0x110
[ 159.723249] alloc_conn_buf+0x42/0xd0
[ 159.723817] __handle_control+0x10bb/0x2120
[ 159.724456] ceph_con_v2_try_read+0x13dc/0x1e00
[ 159.725149] ceph_con_workfn+0x2df/0xb00
[ 159.725755] process_one_work+0x599/0xaa0
[ 159.726382] worker_thread+0x2bc/0x780
[ 159.726957] kthread+0x1de/0x200
[ 159.727445] ret_from_fork+0x22/0x30
[ 159.727976]
[ 159.728216] The buggy address belongs to the object at ffff8881014eb580
[ 159.728216] which belongs to the cache kmalloc-96 of size 96
[ 159.730011] The buggy address is located 64 bytes inside of
[ 159.730011] 96-byte region [ffff8881014eb580, ffff8881014eb5e0)
[ 159.731648] The buggy address belongs to the page:
[ 159.732333] page:00000000892842f3 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x1014eb
[ 159.733656] flags: 0x8000000000000200(slab)
[ 159.734264] raw: 8000000000000200 ffffea000415eac0 0000001900000019 ffff888100041780
[ 159.735355] raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000
[ 159.736437] page dumped because: kasan: bad access detected
[ 159.737229]
[ 159.737454] Memory state around the buggy address:
[ 159.738130] ffff8881014eb480: 00 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc
[ 159.739533] ffff8881014eb500: 00 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc
[ 159.740898] >ffff8881014eb580: 00 00 00 00 00 00 00 00 04 fc fc fc fc fc fc fc
[ 159.742221] ^
[ 159.743035] ffff8881014eb600: fb fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc
[ 159.744053] ffff8881014eb680: fb fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc
[ 159.745091] ==================================================================
Looks like the memset in prepare_head_secure_small() is cleaning behind
the base limits. Unfortunately, I didn't really had time to dig deeper
into this.
Cheers,
--
Luis
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: wip-msgr2
2020-12-15 13:14 ` wip-msgr2 Luis Henriques
@ 2020-12-15 15:30 ` Ilya Dryomov
2020-12-15 16:14 ` wip-msgr2 Luis Henriques
2020-12-17 16:46 ` wip-msgr2 Luis Henriques
0 siblings, 2 replies; 12+ messages in thread
From: Ilya Dryomov @ 2020-12-15 15:30 UTC (permalink / raw)
To: Luis Henriques; +Cc: Ceph Development
On Tue, Dec 15, 2020 at 2:14 PM Luis Henriques <lhenriques@suse.de> wrote:
>
> Luis Henriques <lhenriques@suse.de> writes:
>
> > Ilya Dryomov <idryomov@gmail.com> writes:
> >
> >> On Mon, Dec 14, 2020 at 4:55 PM Luis Henriques <lhenriques@suse.de> wrote:
> >>>
> >>> Ilya Dryomov <idryomov@gmail.com> writes:
> >>>
> >>> > Hello,
> >>> >
> >>> > I've pushed wip-msgr2 and opened a dummy PR in ceph-client:
> >>> >
> >>> > https://github.com/ceph/ceph-client/pull/22
> >>> >
> >>> > This set has been through a over a dozen krbd test suite runs with no
> >>> > issues other than those with the test suite itself. The diffstat is
> >>> > rather big, so I didn't want to spam the list. If someone wants it
> >>> > posted, let me know. Any comments are welcome!
> >>>
> >>> That's *awesome*! Thanks for sharing, Ilya. Obviously this will need a
> >>> lot of time to digest but a quick attempt to do a mount using a v2 monitor
> >>> is just showing me a bunch of:
> >>>
> >>> libceph: mon0 (1)192.168.155.1:40898 socket closed (con state V1_BANNER)
> >>>
> >>> Note that this was just me giving it a try with a dummy vstart cluster
> >>> (octopus IIRC), so nothing that could be considered testing. I'll try to
> >>> find out what I'm doing wrong in the next couple of days or, worst case,
> >>> after EOY vacations.
> >>
> >> Hi Luis,
> >>
> >> This is because the kernel continues to default to msgr1. The socket
> >> gets closed by the mon right after it sees msgr1 banner and you should
> >> see "peer ... is using msgr V1 protocol" error in the log.
> >>
> >> For msgr2, you need to select a connection mode using the new ms_mode
> >> option:
> >>
> >> ms_mode=legacy - msgr1 (default)
> >> ms_mode=crc - crc mode, if denied fail
> >> ms_mode=secure - secure mode, if denied fail
> >> ms_mode=prefer-crc - crc mode, if denied agree to secure mode
> >> ms_mode=prefer-secure - secure mode, if denied agree to crc mode
> >
> > Ah, right. I should have took a quick look at the patches first to check
> > for any new parameters. Thanks for pointing me at that, I'll retry my
> > test using ms_mode.
>
> Maybe you're already aware of this, but here it goes: I'm seeing the
> following BUG when doing a mount using ms_mode=secure:
>
> [ 159.648716] ==================================================================
> [ 159.652328] BUG: KASAN: slab-out-of-bounds in prepare_head_secure_small+0x101/0x110
> [ 159.656203] Write of size 16 at addr ffff8881014eb5c0 by task kworker/0:1/12
> [ 159.659764]
> [ 159.660600] CPU: 0 PID: 12 Comm: kworker/0:1 Not tainted 5.10.0-rc2+ #55
> [ 159.663985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014
> [ 159.669423] Workqueue: ceph-msgr ceph_con_workfn
> [ 159.671559] Call Trace:
> [ 159.672657] dump_stack+0x9a/0xcc
> [ 159.674115] ? prepare_head_secure_small+0x101/0x110
> [ 159.676127] print_address_description.constprop.0+0x1c/0x220
> [ 159.678455] ? prepare_head_secure_small+0x101/0x110
> [ 159.680334] ? prepare_head_secure_small+0x101/0x110
> [ 159.682245] kasan_report.cold+0x1f/0x37
> [ 159.683730] ? queue_zeros+0x141/0x1c0
> [ 159.685069] ? prepare_head_secure_small+0x101/0x110
> [ 159.686849] check_memory_region+0x145/0x1a0
> [ 159.688306] memset+0x20/0x40
> [ 159.689318] prepare_head_secure_small+0x101/0x110
> [ 159.691034] ? gcm_crypt+0x190/0x190
> [ 159.692266] ? memset+0x20/0x40
> [ 159.693349] ? encode_preamble+0xc0/0xd0
> [ 159.694591] __prepare_control+0x3f1/0x450
> [ 159.695463] ? prepare_read_preamble+0x190/0x190
> [ 159.696450] ? kasan_unpoison_shadow+0x33/0x40
> [ 159.697439] ? ceph_kvmalloc+0x7e/0x110
> [ 159.698369] __handle_control+0x1115/0x2120
> [ 159.699294] ? process_hello+0x510/0x510
> [ 159.700130] ? lock_chain_count+0x20/0x20
> [ 159.700995] ? lock_acquire+0x1b1/0x580
> [ 159.701780] ? create_object.isra.0+0x239/0x4e0
> [ 159.702776] ? lockdep_enabled+0x39/0x50
> [ 159.703667] ? find_held_lock+0x85/0xa0
> [ 159.704448] ? __kmalloc+0x16d/0x330
> [ 159.705231] ? mark_held_locks+0x65/0x90
> [ 159.706020] ? lockdep_enabled+0x39/0x50
> [ 159.706856] ? ceph_kvmalloc+0x7e/0x110
> [ 159.707588] ceph_con_v2_try_read+0x13dc/0x1e00
> [ 159.708456] ? populate_out_iter+0x14b0/0x14b0
> [ 159.709306] ? __mutex_lock+0x2e9/0xb70
> [ 159.710000] ? trace_hardirqs_on+0x3e/0x110
> [ 159.710770] ? ceph_con_workfn+0x41/0xb00
> [ 159.711503] ? lock_acquire+0x1b1/0x580
> [ 159.712187] ? process_one_work+0x4b2/0xaa0
> [ 159.712935] ? lock_release+0x410/0x410
> [ 159.713617] ? lock_downgrade+0x380/0x380
> [ 159.714353] ceph_con_workfn+0x2df/0xb00
> [ 159.715048] process_one_work+0x599/0xaa0
> [ 159.715729] ? pwq_dec_nr_in_flight+0x110/0x110
> [ 159.716489] ? rwlock_bug.part.0+0x60/0x60
> [ 159.717187] worker_thread+0x2bc/0x780
> [ 159.717832] ? process_one_work+0xaa0/0xaa0
> [ 159.718542] kthread+0x1de/0x200
> [ 159.719087] ? kthread_create_worker_on_cpu+0xd0/0xd0
> [ 159.719892] ret_from_fork+0x22/0x30
> [ 159.720488]
> [ 159.720744] Allocated by task 12:
> [ 159.721285] kasan_save_stack+0x1b/0x40
> [ 159.721901] __kasan_kmalloc.constprop.0+0xc2/0xd0
> [ 159.722678] ceph_kvmalloc+0x68/0x110
> [ 159.723249] alloc_conn_buf+0x42/0xd0
> [ 159.723817] __handle_control+0x10bb/0x2120
> [ 159.724456] ceph_con_v2_try_read+0x13dc/0x1e00
> [ 159.725149] ceph_con_workfn+0x2df/0xb00
> [ 159.725755] process_one_work+0x599/0xaa0
> [ 159.726382] worker_thread+0x2bc/0x780
> [ 159.726957] kthread+0x1de/0x200
> [ 159.727445] ret_from_fork+0x22/0x30
> [ 159.727976]
> [ 159.728216] The buggy address belongs to the object at ffff8881014eb580
> [ 159.728216] which belongs to the cache kmalloc-96 of size 96
> [ 159.730011] The buggy address is located 64 bytes inside of
> [ 159.730011] 96-byte region [ffff8881014eb580, ffff8881014eb5e0)
> [ 159.731648] The buggy address belongs to the page:
> [ 159.732333] page:00000000892842f3 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x1014eb
> [ 159.733656] flags: 0x8000000000000200(slab)
> [ 159.734264] raw: 8000000000000200 ffffea000415eac0 0000001900000019 ffff888100041780
> [ 159.735355] raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000
> [ 159.736437] page dumped because: kasan: bad access detected
> [ 159.737229]
> [ 159.737454] Memory state around the buggy address:
> [ 159.738130] ffff8881014eb480: 00 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc
> [ 159.739533] ffff8881014eb500: 00 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc
> [ 159.740898] >ffff8881014eb580: 00 00 00 00 00 00 00 00 04 fc fc fc fc fc fc fc
> [ 159.742221] ^
> [ 159.743035] ffff8881014eb600: fb fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc
> [ 159.744053] ffff8881014eb680: fb fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc
> [ 159.745091] ==================================================================
>
> Looks like the memset in prepare_head_secure_small() is cleaning behind
> the base limits. Unfortunately, I didn't really had time to dig deeper
> into this.
Ah, I disabled KASAN for some performance testing and didn't turn
it back on. This doesn't actually corrupt any memory because the
96-byte object that gets allocated is big enough. In fact, the
relevant code used to request 96 bytes independent of the connection
mode until I changed it to follow the on-wire format more strictly.
This frame is 68 bytes in plane mode and 96 bytes in secure mode
but we are requesting 68 bytes in both modes. The following should
fix it:
diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c
index 5e38c847317b..11fd47b36fc8 100644
--- a/net/ceph/messenger_v2.c
+++ b/net/ceph/messenger_v2.c
@@ -1333,7 +1333,8 @@ static int prepare_auth_signature(struct
ceph_connection *con)
void *buf;
int ret;
- buf = alloc_conn_buf(con, head_onwire_len(SHA256_DIGEST_SIZE, false));
+ buf = alloc_conn_buf(con, head_onwire_len(SHA256_DIGEST_SIZE,
+ con_secure(con)));
if (!buf)
return -ENOMEM;
Thanks,
Ilya
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: wip-msgr2
2020-12-15 15:30 ` wip-msgr2 Ilya Dryomov
@ 2020-12-15 16:14 ` Luis Henriques
2020-12-17 16:46 ` wip-msgr2 Luis Henriques
1 sibling, 0 replies; 12+ messages in thread
From: Luis Henriques @ 2020-12-15 16:14 UTC (permalink / raw)
To: Ilya Dryomov; +Cc: Ceph Development
Ilya Dryomov <idryomov@gmail.com> writes:
<snip>
>> Looks like the memset in prepare_head_secure_small() is cleaning behind
>> the base limits. Unfortunately, I didn't really had time to dig deeper
>> into this.
>
> Ah, I disabled KASAN for some performance testing and didn't turn
> it back on. This doesn't actually corrupt any memory because the
> 96-byte object that gets allocated is big enough. In fact, the
> relevant code used to request 96 bytes independent of the connection
> mode until I changed it to follow the on-wire format more strictly.
>
> This frame is 68 bytes in plane mode and 96 bytes in secure mode
> but we are requesting 68 bytes in both modes. The following should
> fix it:
Yep, it does fix it. Thanks for the quick reply.
Cheers,
--
Luis
>
> diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c
> index 5e38c847317b..11fd47b36fc8 100644
> --- a/net/ceph/messenger_v2.c
> +++ b/net/ceph/messenger_v2.c
> @@ -1333,7 +1333,8 @@ static int prepare_auth_signature(struct
> ceph_connection *con)
> void *buf;
> int ret;
>
> - buf = alloc_conn_buf(con, head_onwire_len(SHA256_DIGEST_SIZE, false));
> + buf = alloc_conn_buf(con, head_onwire_len(SHA256_DIGEST_SIZE,
> + con_secure(con)));
> if (!buf)
> return -ENOMEM;
>
> Thanks,
>
> Ilya
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: wip-msgr2
2020-12-14 13:43 wip-msgr2 Ilya Dryomov
2020-12-14 15:55 ` wip-msgr2 Luis Henriques
@ 2020-12-16 15:31 ` Jeff Layton
2020-12-16 17:18 ` wip-msgr2 Ilya Dryomov
1 sibling, 1 reply; 12+ messages in thread
From: Jeff Layton @ 2020-12-16 15:31 UTC (permalink / raw)
To: Ilya Dryomov, Ceph Development
On Mon, 2020-12-14 at 14:43 +0100, Ilya Dryomov wrote:
> Hello,
>
> I've pushed wip-msgr2 and opened a dummy PR in ceph-client:
>
> https://github.com/ceph/ceph-client/pull/22
>
> This set has been through a over a dozen krbd test suite runs with no
> issues other than those with the test suite itself. The diffstat is
> rather big, so I didn't want to spam the list. If someone wants it
> posted, let me know. Any comments are welcome!
>
> drivers/block/rbd.c | 8 +-
> fs/ceph/mds_client.c | 106 +-
> fs/ceph/mdsmap.c | 21 +-
> include/linux/ceph/auth.h | 68 +-
> include/linux/ceph/ceph_features.h | 11 +-
> include/linux/ceph/ceph_fs.h | 11 +
> include/linux/ceph/decode.h | 8 +
> include/linux/ceph/libceph.h | 10 +-
> include/linux/ceph/mdsmap.h | 2 +-
> include/linux/ceph/messenger.h | 285 ++-
> include/linux/ceph/msgr.h | 57 +-
> include/linux/ceph/osdmap.h | 4 +-
> net/ceph/Kconfig | 3 +
> net/ceph/Makefile | 3 +-
> net/ceph/auth.c | 408 ++++-
> net/ceph/auth_none.c | 5 +-
> net/ceph/auth_x.c | 298 +++-
> net/ceph/auth_x_protocol.h | 3 +-
> net/ceph/ceph_common.c | 63 +
> net/ceph/ceph_strings.c | 28 +
> net/ceph/crypto.h | 3 +
> net/ceph/decode.c | 101 ++
> net/ceph/messenger.c | 2252 +++++------------------
> net/ceph/messenger_v1.c | 1506 ++++++++++++++++
> net/ceph/messenger_v2.c | 3443 ++++++++++++++++++++++++++++++++
> net/ceph/mon_client.c | 320 +++-
> net/ceph/osd_client.c | 111 +-
> net/ceph/osdmap.c | 45 +-
> 28 files changed, 7027 insertions(+), 2156 deletions(-)
> create mode 100644 net/ceph/messenger_v1.c
> create mode 100644 net/ceph/messenger_v2.c
>
> Thanks,
>
> Ilya
FWIW, I see these warnings when building with this series with W=1:
./include/linux/ceph/msgr.h:37:24: warning: ‘CEPH_MSGR2_FEATUREMASK_REVISION_1’ defined but not used [-Wunused-const-variable=]
37 | static const uint64_t CEPH_MSGR2_FEATUREMASK_##name = \
| ^~~~~~~~~~~~~~~~~~~~~~~
./include/linux/ceph/msgr.h:43:1: note: in expansion of macro ‘DEFINE_MSGR2_FEATURE’
43 | DEFINE_MSGR2_FEATURE( 0, 1, REVISION_1) // msgr2.1
| ^~~~~~~~~~~~~~~~~~~~
./include/linux/ceph/msgr.h:36:24: warning: ‘CEPH_MSGR2_FEATURE_REVISION_1’ defined but not used [-Wunused-const-variable=]
36 | static const uint64_t CEPH_MSGR2_FEATURE_##name = (1ULL << bit); \
| ^~~~~~~~~~~~~~~~~~~
./include/linux/ceph/msgr.h:43:1: note: in expansion of macro ‘DEFINE_MSGR2_FEATURE’
43 | DEFINE_MSGR2_FEATURE( 0, 1, REVISION_1) // msgr2.1
| ^~~~~~~~~~~~~~~~~~~~
It's probably easy to fix, but I haven't looked in detail yet.
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: wip-msgr2
2020-12-16 15:31 ` wip-msgr2 Jeff Layton
@ 2020-12-16 17:18 ` Ilya Dryomov
0 siblings, 0 replies; 12+ messages in thread
From: Ilya Dryomov @ 2020-12-16 17:18 UTC (permalink / raw)
To: Jeff Layton; +Cc: Ceph Development
On Wed, Dec 16, 2020 at 4:31 PM Jeff Layton <jlayton@kernel.org> wrote:
>
> On Mon, 2020-12-14 at 14:43 +0100, Ilya Dryomov wrote:
> > Hello,
> >
> > I've pushed wip-msgr2 and opened a dummy PR in ceph-client:
> >
> > https://github.com/ceph/ceph-client/pull/22
> >
> > This set has been through a over a dozen krbd test suite runs with no
> > issues other than those with the test suite itself. The diffstat is
> > rather big, so I didn't want to spam the list. If someone wants it
> > posted, let me know. Any comments are welcome!
> >
> > drivers/block/rbd.c | 8 +-
> > fs/ceph/mds_client.c | 106 +-
> > fs/ceph/mdsmap.c | 21 +-
> > include/linux/ceph/auth.h | 68 +-
> > include/linux/ceph/ceph_features.h | 11 +-
> > include/linux/ceph/ceph_fs.h | 11 +
> > include/linux/ceph/decode.h | 8 +
> > include/linux/ceph/libceph.h | 10 +-
> > include/linux/ceph/mdsmap.h | 2 +-
> > include/linux/ceph/messenger.h | 285 ++-
> > include/linux/ceph/msgr.h | 57 +-
> > include/linux/ceph/osdmap.h | 4 +-
> > net/ceph/Kconfig | 3 +
> > net/ceph/Makefile | 3 +-
> > net/ceph/auth.c | 408 ++++-
> > net/ceph/auth_none.c | 5 +-
> > net/ceph/auth_x.c | 298 +++-
> > net/ceph/auth_x_protocol.h | 3 +-
> > net/ceph/ceph_common.c | 63 +
> > net/ceph/ceph_strings.c | 28 +
> > net/ceph/crypto.h | 3 +
> > net/ceph/decode.c | 101 ++
> > net/ceph/messenger.c | 2252 +++++------------------
> > net/ceph/messenger_v1.c | 1506 ++++++++++++++++
> > net/ceph/messenger_v2.c | 3443 ++++++++++++++++++++++++++++++++
> > net/ceph/mon_client.c | 320 +++-
> > net/ceph/osd_client.c | 111 +-
> > net/ceph/osdmap.c | 45 +-
> > 28 files changed, 7027 insertions(+), 2156 deletions(-)
> > create mode 100644 net/ceph/messenger_v1.c
> > create mode 100644 net/ceph/messenger_v2.c
> >
> > Thanks,
> >
> > Ilya
>
> FWIW, I see these warnings when building with this series with W=1:
>
> ./include/linux/ceph/msgr.h:37:24: warning: ‘CEPH_MSGR2_FEATUREMASK_REVISION_1’ defined but not used [-Wunused-const-variable=]
> 37 | static const uint64_t CEPH_MSGR2_FEATUREMASK_##name = \
> | ^~~~~~~~~~~~~~~~~~~~~~~
> ./include/linux/ceph/msgr.h:43:1: note: in expansion of macro ‘DEFINE_MSGR2_FEATURE’
> 43 | DEFINE_MSGR2_FEATURE( 0, 1, REVISION_1) // msgr2.1
> | ^~~~~~~~~~~~~~~~~~~~
> ./include/linux/ceph/msgr.h:36:24: warning: ‘CEPH_MSGR2_FEATURE_REVISION_1’ defined but not used [-Wunused-const-variable=]
> 36 | static const uint64_t CEPH_MSGR2_FEATURE_##name = (1ULL << bit); \
> | ^~~~~~~~~~~~~~~~~~~
> ./include/linux/ceph/msgr.h:43:1: note: in expansion of macro ‘DEFINE_MSGR2_FEATURE’
> 43 | DEFINE_MSGR2_FEATURE( 0, 1, REVISION_1) // msgr2.1
> | ^~~~~~~~~~~~~~~~~~~~
>
> It's probably easy to fix, but I haven't looked in detail yet.
Yeah, consider it fixed. (I probably need to start building with
W=1, considering that people seem to have started caring about it
these days...)
Thanks,
Ilya
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: wip-msgr2
2020-12-15 15:30 ` wip-msgr2 Ilya Dryomov
2020-12-15 16:14 ` wip-msgr2 Luis Henriques
@ 2020-12-17 16:46 ` Luis Henriques
2020-12-17 17:25 ` wip-msgr2 Ilya Dryomov
1 sibling, 1 reply; 12+ messages in thread
From: Luis Henriques @ 2020-12-17 16:46 UTC (permalink / raw)
To: Ilya Dryomov; +Cc: Ceph Development
Ilya Dryomov <idryomov@gmail.com> writes:
<snip>
>
> Ah, I disabled KASAN for some performance testing and didn't turn
> it back on. This doesn't actually corrupt any memory because the
> 96-byte object that gets allocated is big enough. In fact, the
> relevant code used to request 96 bytes independent of the connection
> mode until I changed it to follow the on-wire format more strictly.
>
> This frame is 68 bytes in plane mode and 96 bytes in secure mode
> but we are requesting 68 bytes in both modes. The following should
> fix it:
>
> diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c
> index 5e38c847317b..11fd47b36fc8 100644
> --- a/net/ceph/messenger_v2.c
> +++ b/net/ceph/messenger_v2.c
> @@ -1333,7 +1333,8 @@ static int prepare_auth_signature(struct
> ceph_connection *con)
> void *buf;
> int ret;
>
> - buf = alloc_conn_buf(con, head_onwire_len(SHA256_DIGEST_SIZE, false));
> + buf = alloc_conn_buf(con, head_onwire_len(SHA256_DIGEST_SIZE,
> + con_secure(con)));
> if (!buf)
> return -ENOMEM;
Looks like this fix didn't made it into your pull-request. Did it just
fell through the cracks, or is this fixed somewhere else on the code?
(And wow! I didn't really expect msgrv2 to hit this merge window. Nice.)
Cheers,
--
Luis
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: wip-msgr2
2020-12-17 16:46 ` wip-msgr2 Luis Henriques
@ 2020-12-17 17:25 ` Ilya Dryomov
2020-12-18 16:45 ` wip-msgr2 Luis Henriques
0 siblings, 1 reply; 12+ messages in thread
From: Ilya Dryomov @ 2020-12-17 17:25 UTC (permalink / raw)
To: Luis Henriques; +Cc: Ceph Development
On Thu, Dec 17, 2020 at 5:45 PM Luis Henriques <lhenriques@suse.de> wrote:
>
> Ilya Dryomov <idryomov@gmail.com> writes:
> <snip>
> >
> > Ah, I disabled KASAN for some performance testing and didn't turn
> > it back on. This doesn't actually corrupt any memory because the
> > 96-byte object that gets allocated is big enough. In fact, the
> > relevant code used to request 96 bytes independent of the connection
> > mode until I changed it to follow the on-wire format more strictly.
> >
> > This frame is 68 bytes in plane mode and 96 bytes in secure mode
> > but we are requesting 68 bytes in both modes. The following should
> > fix it:
> >
> > diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c
> > index 5e38c847317b..11fd47b36fc8 100644
> > --- a/net/ceph/messenger_v2.c
> > +++ b/net/ceph/messenger_v2.c
> > @@ -1333,7 +1333,8 @@ static int prepare_auth_signature(struct
> > ceph_connection *con)
> > void *buf;
> > int ret;
> >
> > - buf = alloc_conn_buf(con, head_onwire_len(SHA256_DIGEST_SIZE, false));
> > + buf = alloc_conn_buf(con, head_onwire_len(SHA256_DIGEST_SIZE,
> > + con_secure(con)));
> > if (!buf)
> > return -ENOMEM;
>
> Looks like this fix didn't made it into your pull-request. Did it just
> fell through the cracks, or is this fixed somewhere else on the code?
No, it didn't, it's in the testing branch:
https://github.com/ceph/ceph-client/commit/add7ad675cd1bdaf2751da1af9295fb43896da66
Thanks,
Ilya
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: wip-msgr2
2020-12-17 17:25 ` wip-msgr2 Ilya Dryomov
@ 2020-12-18 16:45 ` Luis Henriques
0 siblings, 0 replies; 12+ messages in thread
From: Luis Henriques @ 2020-12-18 16:45 UTC (permalink / raw)
To: Ilya Dryomov; +Cc: Ceph Development
(Ups! Looks like this email never left my drafts maildir.)
Ilya Dryomov <idryomov@gmail.com> writes:
> On Thu, Dec 17, 2020 at 5:45 PM Luis Henriques <lhenriques@suse.de> wrote:
>>
>> Ilya Dryomov <idryomov@gmail.com> writes:
>> <snip>
>> >
>> > Ah, I disabled KASAN for some performance testing and didn't turn
>> > it back on. This doesn't actually corrupt any memory because the
>> > 96-byte object that gets allocated is big enough. In fact, the
>> > relevant code used to request 96 bytes independent of the connection
>> > mode until I changed it to follow the on-wire format more strictly.
>> >
>> > This frame is 68 bytes in plane mode and 96 bytes in secure mode
>> > but we are requesting 68 bytes in both modes. The following should
>> > fix it:
>> >
>> > diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c
>> > index 5e38c847317b..11fd47b36fc8 100644
>> > --- a/net/ceph/messenger_v2.c
>> > +++ b/net/ceph/messenger_v2.c
>> > @@ -1333,7 +1333,8 @@ static int prepare_auth_signature(struct
>> > ceph_connection *con)
>> > void *buf;
>> > int ret;
>> >
>> > - buf = alloc_conn_buf(con, head_onwire_len(SHA256_DIGEST_SIZE, false));
>> > + buf = alloc_conn_buf(con, head_onwire_len(SHA256_DIGEST_SIZE,
>> > + con_secure(con)));
>> > if (!buf)
>> > return -ENOMEM;
>>
>> Looks like this fix didn't made it into your pull-request. Did it just
>> fell through the cracks, or is this fixed somewhere else on the code?
>
> No, it didn't, it's in the testing branch:
>
> https://github.com/ceph/ceph-client/commit/add7ad675cd1bdaf2751da1af9295fb43896da66
Cool, thanks for confirming. Just wanted to make sure ;-)
Cheers,
--
Luis
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2020-12-18 16:45 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-14 13:43 wip-msgr2 Ilya Dryomov
2020-12-14 15:55 ` wip-msgr2 Luis Henriques
2020-12-14 16:59 ` wip-msgr2 Ilya Dryomov
2020-12-14 17:26 ` wip-msgr2 Luis Henriques
2020-12-15 13:14 ` wip-msgr2 Luis Henriques
2020-12-15 15:30 ` wip-msgr2 Ilya Dryomov
2020-12-15 16:14 ` wip-msgr2 Luis Henriques
2020-12-17 16:46 ` wip-msgr2 Luis Henriques
2020-12-17 17:25 ` wip-msgr2 Ilya Dryomov
2020-12-18 16:45 ` wip-msgr2 Luis Henriques
2020-12-16 15:31 ` wip-msgr2 Jeff Layton
2020-12-16 17:18 ` wip-msgr2 Ilya Dryomov
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.