All of lore.kernel.org
 help / color / mirror / Atom feed
From: yangerkun <yangerkun@huawei.com>
To: yangerkun <yangerkun@huaweicloud.com>,
	Mikulas Patocka <mpatocka@redhat.com>
Cc: dm-devel@redhat.com, Mike Snitzer <snitzer@kernel.org>,
	Bart Van Assche <bvanassche@acm.org>,
	agk@redhat.com, Milan Broz <gmazyland@gmail.com>
Subject: Re: [dm-devel] [PATCH] dm-crypt: add the "high_priority" flag
Date: Sat, 11 Mar 2023 17:26:48 +0800	[thread overview]
Message-ID: <c48b1039-0911-c7f2-dc0e-d2de985ad295@huawei.com> (raw)
In-Reply-To: <76551db7-b5f1-22e5-54c8-c68611a03b5f@huaweicloud.com>



在 2023/3/7 9:30, yangerkun 写道:
> 
> 
> 在 2023/3/7 3:01, Mikulas Patocka 写道:
>>
>>
>> On Mon, 6 Mar 2023, Mikulas Patocka wrote:
>>
>>>> Hi,
>>>>
>>>> Thanks a lot for your review!
>>>>
>>>> It's ok to fix the softlockup, but for async write encrypt,
>>>> kcryptd_crypt_write_io_submit will add bio to write_tree, and once we
>>>> call cond_resched before every kcryptd_io_write, the write performance
>>>> may be poor while we meet a high cpu usage scene.
>>>
>>> Hi
>>>
>>> To fix this problem, find the PID of the process "dmcrypt_write" and
>>> change its priority to -20, for example "renice -n -20 -p 34748".
>>>
>>> This is the proper way how to fix it; locking up the process for one
>>> second is not.
>>>
>>> We used to have high-priority workqueues by default, but it caused audio
>>> playback skipping, so we had to revert it - see
>>> f612b2132db529feac4f965f28a1b9258ea7c22b.
>>>
>>> Perhaps we should add an option to have high-priority kernel threads?
>>>
>>> Mikulas
>>
>> Here I'm sending a patch that makes it optional to raise the priority
>> of dm-crypt workqueues and the write thread. Test it, if it helps for 
>> you.
> 
> Thanks, will test it!

Sorry for the delay, the scene that report the softlockup cannot been 
using(arm64 server), instead, we use a x86_64 server do the test.

Test step:
1. make every cpu busy, stress-ng -c 72 --timeout=360000
2. choose pmem as the disk, cryptsetup -c sm4-xts-plain64 --key-size=256 
--hash sm3 luksFormat --sector-size 4096 --pbkdf=pbkdf2 /dev/pmem0
3. taskset -c 11 fio -filename=/dev/mapper/test -direct=1 -iodepth 64 
-rw=randwrite -ioengine=libaio -bs=4k  -runtime=60 -group_reporting 
-name=sqe_100write_4k -thread -numjobs=1

Results:

origin(without any patch):
[root@fedora hulk-5.10]# taskset -c 11 fio -filename=/dev/mapper/test 
-direct=1 -iodepth 64 -rw=randwrite -ioengine=libaio -bs=4k  -runtime=60 
-group_reporting -name=sqe_100write_4k -thread -numjobs=1
sqe_100write_4k: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 
4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.29
Starting 1 thread
Jobs: 1 (f=1): [w(1)][100.0%][w=189MiB/s][w=48.3k IOPS][eta 00m:00s]
sqe_100write_4k: (groupid=0, jobs=1): err= 0: pid=40205: Sat Mar 11 
17:51:26 2023
   write: IOPS=60.3k, BW=236MiB/s (247MB/s)(13.8GiB/60001msec); 0 zone 
resets
     slat (usec): min=2, max=14393, avg= 4.78, stdev= 9.83
     clat (usec): min=26, max=44200, avg=1054.97, stdev=705.63
      lat (usec): min=31, max=44204, avg=1059.84, stdev=706.58
     clat percentiles (usec):
      |  1.00th=[  188],  5.00th=[  359], 10.00th=[  404], 20.00th=[  469],
      | 30.00th=[  611], 40.00th=[  717], 50.00th=[  914], 60.00th=[ 1172],
      | 70.00th=[ 1303], 80.00th=[ 1467], 90.00th=[ 1827], 95.00th=[ 2180],
      | 99.00th=[ 3294], 99.50th=[ 4146], 99.90th=[ 5800], 99.95th=[ 8225],
      | 99.99th=[14091]
    bw (  KiB/s): min=161560, max=486960, per=100.00%, avg=241961.65, 
stdev=65188.84, samples=119
    iops        : min=40390, max=121740, avg=60490.40, stdev=16297.21, 
samples=119
   lat (usec)   : 50=0.03%, 100=0.33%, 250=1.22%, 500=21.77%, 750=19.12%
   lat (usec)   : 1000=10.40%
   lat (msec)   : 2=40.23%, 4=6.34%, 10=0.53%, 20=0.03%, 50=0.01%
   cpu          : usr=16.16%, sys=29.07%, ctx=1459171, majf=0, minf=32412
   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, 
 >=64=100.0%
      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 >=64=0.0%
      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, 
 >=64=0.0%
      issued rwts: total=0,3620430,0,0 short=0,0,0,0 dropped=0,0,0,0
      latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   WRITE: bw=236MiB/s (247MB/s), 236MiB/s-236MiB/s (247MB/s-247MB/s), 
io=13.8GiB (14.8GB), run=60001-60001msec

origin + schedule everytime:

[root@fedora hulk-5.10]# git diff
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 3d975db86434..17ddca293965 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -1934,6 +1934,7 @@ static int dmcrypt_write(void *data)
                         io = crypt_io_from_node(rb_first(&write_tree));
                         rb_erase(&io->rb_node, &write_tree);
                         kcryptd_io_write(io);
+                       cond_resched();
                 } while (!RB_EMPTY_ROOT(&write_tree));
                 blk_finish_plug(&plug);
         }


[root@fedora hulk-5.10]# taskset -c 11 fio -filename=/dev/mapper/test 
-direct=1 -iodepth 64 -rw=randwrite -ioengine=libaio -bs=4k  -runtime=60 
-group_reporting -name=sqe_100write_4k -thread -numjobs=1
sqe_100write_4k: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 
4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.29
Starting 1 thread
Jobs: 1 (f=1): [w(1)][100.0%][w=197MiB/s][w=50.4k IOPS][eta 00m:00s]
sqe_100write_4k: (groupid=0, jobs=1): err= 0: pid=40287: Sat Mar 11 
17:56:25 2023
   write: IOPS=59.9k, BW=234MiB/s (245MB/s)(13.7GiB/60001msec); 0 zone 
resets
     slat (usec): min=2, max=13251, avg= 5.19, stdev= 9.59
     clat (usec): min=14, max=41490, avg=1062.91, stdev=655.31
      lat (usec): min=26, max=41495, avg=1068.20, stdev=656.78
     clat percentiles (usec):
      |  1.00th=[  269],  5.00th=[  363], 10.00th=[  412], 20.00th=[  494],
      | 30.00th=[  644], 40.00th=[  766], 50.00th=[  979], 60.00th=[ 1172],
      | 70.00th=[ 1303], 80.00th=[ 1450], 90.00th=[ 1827], 95.00th=[ 2147],
      | 99.00th=[ 3097], 99.50th=[ 3785], 99.90th=[ 5145], 99.95th=[ 5735],
      | 99.99th=[13698]
    bw (  KiB/s): min=171976, max=562936, per=100.00%, avg=239748.22, 
stdev=61262.50, samples=119
    iops        : min=42994, max=140734, avg=59937.04, stdev=15315.64, 
samples=119
   lat (usec)   : 20=0.01%, 50=0.01%, 100=0.10%, 250=0.67%, 500=19.60%
   lat (usec)   : 750=18.69%, 1000=11.81%
   lat (msec)   : 2=42.65%, 4=6.05%, 10=0.40%, 20=0.02%, 50=0.01%
   cpu          : usr=15.94%, sys=28.99%, ctx=1564116, majf=0, minf=32437
   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, 
 >=64=100.0%
      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 >=64=0.0%
      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, 
 >=64=0.0%
      issued rwts: total=0,3592061,0,0 short=0,0,0,0 dropped=0,0,0,0
      latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   WRITE: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s), 
io=13.7GiB (14.7GB), run=60001-60001msec

origin + schedule everytime + set nice:

[root@fedora hulk-5.10]# git diff
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 3d975db86434..f2135c223944 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -1934,6 +1934,7 @@ static int dmcrypt_write(void *data)
                         io = crypt_io_from_node(rb_first(&write_tree));
                         rb_erase(&io->rb_node, &write_tree);
                         kcryptd_io_write(io);
+                       cond_resched();
                 } while (!RB_EMPTY_ROOT(&write_tree));
                 blk_finish_plug(&plug);
         }
@@ -3298,18 +3299,18 @@ static int crypt_ctr(struct dm_target *ti, 
unsigned int argc, char **argv)
         }

         ret = -ENOMEM;
-       cc->io_queue = alloc_workqueue("kcryptd_io/%s", WQ_MEM_RECLAIM, 
1, devname);
+       cc->io_queue = alloc_workqueue("kcryptd_io/%s", WQ_MEM_RECLAIM | 
WQ_HIGHPRI, 1, devname);
         if (!cc->io_queue) {
                 ti->error = "Couldn't create kcryptd io queue";
                 goto bad;
         }

         if (test_bit(DM_CRYPT_SAME_CPU, &cc->flags))
-               cc->crypt_queue = alloc_workqueue("kcryptd/%s", 
WQ_CPU_INTENSIVE | WQ_MEM_RECLAIM,
+               cc->crypt_queue = alloc_workqueue("kcryptd/%s", 
WQ_CPU_INTENSIVE | WQ_MEM_RECLAIM | WQ_HIGHPRI,
                                                   1, devname);
         else
                 cc->crypt_queue = alloc_workqueue("kcryptd/%s",
-                                                 WQ_CPU_INTENSIVE | 
WQ_MEM_RECLAIM | WQ_UNBOUND,
+                                                 WQ_CPU_INTENSIVE | 
WQ_MEM_RECLAIM | WQ_UNBOUND | WQ_HIGHPRI,
                                                   num_online_cpus(), 
devname);
         if (!cc->crypt_queue) {
                 ti->error = "Couldn't create kcryptd queue";
@@ -3326,6 +3327,7 @@ static int crypt_ctr(struct dm_target *ti, 
unsigned int argc, char **argv)
                 ti->error = "Couldn't spawn write thread";
                 goto bad;
         }
+       set_user_nice(cc->write_thread, MIN_NICE);
         wake_up_process(cc->write_thread);

         ti->num_flush_bios = 1;


[root@fedora hulk-5.10]# taskset -c 11 fio -filename=/dev/mapper/test 
-direct=1 -iodepth 64 -rw=write -ioengine=libaio -bs=4k  -runtime=60 
-group_reporting -name=sqe_100write_4k -thread -numjobs=1
sqe_100write_4k: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, 
(T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.29
Starting 1 thread
Jobs: 1 (f=1): [W(1)][100.0%][w=132MiB/s][w=33.7k IOPS][eta 00m:00s]
sqe_100write_4k: (groupid=0, jobs=1): err= 0: pid=40483: Sat Mar 11 
18:02:21 2023
   write: IOPS=37.9k, BW=148MiB/s (155MB/s)(8884MiB/60001msec); 0 zone 
resets
     slat (usec): min=2, max=3105, avg=22.68, stdev=11.80
     clat (usec): min=18, max=46657, avg=1664.96, stdev=604.02
      lat (usec): min=39, max=46660, avg=1687.78, stdev=613.12
     clat percentiles (usec):
      |  1.00th=[  383],  5.00th=[  482], 10.00th=[  594], 20.00th=[  914],
      | 30.00th=[ 1696], 40.00th=[ 1860], 50.00th=[ 1909], 60.00th=[ 1958],
      | 70.00th=[ 1991], 80.00th=[ 2073], 90.00th=[ 2180], 95.00th=[ 2212],
      | 99.00th=[ 2376], 99.50th=[ 2442], 99.90th=[ 4359], 99.95th=[ 4883],
      | 99.99th=[12387]
    bw (  KiB/s): min=115368, max=363976, per=100.00%, avg=151823.13, 
stdev=44148.16, samples=119
    iops        : min=28842, max=90994, avg=37955.78, stdev=11037.04, 
samples=119
   lat (usec)   : 20=0.01%, 50=0.01%, 100=0.06%, 250=0.23%, 500=7.38%
   lat (usec)   : 750=4.65%, 1000=9.11%
   lat (msec)   : 2=49.16%, 4=29.29%, 10=0.10%, 20=0.01%, 50=0.01%
   cpu          : usr=8.86%, sys=18.57%, ctx=1878960, majf=0, minf=401
   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, 
 >=64=100.0%
      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 >=64=0.0%
      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, 
 >=64=0.0%
      issued rwts: total=0,2274350,0,0 short=0,0,0,0 dropped=0,0,0,0
      latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   WRITE: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), 
io=8884MiB (9316MB), run=60001-60001msec


I have no idea why set nice will lead to a poor performance for fio... 
But it seems OK that cond_resched every time since it won't result 
performance descent. So, drop this patch and just take '[PATCH] 
dm-crypt: add cond_resched() to dmcrypt_write()' you send?


>>
>> Mikulas
>>
>>
>>
>> From: Mikulas Patocka <mpatocka@redhat.com>
>>
>> It was reported that dm-crypt performs badly when the system is 
>> loaded[1].
>> So I'm adding an option "high_priority" that sets the workqueues and the
>> write_thread to nice level -20. This used to be the default in the past,
>> but it caused audio skipping, so it had to be reverted - see the commit
>> f612b2132db529feac4f965f28a1b9258ea7c22b. This commit makes it optional,
>> so that normal users won't be harmed by it.
>>
>> [1] 
>> https://listman.redhat.com/archives/dm-devel/2023-February/053410.html
>>
>> Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
>>
>> ---
>>   Documentation/admin-guide/device-mapper/dm-crypt.rst |    5 +++
>>   drivers/md/dm-crypt.c                                |   28 
>> +++++++++++++------
>>   2 files changed, 25 insertions(+), 8 deletions(-)
>>
>> Index: linux-2.6/drivers/md/dm-crypt.c
>> ===================================================================
>> --- linux-2.6.orig/drivers/md/dm-crypt.c
>> +++ linux-2.6/drivers/md/dm-crypt.c
>> @@ -133,9 +133,9 @@ struct iv_elephant_private {
>>    * and encrypts / decrypts at the same time.
>>    */
>>   enum flags { DM_CRYPT_SUSPENDED, DM_CRYPT_KEY_VALID,
>> -         DM_CRYPT_SAME_CPU, DM_CRYPT_NO_OFFLOAD,
>> -         DM_CRYPT_NO_READ_WORKQUEUE, DM_CRYPT_NO_WRITE_WORKQUEUE,
>> -         DM_CRYPT_WRITE_INLINE };
>> +         DM_CRYPT_SAME_CPU, DM_CRYPT_HIGH_PRIORITY,
>> +         DM_CRYPT_NO_OFFLOAD, DM_CRYPT_NO_READ_WORKQUEUE,
>> +         DM_CRYPT_NO_WRITE_WORKQUEUE, DM_CRYPT_WRITE_INLINE };
>>   enum cipher_flags {
>>       CRYPT_MODE_INTEGRITY_AEAD,    /* Use authenticated mode for 
>> cipher */
>> @@ -3087,7 +3087,7 @@ static int crypt_ctr_optional(struct dm_
>>       struct crypt_config *cc = ti->private;
>>       struct dm_arg_set as;
>>       static const struct dm_arg _args[] = {
>> -        {0, 8, "Invalid number of feature args"},
>> +        {0, 9, "Invalid number of feature args"},
>>       };
>>       unsigned int opt_params, val;
>>       const char *opt_string, *sval;
>> @@ -3114,6 +3114,8 @@ static int crypt_ctr_optional(struct dm_
>>           else if (!strcasecmp(opt_string, "same_cpu_crypt"))
>>               set_bit(DM_CRYPT_SAME_CPU, &cc->flags);
>> +        else if (!strcasecmp(opt_string, "high_priority"))
>> +            set_bit(DM_CRYPT_HIGH_PRIORITY, &cc->flags);
>>           else if (!strcasecmp(opt_string, "submit_from_crypt_cpus"))
>>               set_bit(DM_CRYPT_NO_OFFLOAD, &cc->flags);
>> @@ -3352,18 +3354,22 @@ static int crypt_ctr(struct dm_target *t
>>       }
>>       ret = -ENOMEM;
>> -    cc->io_queue = alloc_workqueue("kcryptd_io/%s", WQ_MEM_RECLAIM, 
>> 1, devname);
>> +    cc->io_queue = alloc_workqueue("kcryptd_io/%s", WQ_MEM_RECLAIM |
>> +                       test_bit(DM_CRYPT_HIGH_PRIORITY, &cc->flags) * 
>> WQ_HIGHPRI,
>> +                       1, devname);
>>       if (!cc->io_queue) {
>>           ti->error = "Couldn't create kcryptd io queue";
>>           goto bad;
>>       }
>>       if (test_bit(DM_CRYPT_SAME_CPU, &cc->flags))
>> -        cc->crypt_queue = alloc_workqueue("kcryptd/%s", 
>> WQ_CPU_INTENSIVE | WQ_MEM_RECLAIM,
>> +        cc->crypt_queue = alloc_workqueue("kcryptd/%s", 
>> WQ_CPU_INTENSIVE | WQ_MEM_RECLAIM |
>> +                          test_bit(DM_CRYPT_HIGH_PRIORITY, 
>> &cc->flags) * WQ_HIGHPRI,
>>                             1, devname);
>>       else
>>           cc->crypt_queue = alloc_workqueue("kcryptd/%s",
>> -                          WQ_CPU_INTENSIVE | WQ_MEM_RECLAIM | 
>> WQ_UNBOUND,
>> +                          WQ_CPU_INTENSIVE | WQ_MEM_RECLAIM | 
>> WQ_UNBOUND |
>> +                          test_bit(DM_CRYPT_HIGH_PRIORITY, 
>> &cc->flags) * WQ_HIGHPRI,
>>                             num_online_cpus(), devname);
>>       if (!cc->crypt_queue) {
>>           ti->error = "Couldn't create kcryptd queue";
>> @@ -3380,6 +3386,8 @@ static int crypt_ctr(struct dm_target *t
>>           ti->error = "Couldn't spawn write thread";
>>           goto bad;
>>       }
>> +    if (test_bit(DM_CRYPT_HIGH_PRIORITY, &cc->flags))
>> +        set_user_nice(cc->write_thread, MIN_NICE);
>>       ti->num_flush_bios = 1;
>>       ti->limit_swap_bios = true;
>> @@ -3500,6 +3508,7 @@ static void crypt_status(struct dm_targe
>>           num_feature_args += !!ti->num_discard_bios;
>>           num_feature_args += test_bit(DM_CRYPT_SAME_CPU, &cc->flags);
>> +        num_feature_args += test_bit(DM_CRYPT_HIGH_PRIORITY, 
>> &cc->flags);
>>           num_feature_args += test_bit(DM_CRYPT_NO_OFFLOAD, &cc->flags);
>>           num_feature_args += test_bit(DM_CRYPT_NO_READ_WORKQUEUE, 
>> &cc->flags);
>>           num_feature_args += test_bit(DM_CRYPT_NO_WRITE_WORKQUEUE, 
>> &cc->flags);
>> @@ -3513,6 +3522,8 @@ static void crypt_status(struct dm_targe
>>                   DMEMIT(" allow_discards");
>>               if (test_bit(DM_CRYPT_SAME_CPU, &cc->flags))
>>                   DMEMIT(" same_cpu_crypt");
>> +            if (test_bit(DM_CRYPT_HIGH_PRIORITY, &cc->flags))
>> +                DMEMIT(" high_priority");
>>               if (test_bit(DM_CRYPT_NO_OFFLOAD, &cc->flags))
>>                   DMEMIT(" submit_from_crypt_cpus");
>>               if (test_bit(DM_CRYPT_NO_READ_WORKQUEUE, &cc->flags))
>> @@ -3532,6 +3543,7 @@ static void crypt_status(struct dm_targe
>>           DMEMIT_TARGET_NAME_VERSION(ti->type);
>>           DMEMIT(",allow_discards=%c", ti->num_discard_bios ? 'y' : 'n');
>>           DMEMIT(",same_cpu_crypt=%c", test_bit(DM_CRYPT_SAME_CPU, 
>> &cc->flags) ? 'y' : 'n');
>> +        DMEMIT(",high_priority=%c", test_bit(DM_CRYPT_HIGH_PRIORITY, 
>> &cc->flags) ? 'y' : 'n');
>>           DMEMIT(",submit_from_crypt_cpus=%c", 
>> test_bit(DM_CRYPT_NO_OFFLOAD, &cc->flags) ?
>>                  'y' : 'n');
>>           DMEMIT(",no_read_workqueue=%c", 
>> test_bit(DM_CRYPT_NO_READ_WORKQUEUE, &cc->flags) ?
>> @@ -3659,7 +3671,7 @@ static void crypt_io_hints(struct dm_tar
>>   static struct target_type crypt_target = {
>>       .name   = "crypt",
>> -    .version = {1, 24, 0},
>> +    .version = {1, 25, 0},
>>       .module = THIS_MODULE,
>>       .ctr    = crypt_ctr,
>>       .dtr    = crypt_dtr,
>> Index: linux-2.6/Documentation/admin-guide/device-mapper/dm-crypt.rst
>> ===================================================================
>> --- linux-2.6.orig/Documentation/admin-guide/device-mapper/dm-crypt.rst
>> +++ linux-2.6/Documentation/admin-guide/device-mapper/dm-crypt.rst
>> @@ -113,6 +113,11 @@ same_cpu_crypt
>>       The default is to use an unbound workqueue so that encryption work
>>       is automatically balanced between available CPUs.
>> +high_priority
>> +    Set dm-crypt workqueues and the writer thread to high priority. This
>> +    improves throughput and latency of dm-crypt while degrading general
>> +    responsiveness of the system.
>> +
>>   submit_from_crypt_cpus
>>       Disable offloading writes to a separate thread after encryption.
>>       There are some situations where offloading write bios from the
>>

--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel

      reply	other threads:[~2023-03-11  9:27 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-03-06 19:01 [dm-devel] [PATCH] dm-crypt: add the "high_priority" flag Mikulas Patocka
2023-03-07  1:30 ` yangerkun
2023-03-11  9:26   ` yangerkun [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=c48b1039-0911-c7f2-dc0e-d2de985ad295@huawei.com \
    --to=yangerkun@huawei.com \
    --cc=agk@redhat.com \
    --cc=bvanassche@acm.org \
    --cc=dm-devel@redhat.com \
    --cc=gmazyland@gmail.com \
    --cc=mpatocka@redhat.com \
    --cc=snitzer@kernel.org \
    --cc=yangerkun@huaweicloud.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.