* How to estimate the upper bound of the peak memory consumption of cryptsetup itself? @ 2022-06-16 4:43 Coiby Xu 2022-06-18 15:12 ` Milan Broz 0 siblings, 1 reply; 6+ messages in thread From: Coiby Xu @ 2022-06-16 4:43 UTC (permalink / raw) To: cryptsetup Hi, Recently, I notice cryptsetup itself consumes significant amount of memory (~256M) when estimating the memory requirement for dumping vmcore to a LUKS-encrypted disk, $ time -v cryptsetup luksOpen encrypted.img volume --key-file mykey.keyfile | grep "Maximum resident set size" Maximum resident set size (kbytes): 1309828 $ cryptsetup luksDump encrypted.img ... Keyslots: 0: luks2 PBKDF: argon2id Memory: 1048576 ... So is there a way to estimate the upper bound of the peak memory consumption of cryptsetup itself without running cryptsetup? Thanks! -- Best regards, Coiby ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: How to estimate the upper bound of the peak memory consumption of cryptsetup itself? 2022-06-16 4:43 How to estimate the upper bound of the peak memory consumption of cryptsetup itself? Coiby Xu @ 2022-06-18 15:12 ` Milan Broz 2022-06-20 0:19 ` Coiby Xu 0 siblings, 1 reply; 6+ messages in thread From: Milan Broz @ 2022-06-18 15:12 UTC (permalink / raw) To: Coiby Xu, cryptsetup On 16/06/2022 06:43, Coiby Xu wrote: > Hi, > > Recently, I notice cryptsetup itself consumes significant amount of > memory (~256M) when estimating the memory requirement for dumping vmcore > to a LUKS-encrypted disk, > > $ time -v cryptsetup luksOpen encrypted.img volume --key-file mykey.keyfile | grep "Maximum resident set size" > Maximum resident set size (kbytes): 1309828 > $ cryptsetup luksDump encrypted.img > ... > Keyslots: > 0: luks2 > PBKDF: argon2id > Memory: 1048576 > ... > > > So is there a way to estimate the upper bound of the peak memory > consumption of cryptsetup itself without running cryptsetup? As you already found, the major memory consumption is by memory-hard KDF. But this memory is used only while calculating keyslot encryption key, it is released immediately after the Argon call is finished. I do not think we have better estimation here. (Another story is locking all memory, including big areas used by libc, but that should not be problem here, I hope.) Milan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: How to estimate the upper bound of the peak memory consumption of cryptsetup itself? 2022-06-18 15:12 ` Milan Broz @ 2022-06-20 0:19 ` Coiby Xu 2022-06-24 9:14 ` Milan Broz 0 siblings, 1 reply; 6+ messages in thread From: Coiby Xu @ 2022-06-20 0:19 UTC (permalink / raw) To: Milan Broz; +Cc: cryptsetup On Sat, Jun 18, 2022 at 05:12:56PM +0200, Milan Broz wrote: >On 16/06/2022 06:43, Coiby Xu wrote: >>Hi, >> >>Recently, I notice cryptsetup itself consumes significant amount of >>memory (~256M) when estimating the memory requirement for dumping vmcore >>to a LUKS-encrypted disk, >> >>$ time -v cryptsetup luksOpen encrypted.img volume --key-file mykey.keyfile | grep "Maximum resident set size" >> Maximum resident set size (kbytes): 1309828 >>$ cryptsetup luksDump encrypted.img >>... >>Keyslots: >> 0: luks2 >> PBKDF: argon2id >> Memory: 1048576 >> ... >> >> >>So is there a way to estimate the upper bound of the peak memory >>consumption of cryptsetup itself without running cryptsetup? > >As you already found, the major memory consumption is by memory-hard KDF. >But this memory is used only while calculating keyslot encryption key, >it is released immediately after the Argon call is finished. >I do not think we have better estimation here. Thanks for the reply! Sorry I meant the way to estimate the overhead of crypsetup itself i.e. ~256M in the above example. Previously I only take the memory consumption by memory-hard KDF into consideration and neglected the memory consumption of cryptsetup itself. This obviously leads to an underestimation of the memory requirement of cryptsetup. I need to overestimate the memory requirement a bit to make sure OOM won't happen that's why I am asking if there is a way to estimate the upper bound of memory requirement of cryptsetup itself. > >(Another story is locking all memory, including big areas used by libc, >but that should not be problem here, I hope.) Do you mean locking all memory first in order to know the memory requirement? > >Milan > -- Best regards, Coiby ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: How to estimate the upper bound of the peak memory consumption of cryptsetup itself? 2022-06-20 0:19 ` Coiby Xu @ 2022-06-24 9:14 ` Milan Broz 2022-06-24 10:49 ` Coiby Xu 0 siblings, 1 reply; 6+ messages in thread From: Milan Broz @ 2022-06-24 9:14 UTC (permalink / raw) To: Coiby Xu; +Cc: cryptsetup On 20/06/2022 02:19, Coiby Xu wrote: > On Sat, Jun 18, 2022 at 05:12:56PM +0200, Milan Broz wrote: >> On 16/06/2022 06:43, Coiby Xu wrote: >>> Hi, >>> >>> Recently, I notice cryptsetup itself consumes significant amount of >>> memory (~256M) when estimating the memory requirement for dumping vmcore >>> to a LUKS-encrypted disk, >>> >>> $ time -v cryptsetup luksOpen encrypted.img volume --key-file mykey.keyfile | grep "Maximum resident set size" >>> Maximum resident set size (kbytes): 1309828 >>> $ cryptsetup luksDump encrypted.img >>> ... >>> Keyslots: >>> 0: luks2 >>> PBKDF: argon2id >>> Memory: 1048576 >>> ... >>> >>> >>> So is there a way to estimate the upper bound of the peak memory >>> consumption of cryptsetup itself without running cryptsetup? >> >> As you already found, the major memory consumption is by memory-hard KDF. >> But this memory is used only while calculating keyslot encryption key, >> it is released immediately after the Argon call is finished. >> I do not think we have better estimation here. > > Thanks for the reply! Sorry I meant the way to estimate the overhead of > crypsetup itself i.e. ~256M in the above example. Previously I only take > the memory consumption by memory-hard KDF into consideration and > neglected the memory consumption of cryptsetup itself. This obviously > leads to an underestimation of the memory requirement of cryptsetup. I > need to overestimate the memory requirement a bit to make sure OOM won't > happen that's why I am asking if there is a way to estimate the > upper bound of memory requirement of cryptsetup itself. There is no generic way to get a number - it depends on configuration of the distro, libc, translations, everything that is locked including shared libraries. If it is about RHEL, you can perhaps know exact configuration - please ask people in Red Hat. >> (Another story is locking all memory, including big areas used by libc, >> but that should not be problem here, I hope.) > > Do you mean locking all memory first in order to know the memory > requirement? We use (mlockall(MCL_CURRENT | MCL_FUTURE) so it locks all used memory + all future allocated memory. Today, it is not the best option and we will probably lock only specific region with stored keys in the future. Milan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: How to estimate the upper bound of the peak memory consumption of cryptsetup itself? 2022-06-24 9:14 ` Milan Broz @ 2022-06-24 10:49 ` Coiby Xu 2022-06-24 12:22 ` Milan Broz 0 siblings, 1 reply; 6+ messages in thread From: Coiby Xu @ 2022-06-24 10:49 UTC (permalink / raw) To: Milan Broz; +Cc: cryptsetup On Fri, Jun 24, 2022 at 11:14:37AM +0200, Milan Broz wrote: >On 20/06/2022 02:19, Coiby Xu wrote: >>On Sat, Jun 18, 2022 at 05:12:56PM +0200, Milan Broz wrote: >>>On 16/06/2022 06:43, Coiby Xu wrote: >>>>Hi, >>>> >>>>Recently, I notice cryptsetup itself consumes significant amount of >>>>memory (~256M) when estimating the memory requirement for dumping vmcore >>>>to a LUKS-encrypted disk, >>>> >>>>$ time -v cryptsetup luksOpen encrypted.img volume --key-file mykey.keyfile | grep "Maximum resident set size" >>>> Maximum resident set size (kbytes): 1309828 >>>>$ cryptsetup luksDump encrypted.img >>>>... >>>>Keyslots: >>>> 0: luks2 >>>> PBKDF: argon2id >>>> Memory: 1048576 >>>> ... >>>> >>>> >>>>So is there a way to estimate the upper bound of the peak memory >>>>consumption of cryptsetup itself without running cryptsetup? >>> >>>As you already found, the major memory consumption is by memory-hard KDF. >>>But this memory is used only while calculating keyslot encryption key, >>>it is released immediately after the Argon call is finished. >>>I do not think we have better estimation here. >> >>Thanks for the reply! Sorry I meant the way to estimate the overhead of >>crypsetup itself i.e. ~256M in the above example. Previously I only take >>the memory consumption by memory-hard KDF into consideration and >>neglected the memory consumption of cryptsetup itself. This obviously >>leads to an underestimation of the memory requirement of cryptsetup. I >>need to overestimate the memory requirement a bit to make sure OOM won't >>happen that's why I am asking if there is a way to estimate the >>upper bound of memory requirement of cryptsetup itself. > >There is no generic way to get a number - it depends on configuration >of the distro, libc, translations, everything that is locked including >shared libraries. > >If it is about RHEL, you can perhaps know exact configuration - please >ask people in Red Hat. Provided the configuration, is there an golden algorithm or a formula to get the number? If it doesn't exist and I need to do some tests to get an empirical number, are there any big factors I need to be aware of? > >>>(Another story is locking all memory, including big areas used by libc, >>>but that should not be problem here, I hope.) >> >>Do you mean locking all memory first in order to know the memory >>requirement? > >We use (mlockall(MCL_CURRENT | MCL_FUTURE) so it locks all used memory >+ all future allocated memory. > >Today, it is not the best option and we will probably lock only specific >region with stored keys in the future. Thanks for the explanation! > >Milan > -- Best regards, Coiby ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: How to estimate the upper bound of the peak memory consumption of cryptsetup itself? 2022-06-24 10:49 ` Coiby Xu @ 2022-06-24 12:22 ` Milan Broz 0 siblings, 0 replies; 6+ messages in thread From: Milan Broz @ 2022-06-24 12:22 UTC (permalink / raw) To: Coiby Xu; +Cc: cryptsetup On 24/06/2022 12:49, Coiby Xu wrote: >>> Thanks for the reply! Sorry I meant the way to estimate the overhead of >>> crypsetup itself i.e. ~256M in the above example. Previously I only take >>> the memory consumption by memory-hard KDF into consideration and >>> neglected the memory consumption of cryptsetup itself. This obviously >>> leads to an underestimation of the memory requirement of cryptsetup. I >>> need to overestimate the memory requirement a bit to make sure OOM won't >>> happen that's why I am asking if there is a way to estimate the >>> upper bound of memory requirement of cryptsetup itself. >> >> There is no generic way to get a number - it depends on configuration >> of the distro, libc, translations, everything that is locked including >> shared libraries. >> >> If it is about RHEL, you can perhaps know exact configuration - please >> ask people in Red Hat. > > Provided the configuration, is there an golden algorithm or a formula to > get the number? If it doesn't exist and I need to do some tests to get > an empirical number, are there any big factors I need to be aware of? Think about current locales, OpenSSL policy and many other things. (The major I remember was locking all libc loaded translations in memory.) So really, the only answer here would be ... 42 :) You can try to measure it. Not sure if it is reliable. Usually people want to know this for some weird scenarios like kdump or so - this is definitely not something upstream can help with. Milan ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-06-24 12:23 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-06-16 4:43 How to estimate the upper bound of the peak memory consumption of cryptsetup itself? Coiby Xu 2022-06-18 15:12 ` Milan Broz 2022-06-20 0:19 ` Coiby Xu 2022-06-24 9:14 ` Milan Broz 2022-06-24 10:49 ` Coiby Xu 2022-06-24 12:22 ` Milan Broz
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.