[lustre-devel] Recommendation for hsm_restore for directories

[lustre-devel] Recommendation for hsm_restore for directories

[Degremont, Aurelien via lustre-devel]

Hello all

I'm doing some early work toward implementing a directory/namespace support for Lustre/HSM. The idea would be to support something similar to hsm_restore/hsm_release but for directories.

My first thought was for the MDT to not grant LDLM lock for this directory and hold it, while the directory is getting restored by copytool, in a similar fashion than what is done for files. But the hard part here, is to have a way for copytool to access the directory while its access is actually prevented by the above locking.
Restoring a file is using this smart trick of restoring it in a different temporary file and using layout swap at the end to move data to the actual real file. It looks difficult to do the same thing here.

Do you have any recommendation on the right way to frame this directory access for copytools?

Aurélien


_______________________________________________
lustre-devel mailing list
lustre-devel@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

[lustre-devel] 回复: Recommendation for hsm_restore for directories

[Qian Yingjin via lustre-devel]

[-- Attachment #1.1: Type: text/plain, Size: 1520 bytes --]

Hi Aurélien,

There is a rough thought about it: https://jira.whamcloud.com/browse/LU-13024
Lustre HSM support for a directory

Regards,
Qian

________________________________
发件人: lustre-devel <lustre-devel-bounces@lists.lustre.org> 代表 Degremont, Aurelien via lustre-devel <lustre-devel@lists.lustre.org>
发送时间: 2021年11月25日 18:18
收件人: lustre-devel@lists.lustre.org <lustre-devel@lists.lustre.org>
主题: [lustre-devel] Recommendation for hsm_restore for directories

Hello all

I'm doing some early work toward implementing a directory/namespace support for Lustre/HSM. The idea would be to support something similar to hsm_restore/hsm_release but for directories.

My first thought was for the MDT to not grant LDLM lock for this directory and hold it, while the directory is getting restored by copytool, in a similar fashion than what is done for files. But the hard part here, is to have a way for copytool to access the directory while its access is actually prevented by the above locking.
Restoring a file is using this smart trick of restoring it in a different temporary file and using layout swap at the end to move data to the actual real file. It looks difficult to do the same thing here.

Do you have any recommendation on the right way to frame this directory access for copytools?

Aurélien


_______________________________________________
lustre-devel mailing list
lustre-devel@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

[-- Attachment #1.2: Type: text/html, Size: 5726 bytes --]

[-- Attachment #2: Type: text/plain, Size: 165 bytes --]

_______________________________________________
lustre-devel mailing list
lustre-devel@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

Re: [lustre-devel] 回复: Recommendation for hsm_restore for directories

[Degremont, Aurelien via lustre-devel]

[-- Attachment #1.1: Type: text/plain, Size: 2766 bytes --]

Hi

Thanks for the pointer.

> Similar to archive, when restore a directory, we should also use open flags O_IGNORE_IBITS_LOCK | O_IGNORE_EXTENT_LOCK to create files and resync data without taking any locks.
Based on this ticket I understand that your proposal is to add new flags to ignore parent directory locking when creating new files, right?

I thought about that approach but I was afraid this could be a bit painful.
This flag would have to be checked in a lot of different places in the code, probably not only open.
I wanted also to keep the possibility to create new files in a released directory without needing to restore it. Requiring the actual only really when calling readdir
I was wondering if other thinks the best approach, maybe it is.


Aurélien

De : Qian Yingjin <qian@ddn.com>
Date : jeudi 25 novembre 2021 à 15:33
À : "lustre-devel@lists.lustre.org" <lustre-devel@lists.lustre.org>, "Degremont, Aurelien" <degremoa@amazon.fr>
Objet : [EXTERNAL] 回复: Recommendation for hsm_restore for directories


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.


Hi Aurélien,


There is a rough thought about it: https://jira.whamcloud.com/browse/LU-13024
Lustre HSM support for a directory


Regards,
Qian

________________________________
发件人: lustre-devel <lustre-devel-bounces@lists.lustre.org> 代表 Degremont, Aurelien via lustre-devel <lustre-devel@lists.lustre.org>
发送时间: 2021年11月25日 18:18
收件人: lustre-devel@lists.lustre.org <lustre-devel@lists.lustre.org>
主题: [lustre-devel] Recommendation for hsm_restore for directories

Hello all

I'm doing some early work toward implementing a directory/namespace support for Lustre/HSM. The idea would be to support something similar to hsm_restore/hsm_release but for directories.

My first thought was for the MDT to not grant LDLM lock for this directory and hold it, while the directory is getting restored by copytool, in a similar fashion than what is done for files. But the hard part here, is to have a way for copytool to access the directory while its access is actually prevented by the above locking.
Restoring a file is using this smart trick of restoring it in a different temporary file and using layout swap at the end to move data to the actual real file. It looks difficult to do the same thing here.

Do you have any recommendation on the right way to frame this directory access for copytools?

Aurélien


_______________________________________________
lustre-devel mailing list
lustre-devel@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

[-- Attachment #1.2: Type: text/html, Size: 11557 bytes --]

[-- Attachment #2: Type: text/plain, Size: 165 bytes --]

_______________________________________________
lustre-devel mailing list
lustre-devel@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org