[lustre-devel] HSM phase 2 HLD / Roadmap?

[lustre-devel] HSM phase 2 HLD / Roadmap?

[Colin Faber]

Hi Folks,

Does anyone recall if Lustre HSM phase 2 documentation or road map was ever
produced / published anywhere?

-cf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20150616/e7dcf465/attachment.htm>

[lustre-devel] HSM phase 2 HLD / Roadmap?

[DEGREMONT Aurelien]

Le 16/06/2015 17:02, Colin Faber a ?crit :
> Hi Folks,
>
> Does anyone recall if Lustre HSM phase 2 documentation or road map was 
> ever produced / published anywhere?
AFAICR there never was a real official content for HSM phase 2, just 
ideas/wish list. As a consequence, no road map was done.


Aur?lien
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20150616/2e2f71b8/attachment.htm>

[lustre-devel] HSM phase 2 HLD / Roadmap?

[Colin Faber]

Ah yes,

This is what I thought as well, I just could not remember. Thanks for
reminding me!

-cf


On Tue, Jun 16, 2015 at 9:08 AM, DEGREMONT Aurelien <
aurelien.degremont@cea.fr> wrote:

>  Le 16/06/2015 17:02, Colin Faber a ?crit :
>
> Hi Folks,
>
>  Does anyone recall if Lustre HSM phase 2 documentation or road map was
> ever produced / published anywhere?
>
> AFAICR there never was a real official content for HSM phase 2, just
> ideas/wish list. As a consequence, no road map was done.
>
>
> Aur?lien
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20150616/b12e9daf/attachment.htm>

[lustre-devel] HSM phase 2 HLD / Roadmap?

[Dilger, Andreas]

I think there are a few features that would be of interest for HSM
improvement.

* optimization of RobinHood database/POSIX copytool
* better integration with DNE
* partial file restore using composite files
(http://wiki.opensfs.org/images/e/e4/LayoutEnhancementDesign_HighLevelDesig
n.pdf)
   * change HSM to use LOV EA layout instead of dedicated xattr
   * allowing a larger HSM archive ID (e.g. 128-bit UUID) so it can be
     used as the primary archive identifier instead of Lustre FID
* archive/restore of directory trees from the MDT as tarballs
   * allow release/restore of space from the MDT
   * would be needed for handling Data-on-MDT files

Note that we don't have any plans to work on this, just listing some ideas
percolating in my brain.


Cheers, Andreas
-- 
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division



On 2015/06/16, 9:08 AM, "DEGREMONT Aurelien" <aurelien.degremont@cea.fr>
wrote:


>Le 16/06/2015 17:02, Colin Faber a ?crit :
>
>
>Hi Folks,
>
>Does anyone recall if Lustre HSM phase 2 documentation or road map was
>ever produced / published anywhere?
>
>
>
>AFAICR there never was a real official content for HSM phase 2, just
>ideas/wish list. As a consequence, no road map was done.
>
>
>Aur?lien

[lustre-devel] HSM phase 2 HLD / Roadmap?

[Patrick Farrell]

Andreas,

Could you expound on "Better integration with DNE"?  I'm not immediately 
clear on what that might mean.

Thanks,
- Patrick
On 06/16/2015 10:21 AM, Dilger, Andreas wrote:
> I think there are a few features that would be of interest for HSM
> improvement.
>
> * optimization of RobinHood database/POSIX copytool
> * better integration with DNE
> * partial file restore using composite files
> (http://wiki.opensfs.org/images/e/e4/LayoutEnhancementDesign_HighLevelDesig
> n.pdf)
>     * change HSM to use LOV EA layout instead of dedicated xattr
>     * allowing a larger HSM archive ID (e.g. 128-bit UUID) so it can be
>       used as the primary archive identifier instead of Lustre FID
> * archive/restore of directory trees from the MDT as tarballs
>     * allow release/restore of space from the MDT
>     * would be needed for handling Data-on-MDT files
>
> Note that we don't have any plans to work on this, just listing some ideas
> percolating in my brain.
>
>
> Cheers, Andreas

[lustre-devel] HSM phase 2 HLD / Roadmap?

[Dilger, Andreas]

On 2015/06/16, 9:28 AM, "Patrick Farrell" <paf@cray.com> wrote:

>Andreas,
>
>Could you expound on "Better integration with DNE"?  I'm not immediately
>clear on what that might mean.

I was thinking that the current DNE support is still using a single RBH
database fed by multiple ChangeLogs (one from each MDT), which doesn't
scale very well.  That said, I guess it would be possible to have a single
parallel MySQL database that scales horizontally, so the scaling would be
independent of the number of MDTs.

Maybe this one should be dropped from the list, or just be considered part
of the "optimize RBH database".

Cheers, Andreas

>Thanks,
>- Patrick
>On 06/16/2015 10:21 AM, Dilger, Andreas wrote:
>> I think there are a few features that would be of interest for HSM
>> improvement.
>>
>> * optimization of RobinHood database/POSIX copytool
>> * better integration with DNE
>> * partial file restore using composite files
>> 
>>(http://wiki.opensfs.org/images/e/e4/LayoutEnhancementDesign_HighLevelDes
>>ig
>> n.pdf)
>>     * change HSM to use LOV EA layout instead of dedicated xattr
>>     * allowing a larger HSM archive ID (e.g. 128-bit UUID) so it can be
>>       used as the primary archive identifier instead of Lustre FID
>> * archive/restore of directory trees from the MDT as tarballs
>>     * allow release/restore of space from the MDT
>>     * would be needed for handling Data-on-MDT files
>>
>> Note that we don't have any plans to work on this, just listing some
>>ideas
>> percolating in my brain.
>>
>>
>> Cheers, Andreas
>
>_______________________________________________
>lustre-devel mailing list
>lustre-devel at lists.lustre.org
>http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
>


Cheers, Andreas
-- 
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division

[lustre-devel] HSM phase 2 HLD / Roadmap?

[LEIBOVICI Thomas]

On 06/16/15 18:15, Dilger, Andreas wrote:
> On 2015/06/16, 9:28 AM, "Patrick Farrell" <paf@cray.com> wrote:
>
>> Andreas,
>>
>> Could you expound on "Better integration with DNE"?  I'm not immediately
>> clear on what that might mean.
> I was thinking that the current DNE support is still using a single RBH
> database fed by multiple ChangeLogs (one from each MDT), which doesn't
> scale very well.  That said, I guess it would be possible to have a single
> parallel MySQL database that scales horizontally, so the scaling would be
> independent of the number of MDTs.
>
> Maybe this one should be dropped from the list, or just be considered part
> of the "optimize RBH database".
>
> Cheers, Andreas
AFAIK, there are other aspects of DNE that HSM wouldn't support:
HSM does not currently support striped directories (DNE phase 2)
in particular the volatile file used for "restore" operations wouldn't 
be properly located on the right MDT.

Thomas

>> Thanks,
>> - Patrick
>> On 06/16/2015 10:21 AM, Dilger, Andreas wrote:
>>> I think there are a few features that would be of interest for HSM
>>> improvement.
>>>
>>> * optimization of RobinHood database/POSIX copytool
>>> * better integration with DNE
>>> * partial file restore using composite files
>>>
>>> (http://wiki.opensfs.org/images/e/e4/LayoutEnhancementDesign_HighLevelDes
>>> ig
>>> n.pdf)
>>>      * change HSM to use LOV EA layout instead of dedicated xattr
>>>      * allowing a larger HSM archive ID (e.g. 128-bit UUID) so it can be
>>>        used as the primary archive identifier instead of Lustre FID
>>> * archive/restore of directory trees from the MDT as tarballs
>>>      * allow release/restore of space from the MDT
>>>      * would be needed for handling Data-on-MDT files
>>>
>>> Note that we don't have any plans to work on this, just listing some
>>> ideas
>>> percolating in my brain.
>>>
>>>
>>> Cheers, Andreas
>> _______________________________________________
>> lustre-devel mailing list
>> lustre-devel at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
>>
>
> Cheers, Andreas