* [lustre-devel] HSM phase 2 HLD / Roadmap? @ 2015-06-16 15:02 Colin Faber 2015-06-16 15:08 ` DEGREMONT Aurelien 0 siblings, 1 reply; 7+ messages in thread From: Colin Faber @ 2015-06-16 15:02 UTC (permalink / raw) To: lustre-devel Hi Folks, Does anyone recall if Lustre HSM phase 2 documentation or road map was ever produced / published anywhere? -cf -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20150616/e7dcf465/attachment.htm> ^ permalink raw reply [flat|nested] 7+ messages in thread
* [lustre-devel] HSM phase 2 HLD / Roadmap? 2015-06-16 15:02 [lustre-devel] HSM phase 2 HLD / Roadmap? Colin Faber @ 2015-06-16 15:08 ` DEGREMONT Aurelien 2015-06-16 15:10 ` Colin Faber 2015-06-16 15:21 ` Dilger, Andreas 0 siblings, 2 replies; 7+ messages in thread From: DEGREMONT Aurelien @ 2015-06-16 15:08 UTC (permalink / raw) To: lustre-devel Le 16/06/2015 17:02, Colin Faber a ?crit : > Hi Folks, > > Does anyone recall if Lustre HSM phase 2 documentation or road map was > ever produced / published anywhere? AFAICR there never was a real official content for HSM phase 2, just ideas/wish list. As a consequence, no road map was done. Aur?lien -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20150616/2e2f71b8/attachment.htm> ^ permalink raw reply [flat|nested] 7+ messages in thread
* [lustre-devel] HSM phase 2 HLD / Roadmap? 2015-06-16 15:08 ` DEGREMONT Aurelien @ 2015-06-16 15:10 ` Colin Faber 2015-06-16 15:21 ` Dilger, Andreas 1 sibling, 0 replies; 7+ messages in thread From: Colin Faber @ 2015-06-16 15:10 UTC (permalink / raw) To: lustre-devel Ah yes, This is what I thought as well, I just could not remember. Thanks for reminding me! -cf On Tue, Jun 16, 2015 at 9:08 AM, DEGREMONT Aurelien < aurelien.degremont@cea.fr> wrote: > Le 16/06/2015 17:02, Colin Faber a ?crit : > > Hi Folks, > > Does anyone recall if Lustre HSM phase 2 documentation or road map was > ever produced / published anywhere? > > AFAICR there never was a real official content for HSM phase 2, just > ideas/wish list. As a consequence, no road map was done. > > > Aur?lien > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20150616/b12e9daf/attachment.htm> ^ permalink raw reply [flat|nested] 7+ messages in thread
* [lustre-devel] HSM phase 2 HLD / Roadmap? 2015-06-16 15:08 ` DEGREMONT Aurelien 2015-06-16 15:10 ` Colin Faber @ 2015-06-16 15:21 ` Dilger, Andreas 2015-06-16 15:28 ` Patrick Farrell 1 sibling, 1 reply; 7+ messages in thread From: Dilger, Andreas @ 2015-06-16 15:21 UTC (permalink / raw) To: lustre-devel I think there are a few features that would be of interest for HSM improvement. * optimization of RobinHood database/POSIX copytool * better integration with DNE * partial file restore using composite files (http://wiki.opensfs.org/images/e/e4/LayoutEnhancementDesign_HighLevelDesig n.pdf) * change HSM to use LOV EA layout instead of dedicated xattr * allowing a larger HSM archive ID (e.g. 128-bit UUID) so it can be used as the primary archive identifier instead of Lustre FID * archive/restore of directory trees from the MDT as tarballs * allow release/restore of space from the MDT * would be needed for handling Data-on-MDT files Note that we don't have any plans to work on this, just listing some ideas percolating in my brain. Cheers, Andreas -- Andreas Dilger Lustre Software Architect Intel High Performance Data Division On 2015/06/16, 9:08 AM, "DEGREMONT Aurelien" <aurelien.degremont@cea.fr> wrote: >Le 16/06/2015 17:02, Colin Faber a ?crit : > > >Hi Folks, > >Does anyone recall if Lustre HSM phase 2 documentation or road map was >ever produced / published anywhere? > > > >AFAICR there never was a real official content for HSM phase 2, just >ideas/wish list. As a consequence, no road map was done. > > >Aur?lien ^ permalink raw reply [flat|nested] 7+ messages in thread
* [lustre-devel] HSM phase 2 HLD / Roadmap? 2015-06-16 15:21 ` Dilger, Andreas @ 2015-06-16 15:28 ` Patrick Farrell 2015-06-16 16:15 ` Dilger, Andreas 0 siblings, 1 reply; 7+ messages in thread From: Patrick Farrell @ 2015-06-16 15:28 UTC (permalink / raw) To: lustre-devel Andreas, Could you expound on "Better integration with DNE"? I'm not immediately clear on what that might mean. Thanks, - Patrick On 06/16/2015 10:21 AM, Dilger, Andreas wrote: > I think there are a few features that would be of interest for HSM > improvement. > > * optimization of RobinHood database/POSIX copytool > * better integration with DNE > * partial file restore using composite files > (http://wiki.opensfs.org/images/e/e4/LayoutEnhancementDesign_HighLevelDesig > n.pdf) > * change HSM to use LOV EA layout instead of dedicated xattr > * allowing a larger HSM archive ID (e.g. 128-bit UUID) so it can be > used as the primary archive identifier instead of Lustre FID > * archive/restore of directory trees from the MDT as tarballs > * allow release/restore of space from the MDT > * would be needed for handling Data-on-MDT files > > Note that we don't have any plans to work on this, just listing some ideas > percolating in my brain. > > > Cheers, Andreas ^ permalink raw reply [flat|nested] 7+ messages in thread
* [lustre-devel] HSM phase 2 HLD / Roadmap? 2015-06-16 15:28 ` Patrick Farrell @ 2015-06-16 16:15 ` Dilger, Andreas 2015-06-17 8:08 ` LEIBOVICI Thomas 0 siblings, 1 reply; 7+ messages in thread From: Dilger, Andreas @ 2015-06-16 16:15 UTC (permalink / raw) To: lustre-devel On 2015/06/16, 9:28 AM, "Patrick Farrell" <paf@cray.com> wrote: >Andreas, > >Could you expound on "Better integration with DNE"? I'm not immediately >clear on what that might mean. I was thinking that the current DNE support is still using a single RBH database fed by multiple ChangeLogs (one from each MDT), which doesn't scale very well. That said, I guess it would be possible to have a single parallel MySQL database that scales horizontally, so the scaling would be independent of the number of MDTs. Maybe this one should be dropped from the list, or just be considered part of the "optimize RBH database". Cheers, Andreas >Thanks, >- Patrick >On 06/16/2015 10:21 AM, Dilger, Andreas wrote: >> I think there are a few features that would be of interest for HSM >> improvement. >> >> * optimization of RobinHood database/POSIX copytool >> * better integration with DNE >> * partial file restore using composite files >> >>(http://wiki.opensfs.org/images/e/e4/LayoutEnhancementDesign_HighLevelDes >>ig >> n.pdf) >> * change HSM to use LOV EA layout instead of dedicated xattr >> * allowing a larger HSM archive ID (e.g. 128-bit UUID) so it can be >> used as the primary archive identifier instead of Lustre FID >> * archive/restore of directory trees from the MDT as tarballs >> * allow release/restore of space from the MDT >> * would be needed for handling Data-on-MDT files >> >> Note that we don't have any plans to work on this, just listing some >>ideas >> percolating in my brain. >> >> >> Cheers, Andreas > >_______________________________________________ >lustre-devel mailing list >lustre-devel at lists.lustre.org >http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > Cheers, Andreas -- Andreas Dilger Lustre Software Architect Intel High Performance Data Division ^ permalink raw reply [flat|nested] 7+ messages in thread
* [lustre-devel] HSM phase 2 HLD / Roadmap? 2015-06-16 16:15 ` Dilger, Andreas @ 2015-06-17 8:08 ` LEIBOVICI Thomas 0 siblings, 0 replies; 7+ messages in thread From: LEIBOVICI Thomas @ 2015-06-17 8:08 UTC (permalink / raw) To: lustre-devel On 06/16/15 18:15, Dilger, Andreas wrote: > On 2015/06/16, 9:28 AM, "Patrick Farrell" <paf@cray.com> wrote: > >> Andreas, >> >> Could you expound on "Better integration with DNE"? I'm not immediately >> clear on what that might mean. > I was thinking that the current DNE support is still using a single RBH > database fed by multiple ChangeLogs (one from each MDT), which doesn't > scale very well. That said, I guess it would be possible to have a single > parallel MySQL database that scales horizontally, so the scaling would be > independent of the number of MDTs. > > Maybe this one should be dropped from the list, or just be considered part > of the "optimize RBH database". > > Cheers, Andreas AFAIK, there are other aspects of DNE that HSM wouldn't support: HSM does not currently support striped directories (DNE phase 2) in particular the volatile file used for "restore" operations wouldn't be properly located on the right MDT. Thomas >> Thanks, >> - Patrick >> On 06/16/2015 10:21 AM, Dilger, Andreas wrote: >>> I think there are a few features that would be of interest for HSM >>> improvement. >>> >>> * optimization of RobinHood database/POSIX copytool >>> * better integration with DNE >>> * partial file restore using composite files >>> >>> (http://wiki.opensfs.org/images/e/e4/LayoutEnhancementDesign_HighLevelDes >>> ig >>> n.pdf) >>> * change HSM to use LOV EA layout instead of dedicated xattr >>> * allowing a larger HSM archive ID (e.g. 128-bit UUID) so it can be >>> used as the primary archive identifier instead of Lustre FID >>> * archive/restore of directory trees from the MDT as tarballs >>> * allow release/restore of space from the MDT >>> * would be needed for handling Data-on-MDT files >>> >>> Note that we don't have any plans to work on this, just listing some >>> ideas >>> percolating in my brain. >>> >>> >>> Cheers, Andreas >> _______________________________________________ >> lustre-devel mailing list >> lustre-devel at lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org >> > > Cheers, Andreas ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2015-06-17 8:08 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-06-16 15:02 [lustre-devel] HSM phase 2 HLD / Roadmap? Colin Faber 2015-06-16 15:08 ` DEGREMONT Aurelien 2015-06-16 15:10 ` Colin Faber 2015-06-16 15:21 ` Dilger, Andreas 2015-06-16 15:28 ` Patrick Farrell 2015-06-16 16:15 ` Dilger, Andreas 2015-06-17 8:08 ` LEIBOVICI Thomas
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.