[lustre-devel] HSM issues

[lustre-devel] HSM issues

[Nathan Rutman]

Hi all -
I have a number of nagging concerns about current HSM implementation; maybe
I'm just out of date, but I figure this is the place to ask:
1. Changelog size limits. Can changelogs still grow unbounded, resulting in
ENOSPC (or worse) on the MDS? Should there be a size limit? What should be
done at that limit -- stop recording changelogs? Turn FS read-only?
2. Coordinator queue limit. Can coordinator queue grow unbounded? Can we
add some throttling from the coordinator to the PE, maybe an -EAGAIN if the
coordinator queue is large?
3. Error-condition passthrough from hsmtool back to PE. Backend may have
e.g. ENOSPC, reported back to coordinator, but then what? Can future PE
requests be denied by the coordinator with an ENOSPC, presumably prompting
Robinhood to issue hsm_remove commands? ENOSPC should continue to be
returned, until some other rv is returned by copytool.
4. Coordinator should sort incoming requests so that "restores" and
"removes" are placed before "archives". Restores are the highest priority
from user point of view, and removes are next from a space available point
of view.

*--*

*Nathan Rutman ? Principal Systems ArchitectSeagate Technology** ? *+1 503
877-9507* ? *GMT-8
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160623/b4b32168/attachment.htm>

[lustre-devel] HSM issues

[Dilger, Andreas]

The ChangeLogs do have limits on the number of records, but hitting that limit will _also_ result in ENOSPC, as would also happen if the MDT actually runs out of space.  Alex has a patch in flight that reduces the maximum llog size if the MDT is running out of space, so that the old logs can be closed and cleaned up rather than continuing to grow even though they are mostly empty.  There was also a patch landed from Artem to add a /proc file that reports the current ChangeLog size to userspace so that this can be tracked.

We did discuss what should happen if the ChangeLog is running out of space for new records (e.g. because some consumer is not consuming).  It should be tunable by the admin, either to drop the consumer and delete all of the records for that consumer, or to prevent new operations being performed in the filesystem (e.g. if it was critical to not lose any records for some reason).  I'd suspect the first option would be the better default.

No idea about the other items.  Hopefully someone else can reply.

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel High Performance Data Division

On 2016/06/23, 18:10, "Nathan Rutman" <nathan.rutman at seagate.com<mailto:nathan.rutman@seagate.com>> wrote:

Hi all -
I have a number of nagging concerns about current HSM implementation; maybe I'm just out of date, but I figure this is the place to ask:
1. Changelog size limits. Can changelogs still grow unbounded, resulting in ENOSPC (or worse) on the MDS? Should there be a size limit? What should be done at that limit -- stop recording changelogs? Turn FS read-only?
2. Coordinator queue limit. Can coordinator queue grow unbounded? Can we add some throttling from the coordinator to the PE, maybe an -EAGAIN if the coordinator queue is large?
3. Error-condition passthrough from hsmtool back to PE. Backend may have e.g. ENOSPC, reported back to coordinator, but then what? Can future PE requests be denied by the coordinator with an ENOSPC, presumably prompting Robinhood to issue hsm_remove commands? ENOSPC should continue to be returned, until some other rv is returned by copytool.
4. Coordinator should sort incoming requests so that "restores" and "removes" are placed before "archives". Restores are the highest priority from user point of view, and removes are next from a space available point of view.

--
Nathan Rutman ? Principal Systems Architect
Seagate Technology ? +1 503 877-9507 ? GMT-8
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160624/f582d4ba/attachment-0001.htm>

[lustre-devel] HSM issues

[Alexander I Kulyavtsev]

On Jun 23, 2016, at 7:10 PM, Nathan Rutman <nathan.rutman at seagate.com<mailto:nathan.rutman@seagate.com>> wrote:

4. Coordinator should sort incoming requests so that "restores" and "removes" are placed before "archives". Restores are the highest priority from user point of view, and removes are next from a space available point of view.

Restores fill the space on lustre, "archives" with following "purge" release space on lustre.
In situation when the file system is "almost" full it can be desirable to release space with higher priority, i.e. priority may change when space is limited.
Not all restores are equal. It may suffice to have few priority levels: admin/interactive, normal, background (bulk IO).

Tape HSM backend may have it's own priority preferences (due to slow tape mount/rewind ops) and the tape backend may do request reordering to optimize tape access. E.g. tape reads/write go to different hardware types(write to new technology tapes, read old data from old tapes) and one of them busy.  Or, write and read requests come from different user groups, one of them has max mounted tapes but the other can continue.

In this case it can be better to throw  many requests (tens of thousands) to backend and let it figure out (delegate request prioritization and QoS to backend).  The queue depth shall be configurable as the other simpler backend may not survive.

Some workloads like data migration from old tapes to new tapes may not be happy if lustre HSM and backend tape system have conflicting priorities. There shall be configurable option to set the preference or noop.

Alex.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160624/487952bb/attachment.htm>

[lustre-devel] HSM issues

[Quentin BOUGET]

I am working on issue 4: Coordinator should sort incoming requests so 
that "restores" and "removes" are placed before "archives". Restores are 
the highest priority from user point of view, and removes are next from 
a space available point of

Here is the related ticket LU-8324: 
https://jira.hpdd.intel.com/browse/LU-8324

On Jun 23, 2016, at 7:10 PM, Nathan Rutman <nathan.rutman at seagate.com 
<mailto:nathan.rutman@seagate.com>> wrote:
> 4. Coordinator should sort incoming requests so that "restores" and 
> "removes" are placed before "archives". Restores are the highest 
> priority from user point of view, and removes are next from a space 
> available point of view.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160624/37cc0b83/attachment.htm>