Re: thoughts about fanotify and HSM

[Jan Kara <jack@suse.cz>]

Hello Robert!

On Mon 12-09-22 21:10:24, Plaster, Robert wrote:
> HI Jan – Our team has been using fanotify for HSM as a DMAPI replacement
> for a while now. We came from StorageTek/Sun/Oracle HSM product line
> development teams. We have been working on this for about 5 years and
> just this month are supporting end-users. DMAPI was a huge stumbling
> block for us. We figured out what minimum set of api calls were needed to
> make it work.
> 
> Our experience with fanotify has been fantastic. Not much overhead CPU
> load but for boot volumes we do filter out events for swap and other
> (there are so many) OS temp files that are really of no concern to HSM.
> We can create as many files as the file system on NVMe can without any
> back-pressure and the HSM process will go as fast as the target media
> supports.

I'm glad to hear fanotify is useful for you.

> We have tested close to 600M files per HSM client and we keep adding
> client files as time permits, we have no coded limits for the number of
> HSM clients or max number of files in the repository. Also, the
> repository for HSM clients is heterogenous so it allows us to push files
> from one client type to another without any transcoding. I asked the guys
> doing the actual fanotify part to comment but they said it would be a
> couple days as they are heads down on a fix for a customer.
> 
> Currently we have HSM and punch-hole running on xfs and tested it on zfs
> (works but client isn’t finalized) and we have Lustre and SpectrumScale
> on our to-do list. Basically any FS with extended attributes should work
> for HSM and some (not all) will work with punch-hole capabilities.
> 
> We have developed a HSM target for certain object stores (Ceph librados
> and we have our own in-house object store) that support stream-IO and of
> course any tape technology. We have a replication tool for making an S3
> target look like the source FS but its just replication, not HSM. Until
> we get a S3 io-streaming we can’t use it for HSM. Our implementation only
> works with our open-source catalog, archive platform. We tried to
> announce this capability to the ceph community but we could never get
> past their gatekeepers so only people we actually talk to know about it.
> 
> Check out our site (kinda sucks and a little markety) but it’s a good
> primer. In it are links to the code and manuals we have done. We have not
> put out on github yet but will very soon. We are getting ready to post
> some big updates to really simplify installation and configuration and
> some bug fixes for some weird edge-cases.

Thanks for info and the links! It is interesting to learn something about
how users are actually using our code :).

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR