Hi,
 I've been exploring an alternate approach to preparing lustre code for
 upstream submission.

 As you probably know, the approach we have been following is based on
 the code that was removed from 'drivers/staging' a couple of years ago.
 I reverted that removal patch and have ported all the relevant patches
 from lustre "master" across to that tree, as well as other cleanups and
 fixes.

 A problem with this approach is that I find that I don't trust it.
 I've found multiple examples of patches being ported across
 incorrectly, of patches being missed, and of bugs introduced while the
 code was in drivers/staging.
 I've found and fixed a lot of issues.  I doubt that I've found them
 all.
 While I do some testing of this code, as does James, it is minimal
 compared to the testing done on lustre/master.  And as we should all
 remember: untested code is buggy code.

 People who use lustre don't only want correctness and performance, they
 want confidence, and I'm not confident of this code.

 My alternate approach is to use the code straight out of lustre/master,
 transformed to match upstream requirements by a relatively simple
 script.  There is clearly still room for the script to introduce
 problems, but it is much less likely and any problems introduced are
 less likely to go unnoticed.

 I have written such a script, based largely on sed, unifdef, and spatch
 (the latter from the coccinelle project).

 Not all changes that I want can be achieved with these tools so I also
 have a collection of patches to lustre which hopefully will be
 submitted in due course.  Many are the same sorts of cleanups that I've
 been submitting for a while.  Others are a bit different.  For example
 I add a lot more "#ifdef HAVE_SERVER_SUPPORT" preprocessor directives
 so that "unifdef" can strip out more server-only code.  I've also added
 some "#ifndef UPSTREAM_LINUX" directives to remove code that cannot go
 upstream, like the code for extracting a jobid from the environment.

 You can see my current lustre tree at
   https://github.com/neilbrown/lustre/tree/lustre-next
 This contains 119 patches on top of master, some of which have already
 been submitted to gerrit, others could be easily, others need a lot of
 work first.
 The last patch adds the "copy-client" script.

 In
   https://github.com/neilbrown/linux/tree/lustre/lustre-next
 you can find a very recent linux/master tree with two patches added.
 The first was auto-generated by the script from lustre/lustre-next.
 The second fixes some compile errors with fscrypt() (though it may
 actually break the code, I don't know).

 Note that the copied LNET code is TCP-only.  I left out the o2ib code
 as I don't think we need that for the first code drop.

 Apart from providing the code mentioned above, this little project has
 also helped me form a clearer picture of what needs to be done before
 upstream submission is worth attempting.  My list includes:

 - IPv6 support.  I have a lot of patches which add support for extended
   NIDs which can store IPv6 addresses.  There is more to be done here
   (and thanks to Andreas for his review which I haven't responded to
   yet).
   One awkwardness in the conversion that I haven't addressed at all yet
   is ptlrpc/nodemap_range.c.  This allows a range of IP address to have
   a particular nodemap.  This might make sense for IPv4, but makes
   somewhat less sense for IPv6.
   I'm storing IPv6 addresses in network-byte-order (which is the normal
   way to store addresses), and that make ranges particularly cumbersome.
   One idea is to use netmasks rather than ranges to define subsets, for
   IPv6 at least.

 - fscrypt.  There is a comment in lustre-core.m4 which suggests that
    lustre does not support file name encryption, and so cannot use
    the fscrypt support in Linux.  If I understand that correctly, then
    this needs to be rectified.

 - I want to deprecate libcfs as there isn't anywhere convenient to put
   it - as a whole - in the linux kernel. It currently contains a
   mixture of things:
   = back-compat code, which Linux obviously doesn't needed

   - workitem.c and hash.c which are being replaced by
     workqueue and rhashables

   - cpt code, which is largely used by lnet, so can go in lnet/lnet

   - heap tree, which can go in lustre/ptlrpc as it is only used by nrs.

   - fail.c error injection.  It would be nice to convert lustre to
     use the error-injection in linux/lib.  This is a completely
     different user-space interface management.
     Is the current interface seen as a "stable api", or is it only
     used by lustre/tests (which can obviously be changed).

   - tracefile/debug.  I think this should be changed to use
     tracepoints. I believe the capabilities are similar, though the
     details are different.  I think I've raised this before and had an
     explanation of why people aren't keen.  Maybe it is time to revisit
     that.
     LU-8980 was closed "Won't Fix" with no explanatory comment...

   - libcfs_string - I've converted nearly all of this to use
     functionality already in Linux.  Only cfs_str2mask() remains.

   - adler32 - hopefully this can be copied to crypto/ in linux.

 - move stuff out of /proc
    I understand that part of this is statistics, which don't fit in
    debugfs as that is root-only, and don't fit in /sys at that is
    one-value-per-file.
    Firstly I'd like to resolve everything that is not stats.
    James has a netlink proposal for stats.
    NFS uses /proc/self/mountstats to present stats info.

 I plan to create some jira issues to cover:
  - moving code out of libcfs
  - fault injection
  - adding more HAVE_SERVER_SUPPORT and UPSTREAM_LINUX directives

 and start submitting some of the patches I have, and work on some more.

 I'll probably keep my existing linux-lustre tree (based on
 drivers/staging) up to date, as there is still valuable stuff in there
 to be ported across, and being able to 'diff' the two (with filtering)
 helps.

 Thoughts?

Thanks,
NeilBrown