From mboxrd@z Thu Jan  1 00:00:00 1970
From: "Wiles, Keith" <keith.wiles@intel.com>
Subject: Re: [RFC] DFS (DPDK Filesystem)
Date: Tue, 27 Nov 2018 15:00:07 +0000
Message-ID: <897126DE-FC79-48FF-9B35-3E262EBDF093@intel.com>
References: <DD984D0C-526B-4859-BE9F-6D39564D9D8B@intel.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Cc: "Richardson, Bruce" <bruce.richardson@intel.com>, "Yigit, Ferruh"
 <ferruh.yigit@intel.com>, Thomas Monjalon <thomas@monjalon.net>, "Ananyev,
 Konstantin" <konstantin.ananyev@intel.com>
To: dpdk-dev <dev@dpdk.org>
Return-path: <dev-bounces@dpdk.org>
Received: from mga06.intel.com (mga06.intel.com [134.134.136.31])
 by dpdk.org (Postfix) with ESMTP id 01290326C
 for <dev@dpdk.org>; Tue, 27 Nov 2018 16:00:09 +0100 (CET)
In-Reply-To: <DD984D0C-526B-4859-BE9F-6D39564D9D8B@intel.com>
Content-Language: en-US
Content-ID: <B1C5CAC89E7E6E46988789B08F54C545@intel.com>
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>



> On Oct 8, 2018, at 5:42 PM, Wiles, Keith <keith.wiles@intel.com> wrote:
>=20
> Hi Everyone,
>=20
> I would like to request comments on DFS as I presented at the DPDK summit=
 in Ireland.
>=20
> As we know DPDK can be difficult to manage at run time and as DPDK become=
s more dynamic we need to address how to configure and monitor DPDK and DPD=
K-based applications. A possible solution is to use FUSE file system suppor=
t already in the Linux kernel through a DPDK library to expose a directory =
structure for anyone to be able to monitor and control the app by reading/w=
riting files.
>=20
> Primarily DPDK runs on Linux based systems, where FUSE is well supported.=
 FUSE was added to FreeBSD 10 which appears to be the same design as Linux.=
 Windows does not support FUSE directly from MS, but a open source design h=
as a FUSE based set of wrapper APIs to help port FUSE based file systems.
>=20
> Today the only way to get information out of DPDK is to use log files or =
a command line within DPDK to display the information. The information is n=
ot easy for an external application to collect or monitor. Using DFS any ap=
plication can read and write into this virtual file system (FUSE) and retri=
eve information from DPDK/Application via a raw ASCII text file and/or JSON=
 formatted text.
>=20
> Data read from DFS can be static or dynamic e.g. version/copyright text w=
ould be static, but port stats would be dynamic and generated when the user=
 requests the information. Writing new configuration data into DPDK can be =
a simple number or text or a JSON data structure. Writing simple ASCII text=
 data is simple, but writing a long JSON or configuration file may some car=
e, i.e. write a path to a file in DFS to the location of the configuration =
and use that file path for the configuration.
>=20
> DPDK command line is getting to be very complex and long, but we can elim=
inate or greatly reduce the command line by using DFS to configure DPDK at =
startup along with the application configuration. Having DPDK start up with=
 minimum resources and allow updating/creating a file in the DPDK file syst=
em to configure DPDK would be easier.
>=20
> The DFS is a FUSE filesystem anchored at /dpdk in the host file system, t=
hen a directory is created under /dpdk for each DPDK instance or applicatio=
n using <name>-<PID> as a directory name format. Then it is up to DPDK/Appl=
ication to create the directory structures, permissions and files within th=
e directory mount point.
>=20
> Currently I have a directory and files layout for DPDK, that needs more a=
ttention as I was focusing on FUSE then the layout as it can be changed fai=
rly easily. I can email the working documentation for DFS if needed please =
send me a short email or I can post to the dev list. The docs also try to e=
xplain the design and APIs used by the developer for creating files/directo=
ries. The docs are still a work in progress and more attention needs to be =
done as we progress.
>=20
> DFS is just a library and not required to be used plus it does not modify=
 DPDK APIs or structures except for adding a few new dump routines for feat=
ures that do not have one today. The file system or FUSE backend in DPDK is=
 a free floating thread at this time, but could be affinitized to a core if=
 required.
>=20
> The FUSE protocol to the backend code (in DPDK) has a large number of ope=
rations, but we have simplified the interface for developers to 5-6. The in=
terface on top of libfuse3 is very simple and currently provides most of th=
e needed APIs for managing the files and directories. The FUSE file system =
with the added simpler API is dynamic and can be changed at runtime. The di=
rectory/files can be constructed via APIs at startup or anytime after start=
up.
>=20
> I would like to put a repo someplace with DPDK/DFS say on GitHub to allow=
 others to play with the design in the short term. I also have a doc file f=
or DFS if anyone wants to learn more information and will be in the repo as=
 well. If we agree I can then create a set of patches for DPDK.

Well the 18.11 release is out the door, this one seemed pretty big and I lo=
t of hard work was put into this one, thanks to everyone.


It is time to try and get some feedback on this RFC. I know the code in the=
 repo is not finished, but I would like to get some folks to give it a try =
and give feedback. I would like to submit the basic DFS code with some low =
hanging items mostly readonly information for the 19.02 release.

Here is the repo again you can look at and play with. The code is mostly re=
ady for upstream but still more work needs to be done. I will try to update=
 the repo to the 18.11 release code based soon.

https://github.com/pktgen/dfs-dpdk

Thanks for giving this a review.

>=20
> Regards,
> Keith
>=20

Regards,
Keith