Testing CXl with qemu and Linux

Testing CXl with qemu and Linux

[Raj Vishwanathan]

Raj Vishwanathan <raj.vishwanathan@gmail.com>
1:55 PM (1 hour ago)
to linux-cxl

I am trying to setup and take a look at an system with CXL enabled

1. I downloaded and installed linux.5.11 + CXL patches on a regular qemu system
2. I built qemu using Ben Widawsky's CXL-2.0 branch ( I see that there
is a 2.0v4 tree)
3. I also built ndactl  from 2.0v2 branch

From what I understand, I use the ndactl to create a  memory namespace
and I use this in the qemu command line to start. Given this and the
Linux having CXL support  I should be able to see the CXL.mem device
with lspci. Correct me if I am completely messed up.

I am not able to create the memory namespace with ndctl

I tried with

"./ndctl  create-namespace -t pmem -m raw -n mem0 -v -s 268435456" and
it fails with

"failed to create namespace: No such device or address"

Is there a step by step direction of how to test CXL on Linux? I can
document this process if someone can describe where am I making a
mistake

Raj

Re: Testing CXl with qemu and Linux

[Ben Widawsky]

On 21-03-08 15:33:49, Raj Vishwanathan wrote:
> Raj Vishwanathan <raj.vishwanathan@gmail.com>
> 1:55 PM (1 hour ago)
> to linux-cxl
> 
> I am trying to setup and take a look at an system with CXL enabled
> 
> 1. I downloaded and installed linux.5.11 + CXL patches on a regular qemu system
> 2. I built qemu using Ben Widawsky's CXL-2.0 branch ( I see that there
> is a 2.0v4 tree)
> 3. I also built ndactl  from 2.0v2 branch
> 
> From what I understand, I use the ndactl to create a  memory namespace
> and I use this in the qemu command line to start. Given this and the
> Linux having CXL support  I should be able to see the CXL.mem device
> with lspci. Correct me if I am completely messed up.
> 
> I am not able to create the memory namespace with ndctl
> 
> I tried with
> 
> "./ndctl  create-namespace -t pmem -m raw -n mem0 -v -s 268435456" and
> it fails with
> 
> "failed to create namespace: No such device or address"
> 
> Is there a step by step direction of how to test CXL on Linux? I can
> document this process if someone can describe where am I making a
> mistake
> 
> Raj

Aside from command errors, for which I'm not the expert...

The cxl command will be used to control CXL devices, not ndctl. Nothing other
than sending commands to the CXL device are actually hooked in in any of the:
application, kernel. QEMU has minimal support which may need more and I will be
looking into that.

Re: Testing CXl with qemu and Linux

[Verma, Vishal L]

On Thu, 2021-03-11 at 16:44 -0800, Ben Widawsky wrote:
> On 21-03-08 15:33:49, Raj Vishwanathan wrote:
> > Raj Vishwanathan <raj.vishwanathan@gmail.com>
> > 1:55 PM (1 hour ago)
> > to linux-cxl
> > 
> > I am trying to setup and take a look at an system with CXL enabled
> > 
> > 1. I downloaded and installed linux.5.11 + CXL patches on a regular qemu system
> > 2. I built qemu using Ben Widawsky's CXL-2.0 branch ( I see that there
> > is a 2.0v4 tree)
> > 3. I also built ndactl  from 2.0v2 branch
> > 
> > From what I understand, I use the ndactl to create a  memory namespace
> > and I use this in the qemu command line to start. Given this and the
> > Linux having CXL support  I should be able to see the CXL.mem device
> > with lspci. Correct me if I am completely messed up.
> > 
> > I am not able to create the memory namespace with ndctl
> > 
> > I tried with
> > 
> > "./ndctl  create-namespace -t pmem -m raw -n mem0 -v -s 268435456" and
> > it fails with
> > 
> > "failed to create namespace: No such device or address"
> > 
> > Is there a step by step direction of how to test CXL on Linux? I can
> > document this process if someone can describe where am I making a
> > mistake
> > 
> > Raj
> 
> Aside from command errors, for which I'm not the expert...
> 
> The cxl command will be used to control CXL devices, not ndctl. Nothing other
> than sending commands to the CXL device are actually hooked in in any of the:
> application, kernel. QEMU has minimal support which may need more and I will be
> looking into that.

In addition to what Ben said, the things that are added in the ndctl
branch you built are 'libcxl', and 'cxl-cli'. You can run 'cxl list' to
enumerate the cxl devices found, and libcxl has APIs to create and send
a partial subset of mailbox commands.

Eventually 'ndctl create-namespace' will be able to use a region coming
from a CXL device, but that functionality is still some time away.

Re: Testing CXl with qemu and Linux

[Raj Vishwanathan]

Vishal, Ben

Thanks for replying.

I am trying to understand who exposes these CXL devices. If I
understood correctly,  qemu, with the host OS support, will create and
expose a CXL.mem endpoint that will be discovered by the CXL root
complex driver on Linux ( thanks to the CXL patches)

What I am not able to do is to expose the CXL endpoint using Qemu.

Raj

On Thu, Mar 11, 2021 at 5:51 PM Verma, Vishal L
<vishal.l.verma@intel.com> wrote:
>
> On Thu, 2021-03-11 at 16:44 -0800, Ben Widawsky wrote:
> > On 21-03-08 15:33:49, Raj Vishwanathan wrote:
> > > Raj Vishwanathan <raj.vishwanathan@gmail.com>
> > > 1:55 PM (1 hour ago)
> > > to linux-cxl
> > >
> > > I am trying to setup and take a look at an system with CXL enabled
> > >
> > > 1. I downloaded and installed linux.5.11 + CXL patches on a regular qemu system
> > > 2. I built qemu using Ben Widawsky's CXL-2.0 branch ( I see that there
> > > is a 2.0v4 tree)
> > > 3. I also built ndactl  from 2.0v2 branch
> > >
> > > From what I understand, I use the ndactl to create a  memory namespace
> > > and I use this in the qemu command line to start. Given this and the
> > > Linux having CXL support  I should be able to see the CXL.mem device
> > > with lspci. Correct me if I am completely messed up.
> > >
> > > I am not able to create the memory namespace with ndctl
> > >
> > > I tried with
> > >
> > > "./ndctl  create-namespace -t pmem -m raw -n mem0 -v -s 268435456" and
> > > it fails with
> > >
> > > "failed to create namespace: No such device or address"
> > >
> > > Is there a step by step direction of how to test CXL on Linux? I can
> > > document this process if someone can describe where am I making a
> > > mistake
> > >
> > > Raj
> >
> > Aside from command errors, for which I'm not the expert...
> >
> > The cxl command will be used to control CXL devices, not ndctl. Nothing other
> > than sending commands to the CXL device are actually hooked in in any of the:
> > application, kernel. QEMU has minimal support which may need more and I will be
> > looking into that.
>
> In addition to what Ben said, the things that are added in the ndctl
> branch you built are 'libcxl', and 'cxl-cli'. You can run 'cxl list' to
> enumerate the cxl devices found, and libcxl has APIs to create and send
> a partial subset of mailbox commands.
>
> Eventually 'ndctl create-namespace' will be able to use a region coming
> from a CXL device, but that functionality is still some time away.
>

Re: Testing CXl with qemu and Linux

[Verma, Vishal L]

On Fri, 2021-03-12 at 08:55 -0800, Raj Vishwanathan wrote:
> Vishal, Ben
> 
> Thanks for replying.
> 
> I am trying to understand who exposes these CXL devices. If I
> understood correctly,  qemu, with the host OS support, will create and
> expose a CXL.mem endpoint that will be discovered by the CXL root
> complex driver on Linux ( thanks to the CXL patches)

The qemu branch you're using creates and exposes CXL devices when called
with the right cli arguments. The Linux kernel enabling then is able to
comprehend these devices, show them in sysfs, and is able to send down
mailbox commands.

> 
> What I am not able to do is to expose the CXL endpoint using Qemu.

You might take a look at this set of scripts that help create a working
environment for CXL development using qemu:
https://github.com/pmem/run_qemu

> 
> Raj
> 
> On Thu, Mar 11, 2021 at 5:51 PM Verma, Vishal L
> <vishal.l.verma@intel.com> wrote:
> > 
> > On Thu, 2021-03-11 at 16:44 -0800, Ben Widawsky wrote:
> > > On 21-03-08 15:33:49, Raj Vishwanathan wrote:
> > > > Raj Vishwanathan <raj.vishwanathan@gmail.com>
> > > > 1:55 PM (1 hour ago)
> > > > to linux-cxl
> > > > 
> > > > I am trying to setup and take a look at an system with CXL enabled
> > > > 
> > > > 1. I downloaded and installed linux.5.11 + CXL patches on a regular qemu system
> > > > 2. I built qemu using Ben Widawsky's CXL-2.0 branch ( I see that there
> > > > is a 2.0v4 tree)
> > > > 3. I also built ndactl  from 2.0v2 branch
> > > > 
> > > > From what I understand, I use the ndactl to create a  memory namespace
> > > > and I use this in the qemu command line to start. Given this and the
> > > > Linux having CXL support  I should be able to see the CXL.mem device
> > > > with lspci. Correct me if I am completely messed up.
> > > > 
> > > > I am not able to create the memory namespace with ndctl
> > > > 
> > > > I tried with
> > > > 
> > > > "./ndctl  create-namespace -t pmem -m raw -n mem0 -v -s 268435456" and
> > > > it fails with
> > > > 
> > > > "failed to create namespace: No such device or address"
> > > > 
> > > > Is there a step by step direction of how to test CXL on Linux? I can
> > > > document this process if someone can describe where am I making a
> > > > mistake
> > > > 
> > > > Raj
> > > 
> > > Aside from command errors, for which I'm not the expert...
> > > 
> > > The cxl command will be used to control CXL devices, not ndctl. Nothing other
> > > than sending commands to the CXL device are actually hooked in in any of the:
> > > application, kernel. QEMU has minimal support which may need more and I will be
> > > looking into that.
> > 
> > In addition to what Ben said, the things that are added in the ndctl
> > branch you built are 'libcxl', and 'cxl-cli'. You can run 'cxl list' to
> > enumerate the cxl devices found, and libcxl has APIs to create and send
> > a partial subset of mailbox commands.
> > 
> > Eventually 'ndctl create-namespace' will be able to use a region coming
> > from a CXL device, but that functionality is still some time away.
> > 

Re: Testing CXl with qemu and Linux

[Raj Vishwanathan]

Thanks, I'll check it out.

On Fri, Mar 12, 2021 at 4:29 PM Verma, Vishal L
<vishal.l.verma@intel.com> wrote:
>
> On Fri, 2021-03-12 at 08:55 -0800, Raj Vishwanathan wrote:
> > Vishal, Ben
> >
> > Thanks for replying.
> >
> > I am trying to understand who exposes these CXL devices. If I
> > understood correctly,  qemu, with the host OS support, will create and
> > expose a CXL.mem endpoint that will be discovered by the CXL root
> > complex driver on Linux ( thanks to the CXL patches)
>
> The qemu branch you're using creates and exposes CXL devices when called
> with the right cli arguments. The Linux kernel enabling then is able to
> comprehend these devices, show them in sysfs, and is able to send down
> mailbox commands.
>
> >
> > What I am not able to do is to expose the CXL endpoint using Qemu.
>
> You might take a look at this set of scripts that help create a working
> environment for CXL development using qemu:
> https://github.com/pmem/run_qemu
>
> >
> > Raj
> >
> > On Thu, Mar 11, 2021 at 5:51 PM Verma, Vishal L
> > <vishal.l.verma@intel.com> wrote:
> > >
> > > On Thu, 2021-03-11 at 16:44 -0800, Ben Widawsky wrote:
> > > > On 21-03-08 15:33:49, Raj Vishwanathan wrote:
> > > > > Raj Vishwanathan <raj.vishwanathan@gmail.com>
> > > > > 1:55 PM (1 hour ago)
> > > > > to linux-cxl
> > > > >
> > > > > I am trying to setup and take a look at an system with CXL enabled
> > > > >
> > > > > 1. I downloaded and installed linux.5.11 + CXL patches on a regular qemu system
> > > > > 2. I built qemu using Ben Widawsky's CXL-2.0 branch ( I see that there
> > > > > is a 2.0v4 tree)
> > > > > 3. I also built ndactl  from 2.0v2 branch
> > > > >
> > > > > From what I understand, I use the ndactl to create a  memory namespace
> > > > > and I use this in the qemu command line to start. Given this and the
> > > > > Linux having CXL support  I should be able to see the CXL.mem device
> > > > > with lspci. Correct me if I am completely messed up.
> > > > >
> > > > > I am not able to create the memory namespace with ndctl
> > > > >
> > > > > I tried with
> > > > >
> > > > > "./ndctl  create-namespace -t pmem -m raw -n mem0 -v -s 268435456" and
> > > > > it fails with
> > > > >
> > > > > "failed to create namespace: No such device or address"
> > > > >
> > > > > Is there a step by step direction of how to test CXL on Linux? I can
> > > > > document this process if someone can describe where am I making a
> > > > > mistake
> > > > >
> > > > > Raj
> > > >
> > > > Aside from command errors, for which I'm not the expert...
> > > >
> > > > The cxl command will be used to control CXL devices, not ndctl. Nothing other
> > > > than sending commands to the CXL device are actually hooked in in any of the:
> > > > application, kernel. QEMU has minimal support which may need more and I will be
> > > > looking into that.
> > >
> > > In addition to what Ben said, the things that are added in the ndctl
> > > branch you built are 'libcxl', and 'cxl-cli'. You can run 'cxl list' to
> > > enumerate the cxl devices found, and libcxl has APIs to create and send
> > > a partial subset of mailbox commands.
> > >
> > > Eventually 'ndctl create-namespace' will be able to use a region coming
> > > from a CXL device, but that functionality is still some time away.
> > >
>

Re: Testing CXl with qemu and Linux

[Raj Vishwanathan]

Vishal, Ben

I got it working. I can see the cxl mem devices using lspci.

Thanks for your pointers

Raj

On Fri, Mar 12, 2021 at 9:11 PM Raj Vishwanathan
<raj.vishwanathan@gmail.com> wrote:
>
> Thanks, I'll check it out.
>
> On Fri, Mar 12, 2021 at 4:29 PM Verma, Vishal L
> <vishal.l.verma@intel.com> wrote:
> >
> > On Fri, 2021-03-12 at 08:55 -0800, Raj Vishwanathan wrote:
> > > Vishal, Ben
> > >
> > > Thanks for replying.
> > >
> > > I am trying to understand who exposes these CXL devices. If I
> > > understood correctly,  qemu, with the host OS support, will create and
> > > expose a CXL.mem endpoint that will be discovered by the CXL root
> > > complex driver on Linux ( thanks to the CXL patches)
> >
> > The qemu branch you're using creates and exposes CXL devices when called
> > with the right cli arguments. The Linux kernel enabling then is able to
> > comprehend these devices, show them in sysfs, and is able to send down
> > mailbox commands.
> >
> > >
> > > What I am not able to do is to expose the CXL endpoint using Qemu.
> >
> > You might take a look at this set of scripts that help create a working
> > environment for CXL development using qemu:
> > https://github.com/pmem/run_qemu
> >
> > >
> > > Raj
> > >
> > > On Thu, Mar 11, 2021 at 5:51 PM Verma, Vishal L
> > > <vishal.l.verma@intel.com> wrote:
> > > >
> > > > On Thu, 2021-03-11 at 16:44 -0800, Ben Widawsky wrote:
> > > > > On 21-03-08 15:33:49, Raj Vishwanathan wrote:
> > > > > > Raj Vishwanathan <raj.vishwanathan@gmail.com>
> > > > > > 1:55 PM (1 hour ago)
> > > > > > to linux-cxl
> > > > > >
> > > > > > I am trying to setup and take a look at an system with CXL enabled
> > > > > >
> > > > > > 1. I downloaded and installed linux.5.11 + CXL patches on a regular qemu system
> > > > > > 2. I built qemu using Ben Widawsky's CXL-2.0 branch ( I see that there
> > > > > > is a 2.0v4 tree)
> > > > > > 3. I also built ndactl  from 2.0v2 branch
> > > > > >
> > > > > > From what I understand, I use the ndactl to create a  memory namespace
> > > > > > and I use this in the qemu command line to start. Given this and the
> > > > > > Linux having CXL support  I should be able to see the CXL.mem device
> > > > > > with lspci. Correct me if I am completely messed up.
> > > > > >
> > > > > > I am not able to create the memory namespace with ndctl
> > > > > >
> > > > > > I tried with
> > > > > >
> > > > > > "./ndctl  create-namespace -t pmem -m raw -n mem0 -v -s 268435456" and
> > > > > > it fails with
> > > > > >
> > > > > > "failed to create namespace: No such device or address"
> > > > > >
> > > > > > Is there a step by step direction of how to test CXL on Linux? I can
> > > > > > document this process if someone can describe where am I making a
> > > > > > mistake
> > > > > >
> > > > > > Raj
> > > > >
> > > > > Aside from command errors, for which I'm not the expert...
> > > > >
> > > > > The cxl command will be used to control CXL devices, not ndctl. Nothing other
> > > > > than sending commands to the CXL device are actually hooked in in any of the:
> > > > > application, kernel. QEMU has minimal support which may need more and I will be
> > > > > looking into that.
> > > >
> > > > In addition to what Ben said, the things that are added in the ndctl
> > > > branch you built are 'libcxl', and 'cxl-cli'. You can run 'cxl list' to
> > > > enumerate the cxl devices found, and libcxl has APIs to create and send
> > > > a partial subset of mailbox commands.
> > > >
> > > > Eventually 'ndctl create-namespace' will be able to use a region coming
> > > > from a CXL device, but that functionality is still some time away.
> > > >
> >