yet again the atomic operations

yet again the atomic operations

[Rui Machado]

Hi all,

I would like to know how do the IB atomic operations work and how much
can one rely on them? Particularly, how does the interaction with the
CPU takes place when for example the CPU is referencing and possibly
modifying the same address, atomically . Does one need to take care of
memory say, with memory barriers?
Does anyone has experience with this? There must be also different
hardware support from the vendors, right?

I hope I'm not sounding too vague, I'm not such a kernel or hw guy. I
tried to look for information on this but found pretty much nothing.
Can the experts shed some light on the problem?

Cheers,
Rui
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: yet again the atomic operations

[Ralph Campbell]

Atomic ops are optional. Mellanox and QLogic HCAs support them,
I don't know about the other HCAs.

There are two kinds supported. QLogic's driver does them in
the host driver so they are atomic with respect to all the CPUs
in the host. Mellanox uses HCA wide atomic which means the
HCA will do a memory read/write without allowing other reads
or writes from different QP operations passing through that
HCA to get in between. The CPUs on the host won't see
atomic operations since from their perspective, it looks
like a normal read and write from the PCIe bus.

You can see what type the HCA supports with "ibv_devinfo -v"
and look for "atomic_cap: ATOMIC_HCA (1)" or
"atomic_cap: ATOMIC_GLOB (2)".

On Wed, 2010-08-04 at 05:32 -0700, Rui Machado wrote:
> Hi all,
> 
> I would like to know how do the IB atomic operations work and how much
> can one rely on them? Particularly, how does the interaction with the
> CPU takes place when for example the CPU is referencing and possibly
> modifying the same address, atomically . Does one need to take care of
> memory say, with memory barriers?
> Does anyone has experience with this? There must be also different
> hardware support from the vendors, right?
> 
> I hope I'm not sounding too vague, I'm not such a kernel or hw guy. I
> tried to look for information on this but found pretty much nothing.
> Can the experts shed some light on the problem?
> 
> Cheers,
> Rui
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: yet again the atomic operations

[Rui Machado]

Hi there,

> There are two kinds supported. QLogic's driver does them in
> the host driver so they are atomic with respect to all the CPUs
> in the host. Mellanox uses HCA wide atomic which means the
> HCA will do a memory read/write without allowing other reads
> or writes from different QP operations passing through that
> HCA to get in between. The CPUs on the host won't see
> atomic operations since from their perspective, it looks
> like a normal read and write from the PCIe bus.

So if the CPU writes/reads to/from the same address, even atomically
(lock), there might be room for some inconsistency on the values? It
is not really atomic from the whole system point of view, just for the
HCA? If so, is there any possibility to make the whole operation
'system-wide' atomic?


> You can see what type the HCA supports with "ibv_devinfo -v"
> and look for "atomic_cap: ATOMIC_HCA (1)" or
> "atomic_cap: ATOMIC_GLOB (2)".

ATOMIC_HCA (1) is what I see in my Mellanox hardware. This is the case
you mentioned, "without allowing other reads or writes from different
QP operations passing through that HCA to get in between"
ATOMIC_GLOB (2) means with respect to all HCAs and even the CPU?

Cheers,
Rui
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: yet again the atomic operations

[Roland Dreier]

 > So if the CPU writes/reads to/from the same address, even atomically
 > (lock), there might be room for some inconsistency on the values? It
 > is not really atomic from the whole system point of view, just for the
 > HCA? If so, is there any possibility to make the whole operation
 > 'system-wide' atomic?

PCI does not have any capability for atomic operations until PCI Express
3.0 (not available in any real devices yet).  So any current HCA
performing atomic operations across a PCI bus will always have to do
read-modify-write which leaves a window for the CPU to mess things up if
it accesses the same location.

You can work around this by creating a loopback connection (ie an RC
connection from the local HCA to itself) and post atomic operations to
that QP instead of accessing the memory directly with the CPU.

 - R.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: yet again the atomic operations

[Ralph Campbell]

On Fri, 2010-08-06 at 04:43 -0700, Rui Machado wrote:
> Hi there,
> 
> > There are two kinds supported. QLogic's driver does them in
> > the host driver so they are atomic with respect to all the CPUs
> > in the host. Mellanox uses HCA wide atomic which means the
> > HCA will do a memory read/write without allowing other reads
> > or writes from different QP operations passing through that
> > HCA to get in between. The CPUs on the host won't see
> > atomic operations since from their perspective, it looks
> > like a normal read and write from the PCIe bus.
> 
> So if the CPU writes/reads to/from the same address, even atomically
> (lock), there might be room for some inconsistency on the values? It
> is not really atomic from the whole system point of view, just for the
> HCA? If so, is there any possibility to make the whole operation
> 'system-wide' atomic?

Correct.
It won't be consistent from the HCA's point of view if other HCAs
or CPUs are modifying the memory - even if they do it atomically.
It is only consistent if a single HCA is doing atomic ops to the
memory.

There is no possibility to change this unless PCIe atomic
operations are used by the HCA and if the root complex supports
atomic operations. I don't know of any HCAs or root complex
chips which have this support yet.

> > You can see what type the HCA supports with "ibv_devinfo -v"
> > and look for "atomic_cap: ATOMIC_HCA (1)" or
> > "atomic_cap: ATOMIC_GLOB (2)".
> 
> ATOMIC_HCA (1) is what I see in my Mellanox hardware. This is the case
> you mentioned, "without allowing other reads or writes from different
> QP operations passing through that HCA to get in between"
> ATOMIC_GLOB (2) means with respect to all HCAs and even the CPU?

Correct.

> Cheers,
> Rui
> 


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: yet again the atomic operations

[Rui Machado]

> There are two kinds supported. QLogic's driver does them in
> the host driver so they are atomic with respect to all the CPUs
> in the host.

I'm just curious about this: how does this work? Is the CPU getting
interrupted and doing the operation while the Mellanox HCA does
everything in hardware?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: yet again the atomic operations

[Rui Machado]

>
> You can work around this by creating a loopback connection (ie an RC
> connection from the local HCA to itself) and post atomic operations to
> that QP instead of accessing the memory directly with the CPU.
>
Right but that's really slow, specially if you're implementing some
sort of synchronization scheme that happens really often.
Isn't it possible to do something such as interrupting the CPU and
have it do the operation? For an atomic operation (small data) maybe
the overhead is not that large?!

Cheers,
Rui
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: yet again the atomic operations

[Ralph Campbell]

On Tue, 2010-08-10 at 04:46 -0700, Rui Machado wrote:
> > There are two kinds supported. QLogic's driver does them in
> > the host driver so they are atomic with respect to all the CPUs
> > in the host.
> 
> I'm just curious about this: how does this work? Is the CPU getting
> interrupted and doing the operation while the Mellanox HCA does
> everything in hardware?

Yes.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html