Port mirroring (RFC)

Port mirroring (RFC)

[Alex Elder]

I am implementing what amounts to port mirroring functionality
for the IPA driver.

The IPA hardware isn't exactly a network switch (it's sort of
more than that), but it has the ability to supply replicas of
packets transferred within it to a special (read only) interface.

My plan is to implement this using a new "ipa_mirror" network
device, so it could be used with a raw socket to capture the
arriving packets.  There currently exists one other netdev,
which represents access through a modem to a WWAN network.

I would like some advice on how to proceed with this.  I want
the result to match "best practice" upstream, and would like
this to be as well integrated possible with existing network
tools.

A few details about the stream of packets that arrive on
this hardware interface:
- Packet data is truncated if it's larger than a certain size
- Each packet is preceded by a fixed-size header describing it
- Packets (and their headers) are aggregated into a buffer; i.e.
   a single receive might carry a dozen (truncated) packets

Here are a few specific questions, but I would love to get
*any* feedback about what I'm doing.
- Is representing this as a separate netdev a reasonable
   thing to do?
- Is there anything wrong with making a netdev read-only?
   (Any packets supplied for transmit would be dropped)
- Are there things I should do so it's clear this interface
   does not carry IP traffic (or even UDP, etc.)?
- Should the driver de-aggregate the received packets, i.e.
   separating each into a separate SKB for reading?

I might have *many* more questions, but I'd just like to make
sure I'm on the right track, and would like both specific and
general suggestions about how to do this the right way.

Thanks.
					-Alex

Re: Port mirroring (RFC)

[Andrew Lunn]

On Tue, Dec 14, 2021 at 08:47:12AM -0600, Alex Elder wrote:
> I am implementing what amounts to port mirroring functionality
> for the IPA driver.
> 
> The IPA hardware isn't exactly a network switch (it's sort of
> more than that), but it has the ability to supply replicas of
> packets transferred within it to a special (read only) interface.

I think you need to explain "within it" in a bit more detail. Where
are these packets coming from/going to?

> My plan is to implement this using a new "ipa_mirror" network
> device, so it could be used with a raw socket to capture the
> arriving packets.  There currently exists one other netdev,
> which represents access through a modem to a WWAN network.
> 
> I would like some advice on how to proceed with this.  I want
> the result to match "best practice" upstream, and would like
> this to be as well integrated possible with existing network
> tools.
> 
> A few details about the stream of packets that arrive on
> this hardware interface:
> - Packet data is truncated if it's larger than a certain size
> - Each packet is preceded by a fixed-size header describing it
> - Packets (and their headers) are aggregated into a buffer; i.e.
>   a single receive might carry a dozen (truncated) packets

So this sounds something more like what you would attach pcap/tcpdump
to.  I'm not sure port mirroring is the correct model here. Maybe take
a look at wifi adaptors and their monitor mode? See if that fits
better?
 
	Andrew

Re: Port mirroring (RFC)

[Alex Elder]

On 12/14/21 12:27 PM, Andrew Lunn wrote:
> On Tue, Dec 14, 2021 at 08:47:12AM -0600, Alex Elder wrote:
>> I am implementing what amounts to port mirroring functionality
>> for the IPA driver.
>>
>> The IPA hardware isn't exactly a network switch (it's sort of
>> more than that), but it has the ability to supply replicas of
>> packets transferred within it to a special (read only) interface.
> 
> I think you need to explain "within it" in a bit more detail. Where
> are these packets coming from/going to?

Sorry, I didn't want to dive into too much detail up front.

IPA is a device that sits between the main CPU and a modem,
carrying WWAN network data between them.

In addition, there is a small number of other entities that
could be reachable through the IPA hardware, such as a WiFi
device providing access to a WLAN.

Packets can travel "within IPA" between any of these
"connected entities."  So far only the path between the
AP and the modem is supported upstream, but I'm working
on enabling more capability.

Technically, the replicated packets aren't visible on
any one port; the only way to see that traffic is in
using this special port.  To me this seemed like port
mirroring, which is why I suggested that.  I'm want to
use the proper model though, so I appreciate your
response.

>> My plan is to implement this using a new "ipa_mirror" network
>> device, so it could be used with a raw socket to capture the
>> arriving packets.  There currently exists one other netdev,
>> which represents access through a modem to a WWAN network.
>>
>> I would like some advice on how to proceed with this.  I want
>> the result to match "best practice" upstream, and would like
>> this to be as well integrated possible with existing network
>> tools.
>>
>> A few details about the stream of packets that arrive on
>> this hardware interface:
>> - Packet data is truncated if it's larger than a certain size
>> - Each packet is preceded by a fixed-size header describing it
>> - Packets (and their headers) are aggregated into a buffer; i.e.
>>    a single receive might carry a dozen (truncated) packets
> 
> So this sounds something more like what you would attach pcap/tcpdump
> to.  I'm not sure port mirroring is the correct model here. Maybe take
> a look at wifi adaptors and their monitor mode? See if that fits
> better?

Yes, pcap and tcpdump are exactly the model I envisioned.  I had
heard of monitoring but hadn't looked at it closely, so I will.

Thanks a lot for the suggestion.

					-Alex
>   
> 	Andrew
> 

Re: Port mirroring (RFC)

[Andrew Lunn]

> IPA is a device that sits between the main CPU and a modem,
> carrying WWAN network data between them.
> 
> In addition, there is a small number of other entities that
> could be reachable through the IPA hardware, such as a WiFi
> device providing access to a WLAN.
> 
> Packets can travel "within IPA" between any of these
> "connected entities."  So far only the path between the
> AP and the modem is supported upstream, but I'm working
> on enabling more capability.
> 
> Technically, the replicated packets aren't visible on
> any one port; the only way to see that traffic is in
> using this special port.  To me this seemed like port
> mirroring, which is why I suggested that.  I'm want to
> use the proper model though, so I appreciate your
> response.

Do you have netdevs for the modem, the wifi, and whatever other
interfaces the hardware might have?

To setup a mirror you would do something like:

sudo tc filter add dev eth0 parent ffff: protocol all u32 match u32 0 0 action mirred egress mirror dev tun0

where you are mirroring eth0 to tun0. eth0 would have to be your modem
netdev, or your wifi netdev, and tun0 would be your monitor device.

If you do have a netdev on the host for each of these network
interfaces, mirroring could work. Architecturally, it would make sense
to have these netdevs, so you can run wpa_supplicant on the wifi
interface to do authentication, etc.

Do you have control over selecting egress and ingress packets to be
mirrored?

	Andrew

Re: Port mirroring (RFC)

[Alex Elder]

On 12/15/21 3:18 AM, Andrew Lunn wrote:
>> IPA is a device that sits between the main CPU and a modem,
>> carrying WWAN network data between them.
>>
>> In addition, there is a small number of other entities that
>> could be reachable through the IPA hardware, such as a WiFi
>> device providing access to a WLAN.
>>
>> Packets can travel "within IPA" between any of these
>> "connected entities."  So far only the path between the
>> AP and the modem is supported upstream, but I'm working
>> on enabling more capability.
>>
>> Technically, the replicated packets aren't visible on
>> any one port; the only way to see that traffic is in
>> using this special port.  To me this seemed like port
>> mirroring, which is why I suggested that.  I'm want to
>> use the proper model though, so I appreciate your
>> response.
> 
> Do you have netdevs for the modem, the wifi, and whatever other
> interfaces the hardware might have?

Not yet, but yes I expect that's how it will work.

> To setup a mirror you would do something like:
> 
> sudo tc filter add dev eth0 parent ffff: protocol all u32 match u32 0 0 action mirred egress mirror dev tun0

OK so it sounds like the term "mirror" means mirroring using
Linux filtering.  And then I suppose "monitoring" is collecting
all "observed" traffic through an interface?

If that's the case, this seems to me more like monitoring, except
I suggested presenting the replicated data through a separate
netdev (rather than, for example, through the one for the modem).

If it makes more sense, I could probably inject the replicated
packets received through this special interface into one or
another of the existing netdevs, rather than using a separate
one for this purpose.

> where you are mirroring eth0 to tun0. eth0 would have to be your modem
> netdev, or your wifi netdev, and tun0 would be your monitor device.
> 
> If you do have a netdev on the host for each of these network
> interfaces, mirroring could work. Architecturally, it would make sense
> to have these netdevs, so you can run wpa_supplicant on the wifi
> interface to do authentication, etc.
> 
> Do you have control over selecting egress and ingress packets to be
> mirrored?

That I'm not sure about.  If it's possible, it would be controlling
which originators have their traffic replicated.

I don't think it will take me all that long to implement this, but
my goal right now is to be sure that the design I implement is a good
solution.  I'm open to recommendations.

Thanks.

					-Alex

> 	Andrew
> 

Re: Port mirroring (RFC)

[Andrew Lunn]

> > Do you have netdevs for the modem, the wifi, and whatever other
> > interfaces the hardware might have?
> 
> Not yet, but yes I expect that's how it will work.
> 
> > To setup a mirror you would do something like:
> > 
> > sudo tc filter add dev eth0 parent ffff: protocol all u32 match u32 0 0 action mirred egress mirror dev tun0
> 
> OK so it sounds like the term "mirror" means mirroring using
> Linux filtering.  And then I suppose "monitoring" is collecting
> all "observed" traffic through an interface?

Yes, that seems like a good description of the difference. 
 
> If that's the case, this seems to me more like monitoring, except
> I suggested presenting the replicated data through a separate
> netdev (rather than, for example, through the one for the modem).

The wifi model allows you to dynamical add netdev on top of a physical
wireless LAN chipset. So you can have one netdev running as an access
point, and a second netdev running as a client, both sharing the
underlying hardware. And you should be able to add another netdev and
put it into monitor mode. So having a dedicated netdev for your
monitoring is not too far away from what you do with wifi.

> If it makes more sense, I could probably inject the replicated
> packets received through this special interface into one or
> another of the existing netdevs, rather than using a separate
> one for this purpose.

> > Do you have control over selecting egress and ingress packets to be
> > mirrored?
> 
> That I'm not sure about.  If it's possible, it would be controlling
> which originators have their traffic replicated.

You need this if you want to do mirroring, since the API requires to
say if you want to mirror ingress or egress. WiFi monitoring is less
specific as far as i understand. It is whatever is received on the
antenna.

> I don't think it will take me all that long to implement this, but
> my goal right now is to be sure that the design I implement is a good
> solution.  I'm open to recommendations.

You probably want to look at what wifi monitor offers. And maybe check
with the WiFi people what they actually think about monitoring, or if
they have a better suggestion.

     Andrew

Re: Port mirroring (RFC)

[Florian Fainelli]

On 12/14/21 6:47 AM, Alex Elder wrote:
> I am implementing what amounts to port mirroring functionality
> for the IPA driver.
> 
> The IPA hardware isn't exactly a network switch (it's sort of
> more than that), but it has the ability to supply replicas of
> packets transferred within it to a special (read only) interface.
> 
> My plan is to implement this using a new "ipa_mirror" network
> device, so it could be used with a raw socket to capture the
> arriving packets.  There currently exists one other netdev,
> which represents access through a modem to a WWAN network.
> 
> I would like some advice on how to proceed with this.  I want
> the result to match "best practice" upstream, and would like
> this to be as well integrated possible with existing network
> tools.
> 
> A few details about the stream of packets that arrive on
> this hardware interface:
> - Packet data is truncated if it's larger than a certain size
> - Each packet is preceded by a fixed-size header describing it
> - Packets (and their headers) are aggregated into a buffer; i.e.
>   a single receive might carry a dozen (truncated) packets

Andrew has already responded, but there are currently sort of two ways
that mirroring is implemented in upstream supported drivers:

- for "classic" Ethernet switches mirroring is typically from one port,
or a set of ports, to another with a choice in whether you want to
capture ingress, egress, or both towards that port. Because each port is
a netdev already you just run tcpdump/pcap the way you normally do on
your Ethernet NIC and get the captured packets. Configuration is via
offloading the tc_mired action.

- mlxsw implements devlink traps which is not exaclty designed for
capturing mirrored packets but more like management events such as why
the packet was trapped etc. You could however imagine using devlink
traps for that purpose of capturing mirrored packets in the absence of a
suitable network device

Not sure if this helps, but that is the situation.

> 
> Here are a few specific questions, but I would love to get
> *any* feedback about what I'm doing.
> - Is representing this as a separate netdev a reasonable
>   thing to do?

I would think so, given this allows standard tools to work.

> - Is there anything wrong with making a netdev read-only?
>   (Any packets supplied for transmit would be dropped)

This looks reasonable.

> - Are there things I should do so it's clear this interface
>   does not carry IP traffic (or even UDP, etc.)?

There were various patches in the past to prevent a network device from
getting any IP stack to be configured:

https://yhbt.net/lore/all/20150825232021.GA8482@Alexeis-MacBook-Pro-2.local/t/

I forgot whether as a driver you can just refuse to have an IP address
assigned to your network device, AFAIR it requires changes to the
network stack as proposed in the patch set.

> - Should the driver de-aggregate the received packets, i.e.
>   separating each into a separate SKB for reading?

If this is truly a mirroring device, you would expect it to "render" the
mirrored packets exactly the way they are, maybe add an ethtool private
option to de-aggregate packets if this helps debugging?

> 
> I might have *many* more questions, but I'd just like to make
> sure I'm on the right track, and would like both specific and
> general suggestions about how to do this the right way.
> 
> Thanks.
>                     -Alex


-- 
Florian

Re: Port mirroring (RFC)

[Florian Fainelli]

On 12/15/21 6:47 AM, Alex Elder wrote:
> On 12/15/21 3:18 AM, Andrew Lunn wrote:
>>> IPA is a device that sits between the main CPU and a modem,
>>> carrying WWAN network data between them.
>>>
>>> In addition, there is a small number of other entities that
>>> could be reachable through the IPA hardware, such as a WiFi
>>> device providing access to a WLAN.
>>>
>>> Packets can travel "within IPA" between any of these
>>> "connected entities."  So far only the path between the
>>> AP and the modem is supported upstream, but I'm working
>>> on enabling more capability.
>>>
>>> Technically, the replicated packets aren't visible on
>>> any one port; the only way to see that traffic is in
>>> using this special port.  To me this seemed like port
>>> mirroring, which is why I suggested that.  I'm want to
>>> use the proper model though, so I appreciate your
>>> response.
>>
>> Do you have netdevs for the modem, the wifi, and whatever other
>> interfaces the hardware might have?
> 
> Not yet, but yes I expect that's how it will work.
> 
>> To setup a mirror you would do something like:
>>
>> sudo tc filter add dev eth0 parent ffff: protocol all u32 match u32 0
>> 0 action mirred egress mirror dev tun0
> 
> OK so it sounds like the term "mirror" means mirroring using
> Linux filtering.  And then I suppose "monitoring" is collecting
> all "observed" traffic through an interface?

It is mirroring in terms of an action to perform for a given packet
having been matched, now Ethernet switches for instance support
mirroring in hardware and that specific action can be offloaded down to
your hardware. You can take a look at net/dsa/* and drivers/net/dsa/ for
an example of how this is done.

> 
> If that's the case, this seems to me more like monitoring, except
> I suggested presenting the replicated data through a separate
> netdev (rather than, for example, through the one for the modem).
> 
> If it makes more sense, I could probably inject the replicated
> packets received through this special interface into one or
> another of the existing netdevs, rather than using a separate
> one for this purpose.
> 
>> where you are mirroring eth0 to tun0. eth0 would have to be your modem
>> netdev, or your wifi netdev, and tun0 would be your monitor device.
>>
>> If you do have a netdev on the host for each of these network
>> interfaces, mirroring could work. Architecturally, it would make sense
>> to have these netdevs, so you can run wpa_supplicant on the wifi
>> interface to do authentication, etc.
>>
>> Do you have control over selecting egress and ingress packets to be
>> mirrored?
> 
> That I'm not sure about.  If it's possible, it would be controlling
> which originators have their traffic replicated.

And the originators would be represented as network devices, or would
they be another kind of object in the Linux kernel? If they are network
devices then you can use the example from Andrew above because you
basically define the source device to mirror. Else, you may have to
invent your own thing.

> 
> I don't think it will take me all that long to implement this, but
> my goal right now is to be sure that the design I implement is a good
> solution.  I'm open to recommendations.
> 
> Thanks.
> 
>                     -Alex
> 
>>     Andrew
>>
> 


-- 
Florian

Re: Port mirroring (RFC)

[Jakub Kicinski]

On Tue, 14 Dec 2021 08:47:12 -0600 Alex Elder wrote:
> I am implementing what amounts to port mirroring functionality
> for the IPA driver.
> 
> The IPA hardware isn't exactly a network switch (it's sort of
> more than that), but it has the ability to supply replicas of
> packets transferred within it to a special (read only) interface.
> 
> My plan is to implement this using a new "ipa_mirror" network
> device, so it could be used with a raw socket to capture the
> arriving packets.  There currently exists one other netdev,
> which represents access through a modem to a WWAN network.
> 
> I would like some advice on how to proceed with this.  I want
> the result to match "best practice" upstream, and would like
> this to be as well integrated possible with existing network
> tools.
> 
> A few details about the stream of packets that arrive on
> this hardware interface:
> - Packet data is truncated if it's larger than a certain size
> - Each packet is preceded by a fixed-size header describing it
> - Packets (and their headers) are aggregated into a buffer; i.e.
>    a single receive might carry a dozen (truncated) packets
> 
> Here are a few specific questions, but I would love to get
> *any* feedback about what I'm doing.
> - Is representing this as a separate netdev a reasonable
>    thing to do?
> - Is there anything wrong with making a netdev read-only?
>    (Any packets supplied for transmit would be dropped)
> - Are there things I should do so it's clear this interface
>    does not carry IP traffic (or even UDP, etc.)?
> - Should the driver de-aggregate the received packets, i.e.
>    separating each into a separate SKB for reading?
> 
> I might have *many* more questions, but I'd just like to make
> sure I'm on the right track, and would like both specific and
> general suggestions about how to do this the right way.

Maybe the first question to ask is - why do you need this?
Or perhaps - how is it used? There's a significant difference 
between an interface for users and a debug interface.

Do you aim to give users control over the forwarding which happens
on the application processor at some point? If so Andrew and Florian
give great suggestions but starting from debugging of the forwarding
feels a little backward.

Re: Port mirroring (RFC)

[Alex Elder]

On 12/15/21 11:42 AM, Andrew Lunn wrote:
>>> Do you have netdevs for the modem, the wifi, and whatever other
>>> interfaces the hardware might have?
>>
>> Not yet, but yes I expect that's how it will work.
>>
>>> To setup a mirror you would do something like:
>>>
>>> sudo tc filter add dev eth0 parent ffff: protocol all u32 match u32 0 0 action mirred egress mirror dev tun0
>>
>> OK so it sounds like the term "mirror" means mirroring using
>> Linux filtering.  And then I suppose "monitoring" is collecting
>> all "observed" traffic through an interface?
> 
> Yes, that seems like a good description of the difference.
>   
>> If that's the case, this seems to me more like monitoring, except
>> I suggested presenting the replicated data through a separate
>> netdev (rather than, for example, through the one for the modem).
> 
> The wifi model allows you to dynamical add netdev on top of a physical
> wireless LAN chipset. So you can have one netdev running as an access
> point, and a second netdev running as a client, both sharing the
> underlying hardware. And you should be able to add another netdev and
> put it into monitor mode. So having a dedicated netdev for your
> monitoring is not too far away from what you do with wifi.

It sound to me like WiFi monitoring mode could very much be
a model that would work.  I need to spend some time looking
at that in a little more detail.  I don't think there's any
reason the "dedicated" netdev couldn't be created dynamically.

I'll come back again after I've had a chance to look at these
suggestions (yours and others'), possibly with something closer
to a design to follow.

Thank you very much, this is a promising lead.

					-Alex

>> If it makes more sense, I could probably inject the replicated
>> packets received through this special interface into one or
>> another of the existing netdevs, rather than using a separate
>> one for this purpose.
> 
>>> Do you have control over selecting egress and ingress packets to be
>>> mirrored?
>>
>> That I'm not sure about.  If it's possible, it would be controlling
>> which originators have their traffic replicated.
> 
> You need this if you want to do mirroring, since the API requires to
> say if you want to mirror ingress or egress. WiFi monitoring is less
> specific as far as i understand. It is whatever is received on the
> antenna.
> 
>> I don't think it will take me all that long to implement this, but
>> my goal right now is to be sure that the design I implement is a good
>> solution.  I'm open to recommendations.
> 
> You probably want to look at what wifi monitor offers. And maybe check
> with the WiFi people what they actually think about monitoring, or if
> they have a better suggestion.
> 
>       Andrew
> 

Re: Port mirroring (RFC)

[Alex Elder]

On 12/15/21 11:48 AM, Florian Fainelli wrote:
> On 12/14/21 6:47 AM, Alex Elder wrote:
>> I am implementing what amounts to port mirroring functionality
>> for the IPA driver.
>>
>> The IPA hardware isn't exactly a network switch (it's sort of
>> more than that), but it has the ability to supply replicas of
>> packets transferred within it to a special (read only) interface.
>>
>> My plan is to implement this using a new "ipa_mirror" network
>> device, so it could be used with a raw socket to capture the
>> arriving packets.  There currently exists one other netdev,
>> which represents access through a modem to a WWAN network.
>>
>> I would like some advice on how to proceed with this.  I want
>> the result to match "best practice" upstream, and would like
>> this to be as well integrated possible with existing network
>> tools.
>>
>> A few details about the stream of packets that arrive on
>> this hardware interface:
>> - Packet data is truncated if it's larger than a certain size
>> - Each packet is preceded by a fixed-size header describing it
>> - Packets (and their headers) are aggregated into a buffer; i.e.
>>    a single receive might carry a dozen (truncated) packets
> 
> Andrew has already responded, but there are currently sort of two ways
> that mirroring is implemented in upstream supported drivers:
> 
> - for "classic" Ethernet switches mirroring is typically from one port,
> or a set of ports, to another with a choice in whether you want to
> capture ingress, egress, or both towards that port. Because each port is
> a netdev already you just run tcpdump/pcap the way you normally do on
> your Ethernet NIC and get the captured packets. Configuration is via
> offloading the tc_mired action.

I read about switch port mirroring, and it sounded along the
lines of what I'm trying to implement, which is why I suggested
this might be a case of "port mirroring."  That said, it is
not configurable in the same way.  Here, we really have a
dedicated port through which replicated packets arrive.  It is
not possible to just use one of the existing switch ports.  So
it isn't really the same.

> - mlxsw implements devlink traps which is not exaclty designed for
> capturing mirrored packets but more like management events such as why
> the packet was trapped etc. You could however imagine using devlink
> traps for that purpose of capturing mirrored packets in the absence of a
> suitable network device

Just your description tells me this approach is probably not right.
But I'll look into it some more.

> Not sure if this helps, but that is the situation.

Right now I find it useful to hear about anything, even things
I might not ultimately use.

>> Here are a few specific questions, but I would love to get
>> *any* feedback about what I'm doing.
>> - Is representing this as a separate netdev a reasonable
>>    thing to do?
> 
> I would think so, given this allows standard tools to work.

It's reassuring knowing there isn't disagreement.

>> - Is there anything wrong with making a netdev read-only?
>>    (Any packets supplied for transmit would be dropped)
> 
> This looks reasonable.
> 
>> - Are there things I should do so it's clear this interface
>>    does not carry IP traffic (or even UDP, etc.)?
> 
> There were various patches in the past to prevent a network device from
> getting any IP stack to be configured:
> 
> https://yhbt.net/lore/all/20150825232021.GA8482@Alexeis-MacBook-Pro-2.local/t/
> 
> I forgot whether as a driver you can just refuse to have an IP address
> assigned to your network device, AFAIR it requires changes to the
> network stack as proposed in the patch set.

Thanks, I'll follow up with that as well.

>> - Should the driver de-aggregate the received packets, i.e.
>>    separating each into a separate SKB for reading?
> 
> If this is truly a mirroring device, you would expect it to "render" the
> mirrored packets exactly the way they are, maybe add an ethtool private
> option to de-aggregate packets if this helps debugging?

I have some details to work out before I can implement such a
thing, but this would be helpful.

Thanks a lot for your input.

					-Alex

>> I might have *many* more questions, but I'd just like to make
>> sure I'm on the right track, and would like both specific and
>> general suggestions about how to do this the right way.
>>
>> Thanks.
>>                      -Alex
> 
> 

Re: Port mirroring (RFC)

[Alex Elder]

On 12/15/21 2:12 PM, Florian Fainelli wrote:
> On 12/15/21 6:47 AM, Alex Elder wrote:
>> On 12/15/21 3:18 AM, Andrew Lunn wrote:
>>>> IPA is a device that sits between the main CPU and a modem,
>>>> carrying WWAN network data between them.
>>>>
>>>> In addition, there is a small number of other entities that
>>>> could be reachable through the IPA hardware, such as a WiFi
>>>> device providing access to a WLAN.
>>>>
>>>> Packets can travel "within IPA" between any of these
>>>> "connected entities."  So far only the path between the
>>>> AP and the modem is supported upstream, but I'm working
>>>> on enabling more capability.
>>>>
>>>> Technically, the replicated packets aren't visible on
>>>> any one port; the only way to see that traffic is in
>>>> using this special port.  To me this seemed like port
>>>> mirroring, which is why I suggested that.  I'm want to
>>>> use the proper model though, so I appreciate your
>>>> response.
>>>
>>> Do you have netdevs for the modem, the wifi, and whatever other
>>> interfaces the hardware might have?
>>
>> Not yet, but yes I expect that's how it will work.
>>
>>> To setup a mirror you would do something like:
>>>
>>> sudo tc filter add dev eth0 parent ffff: protocol all u32 match u32 0
>>> 0 action mirred egress mirror dev tun0
>>
>> OK so it sounds like the term "mirror" means mirroring using
>> Linux filtering.  And then I suppose "monitoring" is collecting
>> all "observed" traffic through an interface?
> 
> It is mirroring in terms of an action to perform for a given packet
> having been matched, now Ethernet switches for instance support
> mirroring in hardware and that specific action can be offloaded down to
> your hardware. You can take a look at net/dsa/* and drivers/net/dsa/ for
> an example of how this is done.

I've looked a bit at Documentation/networking/dsa/*, not at the
code.  You're right though, this is more like the Ethernet switch
port mirroring.

>> If that's the case, this seems to me more like monitoring, except
>> I suggested presenting the replicated data through a separate
>> netdev (rather than, for example, through the one for the modem).
>>
>> If it makes more sense, I could probably inject the replicated
>> packets received through this special interface into one or
>> another of the existing netdevs, rather than using a separate
>> one for this purpose.
>>
>>> where you are mirroring eth0 to tun0. eth0 would have to be your modem
>>> netdev, or your wifi netdev, and tun0 would be your monitor device.
>>>
>>> If you do have a netdev on the host for each of these network
>>> interfaces, mirroring could work. Architecturally, it would make sense
>>> to have these netdevs, so you can run wpa_supplicant on the wifi
>>> interface to do authentication, etc.
>>>
>>> Do you have control over selecting egress and ingress packets to be
>>> mirrored?
>>
>> That I'm not sure about.  If it's possible, it would be controlling
>> which originators have their traffic replicated.
> 
> And the originators would be represented as network devices, or would
> they be another kind of object in the Linux kernel? If they are network
> devices then you can use the example from Andrew above because you
> basically define the source device to mirror. Else, you may have to
> invent your own thing.

That's a good question, and it's one I still don't have an answer
for.  I'm actually having a parallel discussion internally about
how best to represent some of this stuff.

I don't really want to invent my own thing, but it would be no
surprise if some things needed enhancement to allow IPA to "fit"
naturally.

All that is next year's work though.  For now I'm just trying to
gather input so I can have a reasonable picture of the design in
mind as I start implementing.

					-Alex

>> I don't think it will take me all that long to implement this, but
>> my goal right now is to be sure that the design I implement is a good
>> solution.  I'm open to recommendations.
>>
>> Thanks.
>>
>>                      -Alex
>>
>>>      Andrew
>>>
>>
> 
> 

Re: Port mirroring (RFC)

[Alex Elder]

On 12/15/21 5:33 PM, Jakub Kicinski wrote:
> On Tue, 14 Dec 2021 08:47:12 -0600 Alex Elder wrote:
>> I am implementing what amounts to port mirroring functionality
>> for the IPA driver.
>>
>> The IPA hardware isn't exactly a network switch (it's sort of
>> more than that), but it has the ability to supply replicas of
>> packets transferred within it to a special (read only) interface.
>>
>> My plan is to implement this using a new "ipa_mirror" network
>> device, so it could be used with a raw socket to capture the
>> arriving packets.  There currently exists one other netdev,
>> which represents access through a modem to a WWAN network.
>>
>> I would like some advice on how to proceed with this.  I want
>> the result to match "best practice" upstream, and would like
>> this to be as well integrated possible with existing network
>> tools.
>>
>> A few details about the stream of packets that arrive on
>> this hardware interface:
>> - Packet data is truncated if it's larger than a certain size
>> - Each packet is preceded by a fixed-size header describing it
>> - Packets (and their headers) are aggregated into a buffer; i.e.
>>     a single receive might carry a dozen (truncated) packets
>>
>> Here are a few specific questions, but I would love to get
>> *any* feedback about what I'm doing.
>> - Is representing this as a separate netdev a reasonable
>>     thing to do?
>> - Is there anything wrong with making a netdev read-only?
>>     (Any packets supplied for transmit would be dropped)
>> - Are there things I should do so it's clear this interface
>>     does not carry IP traffic (or even UDP, etc.)?
>> - Should the driver de-aggregate the received packets, i.e.
>>     separating each into a separate SKB for reading?
>>
>> I might have *many* more questions, but I'd just like to make
>> sure I'm on the right track, and would like both specific and
>> general suggestions about how to do this the right way.
> 
> Maybe the first question to ask is - why do you need this?

That is a great question.

> Or perhaps - how is it used? There's a significant difference
> between an interface for users and a debug interface.

This would be a debug interface.  That is, it is not intended
as a normal way of delivering packets to Linux, it's meant as
an aid in understanding what's going on inside the hardware
(exposing the packets that are passing through IPA).

> Do you aim to give users control over the forwarding which happens
> on the application processor at some point? If so Andrew and Florian
> give great suggestions but starting from debugging of the forwarding
> feels a little backward.

This actually goes back to what IPA can do, which is allow
certain IP functionality to be offloaded.  Currently, IPA
simply carries packets between the AP and a Qualcomm modem.
It provides the path needed for Linux to access the modem's
WWAN network.

However it is also possible to have the IPA hardware carry
IP packets between the modem and (for example) a USB device,
without the AP being directly involved in the transfer.
But even though Linux would not handle these packets, one
might still (for debugging purposes) like to see (on the
AP) what the packets moved by IPA look like.

That omits many details, some of which I don't think I could
even explain well right now.

But to answer your question, this would *not* be an interface
used by "normal users."  It might be useful to filter these
with networking tools for the benefit of analysis, but there
would be no forwarding of these packets anywhere else.  It
really is meant to be a monitoring interface, to be used for
development and troubleshooting.

					-Alex

Port mirroring, v2 (RFC)

[Alex Elder]

This is a second RFC for a design to implement new functionality

in the Qualcomm IPA driver.  Since last time I've looked into some

options based on feedback.  This time I'll provide some more detail

about the hardware, and what the feature is doing.  And I'll end

with two possible implementations, and some questions.



My objective is to get a general sense that what I plan to do

is reasonable, so the patches that implement it will be acceptable.





The feature provides the AP access to information about the packets

that the IPA hardware processes as it carries them between its

"ports".  It is intended as a debug/informational interface only.

Before going further I'll briefly explain what the IPA hardware

does.



The upstream driver currently uses the hardware only as the path

that provides access to a 5G/LTE cellular network via a modem

embedded in a Qualcomm SoC.



        \|/

         |

   ------+-----   ------

   | 5G Modem |   | AP |

   ------------   ------

              \\    || <-- IPA channels, or "ports"

             -----------

             |   IPA   |

             -----------


But the hardware also provides a path to carry network traffic to

and from other entities as well, such as a PCIe root complex (or

endpoint).  For example an M.2 WiFi card can use a PCIe slot that

is IPA connected, and the IPA hardware can carry packets between

the AP and that WiFi card.  (A separate MHI host driver manages the

interaction between PCIe and IPA in this case.)



        \|/                PCIe bus --.     \|/

         |                            |      |

   ------+-----  ------   ----------- v ------+-----

   | 5G Modem |  | AP |...| PCIe RC |===| M.2 WiFi |

   ------------  ------   -----------   ------------

              \\   ||    // <-- IPA channels

               -----------

               |   IPA   |

               -----------



In the above scenario, the IPA hardware is actually able to directly

route packets between the embedded modem and the WiFi card without

AP involvement.  But supporting that is a future problem, and I

don't want to get ahead of myself.



The point is that the IPA hardware can carry network packets between

any pair of its "ports".  And the AP can get information about all

of the traffic the IPA handles, using a special interface.



The "information" consists of replicas of each packet transferred

(possibly truncated), each preceded by a fixed-size "status" header.

It amounts to a stream of packets delivered by the IPA hardware to

the AP.  This stream is distinct from "normal" traffic (such as

packets exchanged between the AP and modem); but note that even

those packets would be replicated.





I originally described this feature as "port mirroring" because it

seemed to be similar to that feature of smart network switches.  But

the "mirroring" term was interpreted as a something Linux would do,

so at a minimum, that's the wrong term.  Andrew Lunn (and others)

suggested that WiFi monitor mode might be a good model.  I looked

into that, and I don't think that quite fits either.  I really think

this should be represented separate from the "normal" network

devices associated with IPA.





Below I will describe two possible implementations I'm considering.

I would like to know which approach makes the most sense (or if

neither does, what alternative would be better).  On top of that I

guess I'd like suggestions for the *name* for this (i.e., what

should I call the interface that implements this?).



The two alternative implementations I'm considering are a network

device, and a "misc" (character) device.  In both cases, a user

space program would open the interface and read from it.  The data

read would just be the raw data received--a stream of the (possibly

truncated) packets together with their "status" headers.  I envision

either one could be a source of packets processed by libpcap/tcpdump.



My preference is to use a network device.  I think it fits the

"stream of packets" best, and existing networking code would take

care of all the details of queueing and packet management.  One down

side is that this is not a "normal" network interface--there is no

reason to associate it with an IP stack, for example.



A misc device would avoid the interface being treated as a "normal"

network device.  It could present all packet data to user space, but

the IPA driver would have to manage buffering, including limiting

the amount of received buffers.  Implementing this would be fine,

but I think it would just be nicer to use the network model.





So bottom line, given what I've described above:

- Is a distinct network device a reasonable and acceptable way of

   implementing this feature?  If not, why not?

- Would implementing this as a misc device be preferable?  Why?

- Is there a better alternative than either of the above?

- Can anyone suggest a name for this functionality, something that

   is meaningful but would not be confused with other existing terms?



Thanks.


					-Alex

Re: Port mirroring, v2 (RFC)

[Alex Elder]

Yikes!  I don't know why that turned out double-spaced.  I hope
this one turns out better.

					-Alex

This is a second RFC for a design to implement new functionality
in the Qualcomm IPA driver.  Since last time I've looked into some
options based on feedback.  This time I'll provide some more detail
about the hardware, and what the feature is doing.  And I'll end
with two possible implementations, and some questions.

My objective is to get a general sense that what I plan to do
is reasonable, so the patches that implement it will be acceptable.


The feature provides the AP access to information about the packets
that the IPA hardware processes as it carries them between its
"ports".  It is intended as a debug/informational interface only.
Before going further I'll briefly explain what the IPA hardware
does.

The upstream driver currently uses the hardware only as the path
that provides access to a 5G/LTE cellular network via a modem
embedded in a Qualcomm SoC.

        \|/
         |
   ------+-----   ------
   | 5G Modem |   | AP |
   ------------   ------
              \\    || <-- IPA channels, or "ports"
             -----------
             |   IPA   |
             -----------

But the hardware also provides a path to carry network traffic to
and from other entities as well, such as a PCIe root complex (or
endpoint).  For example an M.2 WiFi card can use a PCIe slot that
is IPA connected, and the IPA hardware can carry packets between
the AP and that WiFi card.  (A separate MHI host driver manages the
interaction between PCIe and IPA in this case.)

        \|/                PCIe bus --.     \|/
         |                            |      |
   ------+-----  ------   ----------- v ------+-----
   | 5G Modem |  | AP |...| PCIe RC |===| M.2 WiFi |
   ------------  ------   -----------   ------------
              \\   ||    // <-- IPA channels
               -----------
               |   IPA   |
               -----------

In the above scenario, the IPA hardware is actually able to directly
route packets between the embedded modem and the WiFi card without
AP involvement.  But supporting that is a future problem, and I
don't want to get ahead of myself.

The point is that the IPA hardware can carry network packets between
any pair of its "ports".  And the AP can get information about all
of the traffic the IPA handles, using a special interface.

The "information" consists of replicas of each packet transferred
(possibly truncated), each preceded by a fixed-size "status" header.
It amounts to a stream of packets delivered by the IPA hardware to
the AP.  This stream is distinct from "normal" traffic (such as
packets exchanged between the AP and modem); but note that even
those packets would be replicated.


I originally described this feature as "port mirroring" because it
seemed to be similar to that feature of smart network switches.  But
the "mirroring" term was interpreted as a something Linux would do,
so at a minimum, that's the wrong term.  Andrew Lunn (and others)
suggested that WiFi monitor mode might be a good model.  I looked
into that, and I don't think that quite fits either.  I really think
this should be represented separate from the "normal" network
devices associated with IPA.


Below I will describe two possible implementations I'm considering.
I would like to know which approach makes the most sense (or if
neither does, what alternative would be better).  On top of that I
guess I'd like suggestions for the *name* for this (i.e., what
should I call the interface that implements this?).

The two alternative implementations I'm considering are a network
device, and a "misc" (character) device.  In both cases, a user
space program would open the interface and read from it.  The data
read would just be the raw data received--a stream of the (possibly
truncated) packets together with their "status" headers.  I envision
either one could be a source of packets processed by libpcap/tcpdump.

My preference is to use a network device.  I think it fits the
"stream of packets" best, and existing networking code would take
care of all the details of queueing and packet management.  One down
side is that this is not a "normal" network interface--there is no
reason to associate it with an IP stack, for example.

A misc device would avoid the interface being treated as a "normal"
network device.  It could present all packet data to user space, but
the IPA driver would have to manage buffering, including limiting
the amount of received buffers.  Implementing this would be fine,
but I think it would just be nicer to use the network model.


So bottom line, given what I've described above:
- Is a distinct network device a reasonable and acceptable way of
   implementing this feature?  If not, why not?
- Would implementing this as a misc device be preferable?  Why?
- Is there a better alternative than either of the above?
- Can anyone suggest a name for this functionality, something that
   is meaningful but would not be confused with other existing terms?

Thanks.

					-Alex

Re: Port mirroring, v2 (RFC)

[Andrew Lunn]

On Fri, Jan 14, 2022 at 11:03:26AM -0600, Alex Elder wrote:
> Yikes!  I don't know why that turned out double-spaced.  I hope
> this one turns out better.
> 
> 					-Alex
> 
> This is a second RFC for a design to implement new functionality
> in the Qualcomm IPA driver.  Since last time I've looked into some
> options based on feedback.  This time I'll provide some more detail
> about the hardware, and what the feature is doing.  And I'll end
> with two possible implementations, and some questions.
> 
> My objective is to get a general sense that what I plan to do
> is reasonable, so the patches that implement it will be acceptable.
> 
> 
> The feature provides the AP access to information about the packets
> that the IPA hardware processes as it carries them between its
> "ports".  It is intended as a debug/informational interface only.
> Before going further I'll briefly explain what the IPA hardware
> does.
> 
> The upstream driver currently uses the hardware only as the path
> that provides access to a 5G/LTE cellular network via a modem
> embedded in a Qualcomm SoC.
> 
>        \|/
>         |
>   ------+-----   ------
>   | 5G Modem |   | AP |
>   ------------   ------
>              \\    || <-- IPA channels, or "ports"
>             -----------
>             |   IPA   |
>             -----------

Hi Alex

I think i need to take a step back here. With my background, an AP is
an 802.11 Access Point.
But here you mean Application Processor?
What does IPA standard for ?
MHI ?

I can probably figure these all out from context, but half the problem
here is making sure we are talking the same language when we are
considering using concepts from another part of the network stack.

	    Andrew

Re: Port mirroring, v2 (RFC)

[Alex Elder]

On 1/14/22 2:46 PM, Andrew Lunn wrote:
> On Fri, Jan 14, 2022 at 11:03:26AM -0600, Alex Elder wrote:
>> Yikes!  I don't know why that turned out double-spaced.  I hope
>> this one turns out better.
>>
>> 					-Alex
>>
>> This is a second RFC for a design to implement new functionality
>> in the Qualcomm IPA driver.  Since last time I've looked into some
>> options based on feedback.  This time I'll provide some more detail
>> about the hardware, and what the feature is doing.  And I'll end
>> with two possible implementations, and some questions.
>>
>> My objective is to get a general sense that what I plan to do
>> is reasonable, so the patches that implement it will be acceptable.
>>
>>
>> The feature provides the AP access to information about the packets
>> that the IPA hardware processes as it carries them between its
>> "ports".  It is intended as a debug/informational interface only.
>> Before going further I'll briefly explain what the IPA hardware
>> does.
>>
>> The upstream driver currently uses the hardware only as the path
>> that provides access to a 5G/LTE cellular network via a modem
>> embedded in a Qualcomm SoC.
>>
>>         \|/
>>          |
>>    ------+-----   ------
>>    | 5G Modem |   | AP |
>>    ------------   ------
>>               \\    || <-- IPA channels, or "ports"
>>              -----------
>>              |   IPA   |
>>              -----------
> 
> Hi Alex
> 
> I think i need to take a step back here. With my background, an AP is
> an 802.11 Access Point.

Again, terminology problems!  Sorry about that.

Yes, when I say "AP" I mean "Application Processor".  Some people
might call it "APSS" for "Application Processor Subsystem."

> But here you mean Application Processor?
> What does IPA standard for ?

"IPA" stands for IP Accelerator, or Internet Protocol Accelerator.

> MHI ?

Modem-Host Interface (which is really a separate topic that I
don't want to get too distracted by at this point).  It is
basically a layer built over PCIe, that abstracts things to
carry multiple logical channels of data over a PCIe bus.
I'm working with others now to support MHI, but it's not
at all present in the IPA driver at the moment.

> I can probably figure these all out from context, but half the problem
> here is making sure we are talking the same language when we are
> considering using concepts from another part of the network stack.

Yes!  It's one reason I asked for input on naming this
feature.  Qualcomm has its own name (which could be fine),
but I'd like to try to use something that avoids confusion
as much as possible.

I really appreciate your considering this Andrew.

					-Alex

> 
> 	    Andrew
> 

Re: Port mirroring, v2 (RFC)

[Andrew Lunn]

> Below I will describe two possible implementations I'm considering.
> I would like to know which approach makes the most sense (or if
> neither does, what alternative would be better).

Hi Alex

Another corner of the kernel you could look for inspiration is usbmon.

https://www.kernel.org/doc/html/latest/usb/usbmon.html

This is similar to your misc char device, but it is actually
implemented as a pseudo filesystem. It is intended for libpcap based
applications and i've used it with tcpdump and wireshark. So exactly
your use cases.

Because it is not a network device, the extra header does not cause
problems, and there is no confusion about what the 'monitoring' netdevs
are good for.

Since you are talking 5G and WiFi, you have a lot of packets
here. Being able to use BPF with libpcap is probably useful to allow
filtering of what packets are passed to user space. I've never looked
at how the BPF core is attached to a netdev. But i suspect your extra
header could be an issue. So you are going to need some custom code to
give it an offset into the packet to the Ethernet header?

Humm, actually, you called the IPA the IP accelerator. Are these L2
frames or L3 packets? Do you see 3 or even 4 MAC addresses in an
802.11 header? Two MAC addresses in an 802.3 header? etc.

       Andrew

Re: Port mirroring, v2 (RFC)

[Alex Elder]

On 1/15/22 9:14 AM, Andrew Lunn wrote:
>> Below I will describe two possible implementations I'm considering.
>> I would like to know which approach makes the most sense (or if
>> neither does, what alternative would be better).
> 
> Hi Alex
> 
> Another corner of the kernel you could look for inspiration is usbmon.
> 
> https://www.kernel.org/doc/html/latest/usb/usbmon.html

Yes this looks very similar.  However IPA only carries
network traffic, unlike USB.  The "peripherals" (packet
sources) are all generating IP packets.

> This is similar to your misc char device, but it is actually
> implemented as a pseudo filesystem. It is intended for libpcap based
> applications and i've used it with tcpdump and wireshark. So exactly
> your use cases.

I suppose I could represent each possible source of packets
with a file system entry.  But that would require the IPA
driver to parse the received data and buffer each packet
separately (with others from its source).  Kind of messy.

I feel like it would be much better to just have one interface,
and have the filtering software exclude things that aren't
interesting.

> Because it is not a network device, the extra header does not cause
> problems, and there is no confusion about what the 'monitoring' netdevs
> are good for.

Yes, agreed.

> Since you are talking 5G and WiFi, you have a lot of packets
> here. Being able to use BPF with libpcap is probably useful to allow
> filtering of what packets are passed to user space. I've never looked
> at how the BPF core is attached to a netdev. But i suspect your extra

I haven't either.  But this too is something I've generally
thought I'd have to investigate.

> header could be an issue. So you are going to need some custom code to
> give it an offset into the packet to the Ethernet header?

Yes I think so.  No Ethernet header though.

> Humm, actually, you called the IPA the IP accelerator. Are these L2
> frames or L3 packets? Do you see 3 or even 4 MAC addresses in an
> 802.11 header? Two MAC addresses in an 802.3 header? etc.

L3 packets.  Except they are truncated if they're long, and they
have a metadata header prepended.

No MAC addresses, no 802.11 header.  Generally speaking, these
packets will start with 4 or 6 as the upper nibble of the first
byte (IP packet).  And of course, this follows the fixed-length
32-byte "status" header.

Here's are IPv4 and IPv6 ICMP examples.

00: 45 00 00 54 ac 5a 40 00 40 01 be 3c c0 00 00 02

10: 08 08 08 08 08 00 10 cf 11 c0 00 03 4f f4 e6 61

20: ac 14 08 00 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13

30: 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23

40: 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33

50: 34 35 36 37


00: 66 00 00 00 00 40 3a 70 26 07 77 00 00 00 00 2d

10: 00 00 00 01 08 08 08 08 26 07 fb 90 99 41 19 56

20: 00 c0 00 00 02 00 00 00 81 00 13 1f 11 c0 00 03

30: 4f f4 e6 61 ac 14 08 00 08 09 0a 0b 0c 0d 0e 0f

40: 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f

50: 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f

60: 30 31 32 33 34 35 36 37



I'm basically ready to go on this, either way (using a
misc device, or--preferably--using a netdev).

I'm just trying to avoid getting that fully working,
then learning when I submit patches that someone thinks
it's the wrong way to go about it.

If a netdev is acceptable, my remaining issues are:
- Whether/how to avoid having the device be treated
   as if it needed support from the network stack
   (i.e., as a "real" network interface, serving to
   send and receive packets).
- Similar, whether there are special configuration
   options that should be used, given the device's
   purpose.
- What to call this functionality.  I'll avoid "mirror"
   and will try to come up with something reasonable,
   but suggestions are welcome.

Thanks.

					-Alex

>         Andrew
> 

Re: Port mirroring, v2 (RFC)

[Jakub Kicinski]

On Fri, 14 Jan 2022 15:12:41 -0600 Alex Elder wrote:
> > I think i need to take a step back here. With my background, an AP is
> > an 802.11 Access Point.  
> 
> Again, terminology problems!  Sorry about that.
> 
> Yes, when I say "AP" I mean "Application Processor".  Some people
> might call it "APSS" for "Application Processor Subsystem."

And AP is where Linux runs, correct? Because there are also APs on NICs
(see ETH_RESET_AP) which are not the host...

Re: Port mirroring, v2 (RFC)

[Alex Elder]

On 1/18/22 12:07 PM, Jakub Kicinski wrote:
> On Fri, 14 Jan 2022 15:12:41 -0600 Alex Elder wrote:
>>> I think i need to take a step back here. With my background, an AP is
>>> an 802.11 Access Point.
>>
>> Again, terminology problems!  Sorry about that.
>>
>> Yes, when I say "AP" I mean "Application Processor".  Some people
>> might call it "APSS" for "Application Processor Subsystem."
> 
> And AP is where Linux runs, correct? Because there are also APs on NICs
> (see ETH_RESET_AP) which are not the host...

Correct.  It is the term I have used consistently in the
IPA code, to refer to "the processor (complex) running
Linux, which (basically) owns the IPA device."

					-Alex

Re: Port mirroring, v2 (RFC)

[Jakub Kicinski]

On Tue, 18 Jan 2022 11:37:05 -0600 Alex Elder wrote:
> I'm basically ready to go on this, either way (using a
> misc device, or--preferably--using a netdev).
> 
> I'm just trying to avoid getting that fully working,
> then learning when I submit patches that someone thinks
> it's the wrong way to go about it.
> 
> If a netdev is acceptable, my remaining issues are:
> - Whether/how to avoid having the device be treated
>    as if it needed support from the network stack
>    (i.e., as a "real" network interface, serving to
>    send and receive packets).
> - Similar, whether there are special configuration
>    options that should be used, given the device's
>    purpose.
> - What to call this functionality.  I'll avoid "mirror"
>    and will try to come up with something reasonable,
>    but suggestions are welcome.

I can't claim that my opinions on this sort of stuff are very stable
but I like Andrew's suggestion and I'd even say maybe just debugfs...

We try hard to prevent any abuse of netdevs for carrying what is not
real networking traffic and keep the semantics clear. netdevs are not
meant as an abstraction, they are more of an exception to the
"everything is a file" Unix rule.

Another thing that could possibly work is devlink traps and
DEVLINK_TRAP_ACTION_MIRROR, but again, not sure if we want to bend that
interface which has pretty nice and clear semantics to support a vendor
use case which is an aberration from our forwarding model in the
first place...

So I'd do something simple in debugfs and if anyone really cares about
the forwarding details put the real effort into modeling/controlling 
the forwarding with Linux. 

Re: Port mirroring, v2 (RFC)

[Alex Elder]

On 1/18/22 12:30 PM, Jakub Kicinski wrote:
> On Tue, 18 Jan 2022 11:37:05 -0600 Alex Elder wrote:
>> I'm basically ready to go on this, either way (using a
>> misc device, or--preferably--using a netdev).
>>
>> I'm just trying to avoid getting that fully working,
>> then learning when I submit patches that someone thinks
>> it's the wrong way to go about it.
>>
>> If a netdev is acceptable, my remaining issues are:
>> - Whether/how to avoid having the device be treated
>>     as if it needed support from the network stack
>>     (i.e., as a "real" network interface, serving to
>>     send and receive packets).
>> - Similar, whether there are special configuration
>>     options that should be used, given the device's
>>     purpose.
>> - What to call this functionality.  I'll avoid "mirror"
>>     and will try to come up with something reasonable,
>>     but suggestions are welcome.
> 
> I can't claim that my opinions on this sort of stuff are very stable
> but I like Andrew's suggestion and I'd even say maybe just debugfs...
> 
> We try hard to prevent any abuse of netdevs for carrying what is not
> real networking traffic and keep the semantics clear. netdevs are not
> meant as an abstraction, they are more of an exception to the
> "everything is a file" Unix rule.
> 
> Another thing that could possibly work is devlink traps and
> DEVLINK_TRAP_ACTION_MIRROR, but again, not sure if we want to bend that
> interface which has pretty nice and clear semantics to support a vendor
> use case which is an aberration from our forwarding model in the
> first place...
> 
> So I'd do something simple in debugfs and if anyone really cares about
> the forwarding details put the real effort into modeling/controlling
> the forwarding with Linux.

This is nice, clear guidance.

I'm going to work through that design and will send one
more RFC as my proposal.  It'll be somewhat short...

Thank you very much!

					-Alex

IPA monitor (Final RFC)

[Alex Elder]

In previous messages I explained how the Qualcomm IP Accelerator

(IPA) sometimes has the ability to replicate all packets it

processes, and supply those replicated packets to the main

application processor (AP).  I initially suggested using a network

device as the interface for this, but after some discussion, Jakub

recommended using a debugfs file to supply these packets.



Below is basically a specification for the design I'll use.  It is

what I intend to implement, so if anyone has any objection, please

voice it now.  I'll be sending this code out for review in the

coming few weeks.


Thank you.

					-Alex


- A new debugfs directory "qcom_ipaX" will be created for each IPA

   instance (X = IPA device number).  There's normally only going to

   be one of these, but there is at least one SoC that has two.

     /sys/kernel/debug/qcom_ipa0/

- If an IPA instance supports a "monitor endpoint", a "monitor" file

   will be created in its "qcom_ipaX" directory.

     /sys/kernel/debug/qcom_ipa0/monitor

- The "monitor" file is opened exclusively (no O_EXCL needed).  An

   attempt to open that file when it's already open produces EBUSY.

- The monitor file is read-only (S_IRUSR), and does not support seeks.

- Once opened, "monitor packets" (which consist of a fixed size

   status header, followed by replicated packet data) will be

   accumulated in *receive* buffers.  If a replicated packet is

   large, it will have been truncated by hardware to reduce

   monitoring bandwidth.

- Once opened, reads to the monitor file are satisfied as follows:

     - If no receive buffers have accumulated, the read will block

       until at least one monitor packet can be returned.

     - If the file is opened with O_NONBLOCK, a read that would block

       will return EAGAIN instead of blocking.

     - A read that blocks is interruptible.

     - A valid monitor packet is supplied to user space at most once.

     - Only "complete" monitor packets are supplied to the reader.

       I.e., a status header will always be supplied together with

       the packet data it describes.

     - A *read* buffer will be filled with as many monitor packets as

       possible.  If they'll fit in the read buffer, all accumulated

       monitor packets will be returned to the reader.

     - If the read buffer has insufficient room for the next

       available monitor packet, the read request returns.

     - If any monitor packet received from hardware is bad, it--along

       with everything beyond it in its page--will be discarded.

         - The received data must be big enough to hold a status

           header.

         - The received data must contain the packet data, meaning

           packet length in the status header lies within range.

Re: IPA monitor (Final RFC)

[Alex Elder]

Damned double spacing.  I'm not sure why it's happening...	-Alex

In previous messages I explained how the Qualcomm IP Accelerator
(IPA) sometimes has the ability to replicate all packets it
processes, and supply those replicated packets to the main
application processor (AP).  I initially suggested using a network
device as the interface for this, but after some discussion, Jakub
recommended using a debugfs file to supply these packets.

Below is basically a specification for the design I'll use.  It is
what I intend to implement, so if anyone has any objection, please
voice it now.  I'll be sending this code out for review in the
coming few weeks.

Thank you.

					-Alex

- A new debugfs directory "qcom_ipaX" will be created for each IPA
   instance (X = IPA device number).  There's normally only going to
   be one of these, but there is at least one SoC that has two.
     /sys/kernel/debug/qcom_ipa0/
- If an IPA instance supports a "monitor endpoint", a "monitor" file
   will be created in its "qcom_ipaX" directory.
     /sys/kernel/debug/qcom_ipa0/monitor
- The "monitor" file is opened exclusively (no O_EXCL needed).  An
   attempt to open that file when it's already open produces EBUSY.
- The monitor file is read-only (S_IRUSR), and does not support seeks.
- Once opened, "monitor packets" (which consist of a fixed size
   status header, followed by replicated packet data) will be
   accumulated in *receive* buffers.  If a replicated packet is
   large, it will have been truncated by hardware to reduce
   monitoring bandwidth.
- Once opened, reads to the monitor file are satisfied as follows:
     - If no receive buffers have accumulated, the read will block
       until at least one monitor packet can be returned.
     - If the file is opened with O_NONBLOCK, a read that would block
       will return EAGAIN instead of blocking.
     - A read that blocks is interruptible.
     - A valid monitor packet is supplied to user space at most once.
     - Only "complete" monitor packets are supplied to the reader.
       I.e., a status header will always be supplied together with
       the packet data it describes.
     - A *read* buffer will be filled with as many monitor packets as
       possible.  If they'll fit in the read buffer, all accumulated
       monitor packets will be returned to the reader.
     - If the read buffer has insufficient room for the next
       available monitor packet, the read request returns.
     - If any monitor packet received from hardware is bad, it--along
       with everything beyond it in its page--will be discarded.
         - The received data must be big enough to hold a status
	  header.
	- The received data must contain the packet data, meaning
	  packet length in the status header lies within range.

Re: IPA monitor (Final RFC)

[Andrew Lunn]

Hi Alex

This looks good in general.


>     - If any monitor packet received from hardware is bad, it--along
> 
>       with everything beyond it in its page--will be discarded.
> 
>         - The received data must be big enough to hold a status
> 
>           header.
> 
>         - The received data must contain the packet data, meaning
> 
>           packet length in the status header lies within range.
 
So bad in just the sense that capturing the packet and passing it to
the application processor somehow went wrong.

What about packets with bad CRC? Since the application processor is
not involved, i assume something in the APA architecture is validating
L2 and L3 CRCs. Do they get dropped, or can they be seen in the
monitor stream? Does the header contain any indication of CRC errors,
since if the packet has been truncated, it won't be possible to
validate them. And you said L2 headers are not present anyway.

Do you look at various libpcap-ng implementations? Since this is
debugfs you are not defining a stable ABI, you can change it any time
you want and break userspace. But maybe there could be small changes
in the API which make it easier to feed to wireshark via libpcap.

	Andrew

Re: IPA monitor (Final RFC)

[Alex Elder]

On 2/1/22 6:19 PM, Andrew Lunn wrote:
> Hi Alex
> 
> This looks good in general.
> 
> 
>>      - If any monitor packet received from hardware is bad, it--along
>>
>>        with everything beyond it in its page--will be discarded.
>>
>>          - The received data must be big enough to hold a status
>>
>>            header.
>>
>>          - The received data must contain the packet data, meaning
>>
>>            packet length in the status header lies within range.
>   
> So bad in just the sense that capturing the packet and passing it to
> the application processor somehow went wrong.

All I'm saying is that the driver will take a little responsibility
for ensuring the stuff delivered to user space isn't complete crap.
To a certain extent that's to protect itself.  It would be easiest
to pass exactly what was received up to user space, without doing
any interpretation of it whatsoever.  But I think the kernel can
add this value at very little net cost.

The reality is, this should not happen.

> What about packets with bad CRC? Since the application processor is
> not involved, i assume something in the APA architecture is validating
> L2 and L3 CRCs. Do they get dropped, or can they be seen in the

The hardware can offload things like CRC calculation, but in that case
it's up to the receiving code to be told "this has a bad CRC" and
then decide to drop the received packet.  I think the replication
occurs early--possibly before hardware CRC calculations, so the
replica just gets delivered out this special endpoint just the way
it arrived.

> monitor stream? Does the header contain any indication of CRC errors,
> since if the packet has been truncated, it won't be possible to
> validate them. And you said L2 headers are not present anyway.

 From what I can tell, CRC errors are not indicated in the status
header.  The status seems to be more oriented toward "this is
the processing that the IPA hardware performed on this packet."
Including "it entered IPA on this 'port' and matched this
filter rule and got routed out this other 'port'."  But to be
honest my focus has been more on providing the feature than
what exactly those bits represent...

> Do you look at various libpcap-ng implementations? Since this is
> debugfs you are not defining a stable ABI, you can change it any time
> you want and break userspace. But maybe there could be small changes
> in the API which make it easier to feed to wireshark via libpcap.

I considered that.  That was really the interface I was envisioning
at first.  Those things don't really align perfectly with the
information that's made available here though.  This is more like
"what is the hardware doing to each packet" (so we can maybe
understand behavior, or identify a bug).  Rather than "what is
the content of packets flowing through?"  It might be useful
to use the powerful capabilities of things like wireshark for
analysis, but I kind of concluded the purpose of exposing this
information is a little different.

I've got most of the code buffering received data and making
it available through a file interface done.  But I have some
fine tuning to do before I'll be ready to post it for review.

Yes, the non-stable API means I can tweak it a bit even after
it's merged.  But we'll see.  It might be reasonable as-is.

Thanks a lot.

					-Alex

> 
> 	Andrew

Re: IPA monitor (Final RFC)

[Andrew Lunn]

> From what I can tell, CRC errors are not indicated in the status
> header.  The status seems to be more oriented toward "this is
> the processing that the IPA hardware performed on this packet."
> Including "it entered IPA on this 'port' and matched this
> filter rule and got routed out this other 'port'."  But to be
> honest my focus has been more on providing the feature than
> what exactly those bits represent...

My experience of using interfaces like this, it is sometimes CRC
problems you are trying to track down. So it would be nice if the
behaviour around CRCs was documented, to know if packets with bad CRC
get dropped, or are part of the stream etc. It probably does not make
much difference to the actual API design.

> > Do you look at various libpcap-ng implementations? Since this is
> > debugfs you are not defining a stable ABI, you can change it any time
> > you want and break userspace. But maybe there could be small changes
> > in the API which make it easier to feed to wireshark via libpcap.
> 
> I considered that.  That was really the interface I was envisioning
> at first.  Those things don't really align perfectly with the
> information that's made available here though.  This is more like
> "what is the hardware doing to each packet" (so we can maybe
> understand behavior, or identify a bug).  Rather than "what is
> the content of packets flowing through?"  It might be useful
> to use the powerful capabilities of things like wireshark for
> analysis, but I kind of concluded the purpose of exposing this
> information is a little different.

I've had good experiences with writing dissectors for wireshark.  I
think you should be able to decode the extra headers, and then pass
the IP packet onto the standard dissectors. And having multiple frames
in one message should also not be a big issue. So i would not discard
the idea of writing into some sort of pcap file and feeding it to
wireshark.

    Andrew