Possible bug? DOM-U network stopped working after fatal error reported in DOM0

Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

Hi all,

I ran into the following error report in the DOM0 kernel after a recent upgrade:
[  501.840816] vif vif-1-0 vif1.0: Cross page boundary, txp->offset:
2872, size: 1460
[  501.840828] vif vif-1-0 vif1.0: fatal error; disabling device
[  501.841076] xenbr0: port 2(vif1.0) entered disabled state
Once this error happens, the DOM-U behind this vif is no-longer
accessible. And recreating the same DOM-U does not fix the problem.
Only a reboot on the physical host machine helps.

The problem showed up after a recent upgrade on the DOM-U OS from
FreeNAS 11.3 to TrueNAS 12.0U7 and breaks the iSCSI service while
leaving other services like NFS intact.
The underlying OS for the NAS is FreeBSD, version 11.3 and 12.2 respectively.
So far I have tried the following combos:
- Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 11.3 DOM-U: Good
- Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed
- Linux 5.10 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed
- Linux 5.10 DOM0 + XEN 4.11 + FreeBSD 12.2 DOM-U: Regressed

I plan to try out the XEN 4.14 version which is the latest I can get
from the distro (Debian).
If that still does not fix the problem, I would build the 4.16 version
from source as my last resort.

I have to admit that this trial process is blind as I have no idea
which component in the combo is to be blamed. Is it a bug in the
backend-driver, frontend-driver or the hypervisor itself? Or due to
incompatible versions? Any suggestion on other diagnose ideas (e.g.
debug logs) will be welcome, while I work on the planned experiments.

Thanks,
G.R.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 1980 bytes --]

On 18.12.21 19:35, G.R. wrote:
> Hi all,
> 
> I ran into the following error report in the DOM0 kernel after a recent upgrade:
> [  501.840816] vif vif-1-0 vif1.0: Cross page boundary, txp->offset:
> 2872, size: 1460
> [  501.840828] vif vif-1-0 vif1.0: fatal error; disabling device

The dom0 network backend has detected an inconsistency in the data
received from the domU's frontend. In this case a request's memory
buffer crossed a page boundary, which is not allowed.

There has been a recent change in the xen netback driver to stop the
interface in such conditions, as such invalid requests are regarded to
be malicious and might lead to crashes in dom0.

So this issue should be reported to FreeBSD maintainers in order to
have the Xen netfornt driver fixed there.

> [  501.841076] xenbr0: port 2(vif1.0) entered disabled state
> Once this error happens, the DOM-U behind this vif is no-longer
> accessible. And recreating the same DOM-U does not fix the problem.
> Only a reboot on the physical host machine helps.
> 
> The problem showed up after a recent upgrade on the DOM-U OS from
> FreeNAS 11.3 to TrueNAS 12.0U7 and breaks the iSCSI service while
> leaving other services like NFS intact.
> The underlying OS for the NAS is FreeBSD, version 11.3 and 12.2 respectively.
> So far I have tried the following combos:
> - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 11.3 DOM-U: Good
> - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed
> - Linux 5.10 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed
> - Linux 5.10 DOM0 + XEN 4.11 + FreeBSD 12.2 DOM-U: Regressed

This information (especially the FreeBSD version affected) is
probably important for the FreeBSD maintainers.

> 
> I plan to try out the XEN 4.14 version which is the latest I can get
> from the distro (Debian).
> If that still does not fix the problem, I would build the 4.16 version
> from source as my last resort.

Xen is NOT to blame here.


Juergen


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3135 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

On Sun, Dec 19, 2021 at 2:35 AM G.R. <firemeteor@users.sourceforge.net> wrote:
>
> Hi all,
>
> I ran into the following error report in the DOM0 kernel after a recent upgrade:
> [  501.840816] vif vif-1-0 vif1.0: Cross page boundary, txp->offset:
> 2872, size: 1460
> [  501.840828] vif vif-1-0 vif1.0: fatal error; disabling device
> [  501.841076] xenbr0: port 2(vif1.0) entered disabled state
> Once this error happens, the DOM-U behind this vif is no-longer
> accessible. And recreating the same DOM-U does not fix the problem.
> Only a reboot on the physical host machine helps.
>
> The problem showed up after a recent upgrade on the DOM-U OS from
> FreeNAS 11.3 to TrueNAS 12.0U7 and breaks the iSCSI service while
> leaving other services like NFS intact.
To clarify -- mounting iSCSI disk will cause the problem immediately.

> The underlying OS for the NAS is FreeBSD, version 11.3 and 12.2 respectively.
> So far I have tried the following combos:
> - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 11.3 DOM-U: Good
> - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed
> - Linux 5.10 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed
> - Linux 5.10 DOM0 + XEN 4.11 + FreeBSD 12.2 DOM-U: Regressed
- Linux 5.10 DOM0 + XEN 4.14 + FreeBSD 12.2 DOM-U: Regressed
>
> I plan to try out the XEN 4.14 version which is the latest I can get
> from the distro (Debian).
I just upgraded to Debian bullseye (11) from buster (10) and migrated
to XEN4.14 as a result.
The syndrome persists, unfortunately.
BTW, my Dom0 kernel is a custom built version. Does any kernel config
could contribute to this problem?

> If that still does not fix the problem, I would build the 4.16 version
> from source as my last resort.
>
> I have to admit that this trial process is blind as I have no idea
> which component in the combo is to be blamed. Is it a bug in the
> backend-driver, frontend-driver or the hypervisor itself? Or due to
> incompatible versions? Any suggestion on other diagnose ideas (e.g.
> debug logs) will be welcome, while I work on the planned experiments.
>
> Thanks,
> G.R.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

First of all, thank you for your quick response, Juergen and Roger.
I just realized that I run into mail forwarding issue from sourceforge
mail alias service, and only found the responses when I checked the
list archive. As a result, I have to manually merge Roger's response
to reply...

> > I have to admit that this trial process is blind as I have no idea
> > which component in the combo is to be blamed. Is it a bug in the
> > backend-driver, frontend-driver or the hypervisor itself? Or due to
> > incompatible versions? Any suggestion on other diagnose ideas (e.g.
> > debug logs) will be welcome, while I work on the planned experiments.
>
> This is a bug in FreeBSD netfront, so no matter which Linux or Xen
> version you use.
>
> Does it make a difference if you disable TSO and LRO from netfront?
>
> $ ifconfig xn0 -tso -lro
It does not, the fatal error still show up after this command.

>
> Do you have instructions I can follow in order to try to reproduce the
> issue?
I don't know if there are any special details in my setup.
Hopefully I don't miss anything useful:
1. Build a TrueNAS 12.0U7 DOM-U by flushing the OS image into a vdisk
2. Create / import a zfs pool to the DOM-U
3. Create and share some file based iSCSI extents on the pool
4. Mount the iSCSI extent through some initiator clients.
The domU xn0 should be disabled immediately after step #4.

I omitted all operational details with the assumption that you are familiar
with TrueNAS and iSCSI setup.
For step #4, I can reproduce it with both ipxe initiator and the win7
built-in client.
As a result, I assume the client version does not matter.
For #2, I actually have a physical disk and controller assigned to DOM-U.
But I suspect this is probably irrelevant.
For #3, I'm not sure if the content in the extent matters.
So far I have been testing the same extent, which is formatted as an NTFS disk.

>
> Thanks, Roger.

> >
> > Thanks,
> > G.R.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Roger Pau Monné]

On Tue, Dec 21, 2021 at 01:13:43AM +0800, G.R. wrote:
> First of all, thank you for your quick response, Juergen and Roger.
> I just realized that I run into mail forwarding issue from sourceforge
> mail alias service, and only found the responses when I checked the
> list archive. As a result, I have to manually merge Roger's response
> to reply...
> 
> > > I have to admit that this trial process is blind as I have no idea
> > > which component in the combo is to be blamed. Is it a bug in the
> > > backend-driver, frontend-driver or the hypervisor itself? Or due to
> > > incompatible versions? Any suggestion on other diagnose ideas (e.g.
> > > debug logs) will be welcome, while I work on the planned experiments.
> >
> > This is a bug in FreeBSD netfront, so no matter which Linux or Xen
> > version you use.
> >
> > Does it make a difference if you disable TSO and LRO from netfront?
> >
> > $ ifconfig xn0 -tso -lro
> It does not, the fatal error still show up after this command.

Thanks for testing.

> >
> > Do you have instructions I can follow in order to try to reproduce the
> > issue?
> I don't know if there are any special details in my setup.
> Hopefully I don't miss anything useful:
> 1. Build a TrueNAS 12.0U7 DOM-U by flushing the OS image into a vdisk
> 2. Create / import a zfs pool to the DOM-U
> 3. Create and share some file based iSCSI extents on the pool
> 4. Mount the iSCSI extent through some initiator clients.
> The domU xn0 should be disabled immediately after step #4.
> 
> I omitted all operational details with the assumption that you are familiar
> with TrueNAS and iSCSI setup.

Not really. Ideally I would like a way to reproduce that can be done
using iperf, nc or similar simple command line tool, without requiring
to setup iSCSI.

> For step #4, I can reproduce it with both ipxe initiator and the win7
> built-in client.
> As a result, I assume the client version does not matter.
> For #2, I actually have a physical disk and controller assigned to DOM-U.
> But I suspect this is probably irrelevant.
> For #3, I'm not sure if the content in the extent matters.
> So far I have been testing the same extent, which is formatted as an NTFS disk.

Can you also paste the output of `ifconfig xn0`?

If I provided a patch for the FreeBSD kernel, would you be able to
apply and test it?

Thanks, Roger.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

> > I omitted all operational details with the assumption that you are familiar
> > with TrueNAS and iSCSI setup.
>
> Not really. Ideally I would like a way to reproduce that can be done
> using iperf, nc or similar simple command line tool, without requiring
> to setup iSCSI.
I think it would be tricky then. The problem hide itself well enough
that I wasn't
aware soon after upgrading since everything else works flawlessly --
nfs, ssh, web etc.

> Can you also paste the output of `ifconfig xn0`?
Here it is:
xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=503<RXCSUM,TXCSUM,TSO4,LRO>
    ether 00:18:3c:51:6e:4c
    inet 192.168.1.9 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet manual
    status: active
    nd6 options=1<PERFORMNUD>

>
> If I provided a patch for the FreeBSD kernel, would you be able to
> apply and test it?
Probably. I did this before when your XEN support for freeBSD was not
available out-of-box.
Just need to recreate all the required environments to apply the patch.

BTW, uname -a gives me the following;
>12.2-RELEASE-p11 FreeBSD 12.2-RELEASE-p11 75566f060d4(HEAD) TRUENAS  amd64

Thanks,
Timothy

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Roger Pau Monné]

On Wed, Dec 22, 2021 at 02:19:03AM +0800, G.R. wrote:
> > > I omitted all operational details with the assumption that you are familiar
> > > with TrueNAS and iSCSI setup.
> >
> > Not really. Ideally I would like a way to reproduce that can be done
> > using iperf, nc or similar simple command line tool, without requiring
> > to setup iSCSI.
> I think it would be tricky then. The problem hide itself well enough
> that I wasn't
> aware soon after upgrading since everything else works flawlessly --
> nfs, ssh, web etc.
> 
> > Can you also paste the output of `ifconfig xn0`?
> Here it is:
> xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>     options=503<RXCSUM,TXCSUM,TSO4,LRO>
>     ether 00:18:3c:51:6e:4c
>     inet 192.168.1.9 netmask 0xffffff00 broadcast 192.168.1.255
>     media: Ethernet manual
>     status: active
>     nd6 options=1<PERFORMNUD>
> 
> >
> > If I provided a patch for the FreeBSD kernel, would you be able to
> > apply and test it?
> Probably. I did this before when your XEN support for freeBSD was not
> available out-of-box.
> Just need to recreate all the required environments to apply the patch.

Could you build a debug kernel with the following patch applied and
give me the trace when it explodes?

Thanks, Roger.
---
diff --git a/sys/dev/xen/netfront/netfront.c b/sys/dev/xen/netfront/netfront.c
index fd2d97a7c70c..87bc3ecfc4dd 100644
--- a/sys/dev/xen/netfront/netfront.c
+++ b/sys/dev/xen/netfront/netfront.c
@@ -1519,8 +1519,12 @@ xn_count_frags(struct mbuf *m)
 {
 	int nfrags;
 
-	for (nfrags = 0; m != NULL; m = m->m_next)
+	for (nfrags = 0; m != NULL; m = m->m_next) {
+		KASSERT(
+		   (mtod(m, vm_offset_t) & PAGE_MASK) + m->m_len <= PAGE_SIZE,
+		    ("mbuf fragment crosses a page boundary"));
 		nfrags++;
+	}
 
 	return (nfrags);
 }

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

On Wed, Dec 22, 2021 at 3:13 AM Roger Pau Monné <roger.pau@citrix.com> wrote:

> Could you build a debug kernel with the following patch applied and
> give me the trace when it explodes?

Please find the trace and the kernel CL below.
Note, the domU get stuck into a bootloop with this assertion as the
situation will come back after domU restart and only dom0 reboot can
get the situation back to normal.
The trace I captured below is within the boot loop. I suspect the
initial trigger may look different. Will give it another try soon.

FreeBSD 12.2-RELEASE-p11 #0 c8625d629c3(truenas/12.0-stable)-dirty:
Wed Dec 22 20:26:46 UTC 2021
The repo is here: https://github.com/freenas/os.git

db:0:kdb.enter.default>  bt
Tracing pid 0 tid 101637 td 0xfffff80069cc4000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe009f121460
vpanic() at vpanic+0x197/frame 0xfffffe009f1214b0
panic() at panic+0x43/frame 0xfffffe009f121510
xn_txq_mq_start_locked() at xn_txq_mq_start_locked+0x4c6/frame
0xfffffe009f121580
xn_txq_mq_start() at xn_txq_mq_start+0x84/frame 0xfffffe009f1215b0
ether_output_frame() at ether_output_frame+0xb4/frame 0xfffffe009f1215e0
ether_output() at ether_output+0x6a5/frame 0xfffffe009f121680
ip_output() at ip_output+0x1319/frame 0xfffffe009f1217e0
tcp_output() at tcp_output+0x1dbf/frame 0xfffffe009f121980
tcp_usr_send() at tcp_usr_send+0x3c9/frame 0xfffffe009f121a40
sosend_generic() at sosend_generic+0x440/frame 0xfffffe009f121af0
sosend() at sosend+0x66/frame 0xfffffe009f121b20
icl_send_thread() at icl_send_thread+0x44e/frame 0xfffffe009f121bb0
fork_exit() at fork_exit+0x80/frame 0xfffffe009f121bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009f121bf0

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Roger Pau Monné]

On Thu, Dec 23, 2021 at 11:49:08PM +0800, G.R. wrote:
> On Wed, Dec 22, 2021 at 3:13 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
> 
> > Could you build a debug kernel with the following patch applied and
> > give me the trace when it explodes?
> 
> Please find the trace and the kernel CL below.
> Note, the domU get stuck into a bootloop with this assertion as the
> situation will come back after domU restart and only dom0 reboot can
> get the situation back to normal.
> The trace I captured below is within the boot loop. I suspect the
> initial trigger may look different. Will give it another try soon.
> 
> FreeBSD 12.2-RELEASE-p11 #0 c8625d629c3(truenas/12.0-stable)-dirty:
> Wed Dec 22 20:26:46 UTC 2021
> The repo is here: https://github.com/freenas/os.git
> 
> db:0:kdb.enter.default>  bt
> Tracing pid 0 tid 101637 td 0xfffff80069cc4000
> kdb_enter() at kdb_enter+0x37/frame 0xfffffe009f121460
> vpanic() at vpanic+0x197/frame 0xfffffe009f1214b0
> panic() at panic+0x43/frame 0xfffffe009f121510
> xn_txq_mq_start_locked() at xn_txq_mq_start_locked+0x4c6/frame
> 0xfffffe009f121580
> xn_txq_mq_start() at xn_txq_mq_start+0x84/frame 0xfffffe009f1215b0
> ether_output_frame() at ether_output_frame+0xb4/frame 0xfffffe009f1215e0
> ether_output() at ether_output+0x6a5/frame 0xfffffe009f121680
> ip_output() at ip_output+0x1319/frame 0xfffffe009f1217e0
> tcp_output() at tcp_output+0x1dbf/frame 0xfffffe009f121980
> tcp_usr_send() at tcp_usr_send+0x3c9/frame 0xfffffe009f121a40
> sosend_generic() at sosend_generic+0x440/frame 0xfffffe009f121af0
> sosend() at sosend+0x66/frame 0xfffffe009f121b20
> icl_send_thread() at icl_send_thread+0x44e/frame 0xfffffe009f121bb0
> fork_exit() at fork_exit+0x80/frame 0xfffffe009f121bf0
> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009f121bf0

Thanks. I've raised this on freensd-net for advice [0]. IMO netfront
shouldn't receive an mbuf that crosses a page boundary, but if that's
indeed a legit mbuf I will figure out the best way to handle it.

I have a clumsy patch (below) that might solve this, if you want to
give it a try.

Regards, Roger.

[0] https://lists.freebsd.org/archives/freebsd-net/2021-December/001179.html
---
diff --git a/sys/dev/xen/netfront/netfront.c b/sys/dev/xen/netfront/netfront.c
index 87bc3ecfc4dd..c8f807778b75 100644
--- a/sys/dev/xen/netfront/netfront.c
+++ b/sys/dev/xen/netfront/netfront.c
@@ -1529,6 +1529,35 @@ xn_count_frags(struct mbuf *m)
 	return (nfrags);
 }
 
+static inline int fragment(struct mbuf *m)
+{
+	while (m != NULL) {
+		vm_offset_t offset = mtod(m, vm_offset_t) & PAGE_MASK;
+
+		if (offset + m->m_len > PAGE_SIZE) {
+			/* Split mbuf because it crosses a page boundary. */
+			struct mbuf *m_new = m_getcl(M_NOWAIT, MT_DATA, 0);
+
+			if (m_new == NULL)
+				return (ENOMEM);
+
+			m_copydata(m, 0, m->m_len - (PAGE_SIZE - offset),
+			    mtod(m_new, caddr_t));
+
+			/* Set adjusted mbuf sizes. */
+			m_new->m_len = m->m_len - (PAGE_SIZE - offset);
+			m->m_len = PAGE_SIZE - offset;
+
+			/* Insert new mbuf into chain. */
+			m_new->m_next = m->m_next;
+			m->m_next = m_new;
+		}
+		m = m->m_next;
+	}
+
+	return (0);
+}
+
 /**
  * Given an mbuf chain, make sure we have enough room and then push
  * it onto the transmit ring.
@@ -1541,6 +1570,12 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 	struct ifnet *ifp = np->xn_ifp;
 	u_int nfrags;
 	int otherend_id;
+	int rc;
+
+	/* Fragment if mbuf crosses a page boundary. */
+	rc = fragment(m_head);
+	if (rc != 0)
+		return (rc);
 
 	/**
 	 * Defragment the mbuf if necessary.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

> > Please find the trace and the kernel CL below.
> > Note, the domU get stuck into a bootloop with this assertion as the
> > situation will come back after domU restart and only dom0 reboot can
> > get the situation back to normal.
> > The trace I captured below is within the boot loop. I suspect the
> > initial trigger may look different. Will give it another try soon.
> >
I think I figured out the cause of the boot loop.
It was not due to some mystery offender packet from the FreeBSD domU
surviving across reboot,
but because the windows iSCSI initiator keeps retrying :-).

That said, I did pay some prices figuring this out.
The boot-loop seems to have brought my box into a weird state that those disks
behind the controller keep detaching soon after NAS domU reboot.
Rebooting the physical host does not help this time.
I was almost desperate but thankfully running the NAS on the physical
host directly still works.
And switching it back to domU together with config reloading fixed
this problem :-)
Not sure if it was the domU config being corrupted or something left
sticky in the PCI pass-through?

> Thanks. I've raised this on freensd-net for advice [0]. IMO netfront
> shouldn't receive an mbuf that crosses a page boundary, but if that's
> indeed a legit mbuf I will figure out the best way to handle it.
>
> I have a clumsy patch (below) that might solve this, if you want to
> give it a try.

Applied the patch and it worked like a charm!
Thank you so much for your quick help!
Wish you a wonderful holiday!

Thanks,
G.R.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

> > Thanks. I've raised this on freensd-net for advice [0]. IMO netfront
> > shouldn't receive an mbuf that crosses a page boundary, but if that's
> > indeed a legit mbuf I will figure out the best way to handle it.
> >
> > I have a clumsy patch (below) that might solve this, if you want to
> > give it a try.
>
> Applied the patch and it worked like a charm!
> Thank you so much for your quick help!
> Wish you a wonderful holiday!

I may have said too quickly...
With the patch I can attach the iscsi disk and neither the dom0 nor
the NAS domU complains this time.
But when I attempt to mount the attached disk it reports I/O errors randomly.
By randomly I mean different disks behave differently...
I don't see any error logs from kernels this time.
(most of the iscsi disks are NTFS FS and mounted through the user mode
fuse library)
But since I have a local backup copy of the image, I can confirm that
mounting that backup image does not result in any I/O error.
Looks like something is still broken here...

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Roger Pau Monné]

On Sun, Dec 26, 2021 at 02:06:55AM +0800, G.R. wrote:
> > > Thanks. I've raised this on freensd-net for advice [0]. IMO netfront
> > > shouldn't receive an mbuf that crosses a page boundary, but if that's
> > > indeed a legit mbuf I will figure out the best way to handle it.
> > >
> > > I have a clumsy patch (below) that might solve this, if you want to
> > > give it a try.
> >
> > Applied the patch and it worked like a charm!
> > Thank you so much for your quick help!
> > Wish you a wonderful holiday!
> 
> I may have said too quickly...
> With the patch I can attach the iscsi disk and neither the dom0 nor
> the NAS domU complains this time.
> But when I attempt to mount the attached disk it reports I/O errors randomly.
> By randomly I mean different disks behave differently...
> I don't see any error logs from kernels this time.
> (most of the iscsi disks are NTFS FS and mounted through the user mode
> fuse library)
> But since I have a local backup copy of the image, I can confirm that
> mounting that backup image does not result in any I/O error.
> Looks like something is still broken here...

Indeed. That patch was likely too simple, and didn't properly handle
the split of mbuf data buffers.

I have another version based on using sglist, which I think it's also
a worthwhile change for netfront. Can you please give it a try? I've
done a very simple test and seems fine, but you certainly have more
interesting cases.

You will have to apply it on top of a clean tree, without any of the
other patches applied.

Thanks, Roger.
---
diff --git a/sys/dev/xen/netfront/netfront.c b/sys/dev/xen/netfront/netfront.c
index 8dba5a8dc6d5..37ea7b1fa059 100644
--- a/sys/dev/xen/netfront/netfront.c
+++ b/sys/dev/xen/netfront/netfront.c
@@ -33,6 +33,8 @@ __FBSDID("$FreeBSD$");
 #include "opt_inet.h"
 #include "opt_inet6.h"
 
+#include <sys/types.h>
+
 #include <sys/param.h>
 #include <sys/sockio.h>
 #include <sys/limits.h>
@@ -40,6 +42,7 @@ __FBSDID("$FreeBSD$");
 #include <sys/malloc.h>
 #include <sys/module.h>
 #include <sys/kernel.h>
+#include <sys/sglist.h>
 #include <sys/socket.h>
 #include <sys/sysctl.h>
 #include <sys/taskqueue.h>
@@ -199,6 +202,12 @@ struct netfront_txq {
 	struct taskqueue 	*tq;
 	struct task       	defrtask;
 
+	struct sglist 		*segments;
+	struct mbuf_refcount {
+		struct m_tag 	tag;
+		u_int 		count;
+	}			refcount_tag[NET_TX_RING_SIZE + 1];
+
 	bool			full;
 };
 
@@ -301,6 +310,38 @@ xn_get_rx_ref(struct netfront_rxq *rxq, RING_IDX ri)
 	return (ref);
 }
 
+#define MTAG_REFCOUNT 0
+
+static void mbuf_grab(uint32_t cookie, struct mbuf *m)
+{
+	struct mbuf_refcount *ref;
+
+	ref = (struct mbuf_refcount *)m_tag_locate(m, cookie, MTAG_REFCOUNT,
+	    NULL);
+	KASSERT(ref != NULL, ("Cannot find refcount"));
+	ref->count++;
+}
+
+static void mbuf_release(uint32_t cookie, struct mbuf *m)
+{
+	struct mbuf_refcount *ref;
+
+	ref = (struct mbuf_refcount *)m_tag_locate(m, cookie, MTAG_REFCOUNT,
+	    NULL);
+	KASSERT(ref != NULL, ("Cannot find refcount"));
+	KASSERT(ref->count > 0, ("Invalid reference count"));
+
+	if(--ref->count == 0)
+		m_freem(m);
+}
+
+static void tag_free(struct m_tag *t)
+{
+	struct mbuf_refcount *ref = (struct mbuf_refcount *)t;
+
+	KASSERT(ref->count == 0, ("Free mbuf tag with pending refcnt"));
+}
+
 #define IPRINTK(fmt, args...) \
     printf("[XEN] " fmt, ##args)
 #ifdef INVARIANTS
@@ -778,7 +819,7 @@ disconnect_txq(struct netfront_txq *txq)
 static void
 destroy_txq(struct netfront_txq *txq)
 {
-
+	sglist_free(txq->segments);
 	free(txq->ring.sring, M_DEVBUF);
 	buf_ring_free(txq->br, M_DEVBUF);
 	taskqueue_drain_all(txq->tq);
@@ -826,6 +867,11 @@ setup_txqs(device_t dev, struct netfront_info *info,
 		for (i = 0; i <= NET_TX_RING_SIZE; i++) {
 			txq->mbufs[i] = (void *) ((u_long) i+1);
 			txq->grant_ref[i] = GRANT_REF_INVALID;
+			m_tag_setup(&txq->refcount_tag[i].tag,
+			    (unsigned long)txq, MTAG_REFCOUNT,
+			    sizeof(txq->refcount_tag[i]) -
+			    sizeof(txq->refcount_tag[i].tag));
+			txq->refcount_tag[i].tag.m_tag_free = &tag_free;
 		}
 		txq->mbufs[NET_TX_RING_SIZE] = (void *)0;
 
@@ -874,10 +920,18 @@ setup_txqs(device_t dev, struct netfront_info *info,
 			device_printf(dev, "xen_intr_alloc_and_bind_local_port failed\n");
 			goto fail_bind_port;
 		}
+
+		txq->segments = sglist_alloc(MAX_TX_REQ_FRAGS, M_WAITOK);
+		if (txq->segments == NULL) {
+			device_printf(dev, "failed to allocate sglist\n");
+			goto fail_sglist;
+		}
 	}
 
 	return (0);
 
+fail_sglist:
+	xen_intr_unbind(&txq->xen_intr_handle);
 fail_bind_port:
 	taskqueue_drain_all(txq->tq);
 fail_start_thread:
@@ -1041,7 +1095,7 @@ xn_release_tx_bufs(struct netfront_txq *txq)
 		if (txq->mbufs_cnt < 0) {
 			panic("%s: tx_chain_cnt must be >= 0", __func__);
 		}
-		m_free(m);
+		mbuf_release((unsigned long)txq, m);
 	}
 }
 
@@ -1311,7 +1365,7 @@ xn_txeof(struct netfront_txq *txq)
 			txq->mbufs[id] = NULL;
 			add_id_to_freelist(txq->mbufs, id);
 			txq->mbufs_cnt--;
-			m_free(m);
+			mbuf_release((unsigned long)txq, m);
 			/* Only mark the txq active if we've freed up at least one slot to try */
 			ifp->if_drv_flags &= ~IFF_DRV_OACTIVE;
 		}
@@ -1507,22 +1561,6 @@ xn_get_responses(struct netfront_rxq *rxq,
 	return (err);
 }
 
-/**
- * \brief Count the number of fragments in an mbuf chain.
- *
- * Surprisingly, there isn't an M* macro for this.
- */
-static inline int
-xn_count_frags(struct mbuf *m)
-{
-	int nfrags;
-
-	for (nfrags = 0; m != NULL; m = m->m_next)
-		nfrags++;
-
-	return (nfrags);
-}
-
 /**
  * Given an mbuf chain, make sure we have enough room and then push
  * it onto the transmit ring.
@@ -1530,16 +1568,22 @@ xn_count_frags(struct mbuf *m)
 static int
 xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 {
-	struct mbuf *m;
 	struct netfront_info *np = txq->info;
 	struct ifnet *ifp = np->xn_ifp;
-	u_int nfrags;
-	int otherend_id;
+	u_int nfrags, i;
+	int otherend_id, rc;
+
+	sglist_reset(txq->segments);
+	rc = sglist_append_mbuf(txq->segments, m_head);
+	if (rc != 0) {
+		m_freem(m_head);
+		return (rc);
+	}
 
 	/**
 	 * Defragment the mbuf if necessary.
 	 */
-	nfrags = xn_count_frags(m_head);
+	nfrags = txq->segments->sg_nseg;
 
 	/*
 	 * Check to see whether this request is longer than netback
@@ -1551,6 +1595,8 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 	 * the Linux network stack.
 	 */
 	if (nfrags > np->maxfrags) {
+		struct mbuf *m;
+
 		m = m_defrag(m_head, M_NOWAIT);
 		if (!m) {
 			/*
@@ -1561,11 +1607,15 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 			return (EMSGSIZE);
 		}
 		m_head = m;
+		sglist_reset(txq->segments);
+		rc = sglist_append_mbuf(txq->segments, m_head);
+		if (rc != 0) {
+			m_freem(m_head);
+			return (rc);
+		}
+		nfrags = txq->segments->sg_nseg;
 	}
 
-	/* Determine how many fragments now exist */
-	nfrags = xn_count_frags(m_head);
-
 	/*
 	 * Check to see whether the defragmented packet has too many
 	 * segments for the Linux netback driver.
@@ -1604,14 +1654,15 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 	 * the fragment pointers. Stop when we run out
 	 * of fragments or hit the end of the mbuf chain.
 	 */
-	m = m_head;
 	otherend_id = xenbus_get_otherend_id(np->xbdev);
-	for (m = m_head; m; m = m->m_next) {
+	for (i = 0; i < nfrags; i++) {
 		netif_tx_request_t *tx;
 		uintptr_t id;
 		grant_ref_t ref;
 		u_long mfn; /* XXX Wrong type? */
+		struct sglist_seg *seg;
 
+		seg = &txq->segments->sg_segs[i];
 		tx = RING_GET_REQUEST(&txq->ring, txq->ring.req_prod_pvt);
 		id = get_id_from_freelist(txq->mbufs);
 		if (id == 0)
@@ -1621,17 +1672,22 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 		if (txq->mbufs_cnt > NET_TX_RING_SIZE)
 			panic("%s: tx_chain_cnt must be <= NET_TX_RING_SIZE\n",
 			    __func__);
-		txq->mbufs[id] = m;
+		if (i == 0)
+			m_tag_prepend(m_head, &txq->refcount_tag[id].tag);
+		mbuf_grab((unsigned long)txq, m_head);
+		txq->mbufs[id] = m_head;
 		tx->id = id;
 		ref = gnttab_claim_grant_reference(&txq->gref_head);
 		KASSERT((short)ref >= 0, ("Negative ref"));
-		mfn = virt_to_mfn(mtod(m, vm_offset_t));
+		mfn = atop(seg->ss_paddr);
 		gnttab_grant_foreign_access_ref(ref, otherend_id,
 		    mfn, GNTMAP_readonly);
 		tx->gref = txq->grant_ref[id] = ref;
-		tx->offset = mtod(m, vm_offset_t) & (PAGE_SIZE - 1);
+		tx->offset = seg->ss_paddr & PAGE_MASK;
+		KASSERT(tx->offset + seg->ss_len <= PAGE_SIZE,
+		    ("mbuf segment crosses a page boundary"));
 		tx->flags = 0;
-		if (m == m_head) {
+		if (i == 0) {
 			/*
 			 * The first fragment has the entire packet
 			 * size, subsequent fragments have just the
@@ -1640,7 +1696,7 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 			 * subtracting the sizes of the other
 			 * fragments.
 			 */
-			tx->size = m->m_pkthdr.len;
+			tx->size = m_head->m_pkthdr.len;
 
 			/*
 			 * The first fragment contains the checksum flags
@@ -1654,12 +1710,12 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 			 * so we have to test for CSUM_TSO
 			 * explicitly.
 			 */
-			if (m->m_pkthdr.csum_flags
+			if (m_head->m_pkthdr.csum_flags
 			    & (CSUM_DELAY_DATA | CSUM_TSO)) {
 				tx->flags |= (NETTXF_csum_blank
 				    | NETTXF_data_validated);
 			}
-			if (m->m_pkthdr.csum_flags & CSUM_TSO) {
+			if (m_head->m_pkthdr.csum_flags & CSUM_TSO) {
 				struct netif_extra_info *gso =
 					(struct netif_extra_info *)
 					RING_GET_REQUEST(&txq->ring,
@@ -1667,7 +1723,7 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 
 				tx->flags |= NETTXF_extra_info;
 
-				gso->u.gso.size = m->m_pkthdr.tso_segsz;
+				gso->u.gso.size = m_head->m_pkthdr.tso_segsz;
 				gso->u.gso.type =
 					XEN_NETIF_GSO_TYPE_TCPV4;
 				gso->u.gso.pad = 0;
@@ -1677,9 +1733,9 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 				gso->flags = 0;
 			}
 		} else {
-			tx->size = m->m_len;
+			tx->size = seg->ss_len;
 		}
-		if (m->m_next)
+		if (i != nfrags - 1)
 			tx->flags |= NETTXF_more_data;
 
 		txq->ring.req_prod_pvt++;

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Roger Pau Monné]

Adding xen-devel back.

On Wed, Dec 29, 2021 at 01:44:18AM +0800, G.R. wrote:
> On Tue, Dec 28, 2021 at 3:05 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Sun, Dec 26, 2021 at 02:06:55AM +0800, G.R. wrote:
> > > > > Thanks. I've raised this on freensd-net for advice [0]. IMO netfront
> > > > > shouldn't receive an mbuf that crosses a page boundary, but if that's
> > > > > indeed a legit mbuf I will figure out the best way to handle it.
> > > > >
> > > > > I have a clumsy patch (below) that might solve this, if you want to
> > > > > give it a try.
> > > >
> > > > Applied the patch and it worked like a charm!
> > > > Thank you so much for your quick help!
> > > > Wish you a wonderful holiday!
> > >
> > > I may have said too quickly...
> > > With the patch I can attach the iscsi disk and neither the dom0 nor
> > > the NAS domU complains this time.
> > > But when I attempt to mount the attached disk it reports I/O errors randomly.
> > > By randomly I mean different disks behave differently...
> > > I don't see any error logs from kernels this time.
> > > (most of the iscsi disks are NTFS FS and mounted through the user mode
> > > fuse library)
> > > But since I have a local backup copy of the image, I can confirm that
> > > mounting that backup image does not result in any I/O error.
> > > Looks like something is still broken here...
> >
> > Indeed. That patch was likely too simple, and didn't properly handle
> > the split of mbuf data buffers.
> >
> > I have another version based on using sglist, which I think it's also
> > a worthwhile change for netfront. Can you please give it a try? I've
> > done a very simple test and seems fine, but you certainly have more
> > interesting cases.
> >
> > You will have to apply it on top of a clean tree, without any of the
> > other patches applied.
> 
> Unfortunately this new version is even worse.
> It not only does not fix the known issue on iSCSI, but also creating
> regression on NFS.
> The regression on NFS is kind of random that it takes a
> non-deterministic time to show up.
> Here is a stack trace for reference:
> db:0:kdb.enter.default>  bt
> Tracing pid 1696 tid 100622 td 0xfffff800883d5740
> kdb_enter() at kdb_enter+0x37/frame 0xfffffe009f80d900
> vpanic() at vpanic+0x197/frame 0xfffffe009f80d950
> panic() at panic+0x43/frame 0xfffffe009f80d9b0
> xn_txq_mq_start_locked() at xn_txq_mq_start_locked+0x5bc/frame
> 0xfffffe009f80da50

I think this is hitting a KASSERT, could you paste the text printed as
part of the panic (not just he backtrace)?

Sorry this is taking a bit of time to solve.

Thanks!

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

[-- Attachment #1: Type: text/plain, Size: 419 bytes --]

>
> I think this is hitting a KASSERT, could you paste the text printed as
> part of the panic (not just he backtrace)?
>
> Sorry this is taking a bit of time to solve.
>
> Thanks!
>
Sorry that I didn't make it clear in the first place.
It is the same cross boundary assertion.

Also sorry about the email format if it mess up in your side. I am typing
in the Gmail app and don't find a way to switch to plain text.

>

[-- Attachment #2: Type: text/html, Size: 866 bytes --]

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Roger Pau Monné]

On Wed, Dec 29, 2021 at 05:13:00PM +0800, G.R. wrote:
> >
> > I think this is hitting a KASSERT, could you paste the text printed as
> > part of the panic (not just he backtrace)?
> >
> > Sorry this is taking a bit of time to solve.
> >
> > Thanks!
> >
> Sorry that I didn't make it clear in the first place.
> It is the same cross boundary assertion.

I see. After looking at the code it seems like sglist will coalesce
contiguous physical ranges without taking page boundaries into
account, which is not suitable for our purpose here. I guess I will
either have to modify sglist, or switch to using bus_dma. The main
problem with using bus_dma is that it will require bigger changes to
netfront I think.

Thanks, Roger.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Roger Pau Monné]

On Wed, Dec 29, 2021 at 11:27:50AM +0100, Roger Pau Monné wrote:
> On Wed, Dec 29, 2021 at 05:13:00PM +0800, G.R. wrote:
> > >
> > > I think this is hitting a KASSERT, could you paste the text printed as
> > > part of the panic (not just he backtrace)?
> > >
> > > Sorry this is taking a bit of time to solve.
> > >
> > > Thanks!
> > >
> > Sorry that I didn't make it clear in the first place.
> > It is the same cross boundary assertion.
> 
> I see. After looking at the code it seems like sglist will coalesce
> contiguous physical ranges without taking page boundaries into
> account, which is not suitable for our purpose here. I guess I will
> either have to modify sglist, or switch to using bus_dma. The main
> problem with using bus_dma is that it will require bigger changes to
> netfront I think.

I have a crappy patch to use bus_dma. It's not yet ready for upstream
but you might want to give it a try to see if it solves the cross page
boundary issues.

Thanks, Roger.
---
diff --git a/sys/dev/xen/netfront/netfront.c b/sys/dev/xen/netfront/netfront.c
index 8dba5a8dc6d5..693ef25c8783 100644
--- a/sys/dev/xen/netfront/netfront.c
+++ b/sys/dev/xen/netfront/netfront.c
@@ -71,6 +71,8 @@ __FBSDID("$FreeBSD$");
 #include <xen/interface/io/netif.h>
 #include <xen/xenbus/xenbusvar.h>
 
+#include <machine/bus.h>
+
 #include "xenbus_if.h"
 
 /* Features supported by all backends.  TSO and LRO can be negotiated */
@@ -199,6 +201,12 @@ struct netfront_txq {
 	struct taskqueue 	*tq;
 	struct task       	defrtask;
 
+	bus_dmamap_t		dma_map;
+	struct mbuf_refcount {
+		struct m_tag 	tag;
+		u_int 		count;
+	}			refcount_tag[NET_TX_RING_SIZE + 1];
+
 	bool			full;
 };
 
@@ -221,6 +229,8 @@ struct netfront_info {
 
 	struct ifmedia		sc_media;
 
+	bus_dma_tag_t		dma_tag;
+
 	bool			xn_reset;
 };
 
@@ -301,6 +311,39 @@ xn_get_rx_ref(struct netfront_rxq *rxq, RING_IDX ri)
 	return (ref);
 }
 
+#define MTAG_COOKIE 1218492000
+#define MTAG_REFCOUNT 0
+
+static void mbuf_grab(struct mbuf *m)
+{
+	struct mbuf_refcount *ref;
+
+	ref = (struct mbuf_refcount *)m_tag_locate(m, MTAG_COOKIE,
+	    MTAG_REFCOUNT, NULL);
+	KASSERT(ref != NULL, ("Cannot find refcount"));
+	ref->count++;
+}
+
+static void mbuf_release(struct mbuf *m)
+{
+	struct mbuf_refcount *ref;
+
+	ref = (struct mbuf_refcount *)m_tag_locate(m, MTAG_COOKIE,
+	    MTAG_REFCOUNT, NULL);
+	KASSERT(ref != NULL, ("Cannot find refcount"));
+	KASSERT(ref->count > 0, ("Invalid reference count"));
+
+	if (--ref->count == 0)
+		m_freem(m);
+}
+
+static void tag_free(struct m_tag *t)
+{
+	struct mbuf_refcount *ref = (struct mbuf_refcount *)t;
+
+	KASSERT(ref->count == 0, ("Free mbuf tag with pending refcnt"));
+}
+
 #define IPRINTK(fmt, args...) \
     printf("[XEN] " fmt, ##args)
 #ifdef INVARIANTS
@@ -783,6 +826,7 @@ destroy_txq(struct netfront_txq *txq)
 	buf_ring_free(txq->br, M_DEVBUF);
 	taskqueue_drain_all(txq->tq);
 	taskqueue_free(txq->tq);
+	bus_dmamap_destroy(txq->info->dma_tag, txq->dma_map);
 }
 
 static void
@@ -826,6 +870,11 @@ setup_txqs(device_t dev, struct netfront_info *info,
 		for (i = 0; i <= NET_TX_RING_SIZE; i++) {
 			txq->mbufs[i] = (void *) ((u_long) i+1);
 			txq->grant_ref[i] = GRANT_REF_INVALID;
+			m_tag_setup(&txq->refcount_tag[i].tag,
+			    MTAG_COOKIE, MTAG_REFCOUNT,
+			    sizeof(txq->refcount_tag[i]) -
+			    sizeof(txq->refcount_tag[i].tag));
+			txq->refcount_tag[i].tag.m_tag_free = &tag_free;
 		}
 		txq->mbufs[NET_TX_RING_SIZE] = (void *)0;
 
@@ -874,10 +923,18 @@ setup_txqs(device_t dev, struct netfront_info *info,
 			device_printf(dev, "xen_intr_alloc_and_bind_local_port failed\n");
 			goto fail_bind_port;
 		}
+
+		error = bus_dmamap_create(info->dma_tag, 0, &txq->dma_map);
+		if (error != 0) {
+			device_printf(dev, "failed to create dma map\n");
+			goto fail_dma_map;
+		}
 	}
 
 	return (0);
 
+fail_dma_map:
+	xen_intr_unbind(&txq->xen_intr_handle);
 fail_bind_port:
 	taskqueue_drain_all(txq->tq);
 fail_start_thread:
@@ -1041,7 +1098,7 @@ xn_release_tx_bufs(struct netfront_txq *txq)
 		if (txq->mbufs_cnt < 0) {
 			panic("%s: tx_chain_cnt must be >= 0", __func__);
 		}
-		m_free(m);
+		mbuf_release(m);
 	}
 }
 
@@ -1311,7 +1368,7 @@ xn_txeof(struct netfront_txq *txq)
 			txq->mbufs[id] = NULL;
 			add_id_to_freelist(txq->mbufs, id);
 			txq->mbufs_cnt--;
-			m_free(m);
+			mbuf_release(m);
 			/* Only mark the txq active if we've freed up at least one slot to try */
 			ifp->if_drv_flags &= ~IFF_DRV_OACTIVE;
 		}
@@ -1530,27 +1587,18 @@ xn_count_frags(struct mbuf *m)
 static int
 xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 {
-	struct mbuf *m;
 	struct netfront_info *np = txq->info;
 	struct ifnet *ifp = np->xn_ifp;
-	u_int nfrags;
-	int otherend_id;
+	int otherend_id, error, nfrags;
+	unsigned int i;
+	bus_dma_segment_t segs[MAX_TX_REQ_FRAGS];
 
-	/**
-	 * Defragment the mbuf if necessary.
-	 */
-	nfrags = xn_count_frags(m_head);
+	error = bus_dmamap_load_mbuf_sg(np->dma_tag, txq->dma_map, m_head,
+	    segs, &nfrags, 0);
+	if (error == EFBIG || nfrags > np->maxfrags) {
+		struct mbuf *m;
 
-	/*
-	 * Check to see whether this request is longer than netback
-	 * can handle, and try to defrag it.
-	 */
-	/**
-	 * It is a bit lame, but the netback driver in Linux can't
-	 * deal with nfrags > MAX_TX_REQ_FRAGS, which is a quirk of
-	 * the Linux network stack.
-	 */
-	if (nfrags > np->maxfrags) {
+		bus_dmamap_unload(np->dma_tag, txq->dma_map);
 		m = m_defrag(m_head, M_NOWAIT);
 		if (!m) {
 			/*
@@ -1561,15 +1609,18 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 			return (EMSGSIZE);
 		}
 		m_head = m;
+		error = bus_dmamap_load_mbuf_sg(np->dma_tag, txq->dma_map,
+		    m_head, segs, &nfrags, 0);
+		if (error != 0 || nfrags > np->maxfrags) {
+			bus_dmamap_unload(np->dma_tag, txq->dma_map);
+			m_freem(m_head);
+			return (error ?: EFBIG);
+		}
+	} else if (error != 0) {
+		m_freem(m_head);
+		return (error);
 	}
 
-	/* Determine how many fragments now exist */
-	nfrags = xn_count_frags(m_head);
-
-	/*
-	 * Check to see whether the defragmented packet has too many
-	 * segments for the Linux netback driver.
-	 */
 	/**
 	 * The FreeBSD TCP stack, with TSO enabled, can produce a chain
 	 * of mbufs longer than Linux can handle.  Make sure we don't
@@ -1604,9 +1655,8 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 	 * the fragment pointers. Stop when we run out
 	 * of fragments or hit the end of the mbuf chain.
 	 */
-	m = m_head;
 	otherend_id = xenbus_get_otherend_id(np->xbdev);
-	for (m = m_head; m; m = m->m_next) {
+	for (i = 0; i < nfrags; i++) {
 		netif_tx_request_t *tx;
 		uintptr_t id;
 		grant_ref_t ref;
@@ -1621,17 +1671,22 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 		if (txq->mbufs_cnt > NET_TX_RING_SIZE)
 			panic("%s: tx_chain_cnt must be <= NET_TX_RING_SIZE\n",
 			    __func__);
-		txq->mbufs[id] = m;
+		if (i == 0)
+			m_tag_prepend(m_head, &txq->refcount_tag[id].tag);
+		mbuf_grab(m_head);
+		txq->mbufs[id] = m_head;
 		tx->id = id;
 		ref = gnttab_claim_grant_reference(&txq->gref_head);
 		KASSERT((short)ref >= 0, ("Negative ref"));
-		mfn = virt_to_mfn(mtod(m, vm_offset_t));
+		mfn = atop(segs[i].ds_addr);
 		gnttab_grant_foreign_access_ref(ref, otherend_id,
 		    mfn, GNTMAP_readonly);
 		tx->gref = txq->grant_ref[id] = ref;
-		tx->offset = mtod(m, vm_offset_t) & (PAGE_SIZE - 1);
+		tx->offset = segs[i].ds_addr & PAGE_MASK;
+		KASSERT(tx->offset + segs[i].ds_len <= PAGE_SIZE,
+		    ("mbuf segment crosses a page boundary"));
 		tx->flags = 0;
-		if (m == m_head) {
+		if (i == 0) {
 			/*
 			 * The first fragment has the entire packet
 			 * size, subsequent fragments have just the
@@ -1640,7 +1695,7 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 			 * subtracting the sizes of the other
 			 * fragments.
 			 */
-			tx->size = m->m_pkthdr.len;
+			tx->size = m_head->m_pkthdr.len;
 
 			/*
 			 * The first fragment contains the checksum flags
@@ -1654,12 +1709,12 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 			 * so we have to test for CSUM_TSO
 			 * explicitly.
 			 */
-			if (m->m_pkthdr.csum_flags
+			if (m_head->m_pkthdr.csum_flags
 			    & (CSUM_DELAY_DATA | CSUM_TSO)) {
 				tx->flags |= (NETTXF_csum_blank
 				    | NETTXF_data_validated);
 			}
-			if (m->m_pkthdr.csum_flags & CSUM_TSO) {
+			if (m_head->m_pkthdr.csum_flags & CSUM_TSO) {
 				struct netif_extra_info *gso =
 					(struct netif_extra_info *)
 					RING_GET_REQUEST(&txq->ring,
@@ -1667,7 +1722,7 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 
 				tx->flags |= NETTXF_extra_info;
 
-				gso->u.gso.size = m->m_pkthdr.tso_segsz;
+				gso->u.gso.size = m_head->m_pkthdr.tso_segsz;
 				gso->u.gso.type =
 					XEN_NETIF_GSO_TYPE_TCPV4;
 				gso->u.gso.pad = 0;
@@ -1677,13 +1732,14 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 				gso->flags = 0;
 			}
 		} else {
-			tx->size = m->m_len;
+			tx->size = segs[i].ds_len;
 		}
-		if (m->m_next)
+		if (i != nfrags - 1)
 			tx->flags |= NETTXF_more_data;
 
 		txq->ring.req_prod_pvt++;
 	}
+	bus_dmamap_unload(np->dma_tag, txq->dma_map);
 	BPF_MTAP(ifp, m_head);
 
 	if_inc_counter(ifp, IFCOUNTER_OPACKETS, 1);
@@ -2244,7 +2300,20 @@ create_netdev(device_t dev)
     	ether_ifattach(ifp, np->mac);
 	netfront_carrier_off(np);
 
-	return (0);
+	err = bus_dma_tag_create(
+	    bus_get_dma_tag(dev),		/* parent */
+	    1, PAGE_SIZE,			/* algnmnt, boundary */
+	    BUS_SPACE_MAXADDR,			/* lowaddr */
+	    BUS_SPACE_MAXADDR,			/* highaddr */
+	    NULL, NULL,				/* filter, filterarg */
+	    PAGE_SIZE * MAX_TX_REQ_FRAGS,	/* max request size */
+	    MAX_TX_REQ_FRAGS,			/* max segments */
+	    PAGE_SIZE,				/* maxsegsize */
+	    BUS_DMA_ALLOCNOW,			/* flags */
+	    NULL, NULL,				/* lockfunc, lockarg */
+	    &np->dma_tag);
+
+	return (err);
 
 error:
 	KASSERT(err != 0, ("Error path with no error code specified"));
@@ -2277,6 +2346,7 @@ netif_free(struct netfront_info *np)
 	if_free(np->xn_ifp);
 	np->xn_ifp = NULL;
 	ifmedia_removeall(&np->sc_media);
+	bus_dma_tag_destroy(np->dma_tag);
 }
 
 static void

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

On Thu, Dec 30, 2021 at 3:07 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Wed, Dec 29, 2021 at 11:27:50AM +0100, Roger Pau Monné wrote:
> > On Wed, Dec 29, 2021 at 05:13:00PM +0800, G.R. wrote:
> > > >
> > > > I think this is hitting a KASSERT, could you paste the text printed as
> > > > part of the panic (not just he backtrace)?
> > > >
> > > > Sorry this is taking a bit of time to solve.
> > > >
> > > > Thanks!
> > > >
> > > Sorry that I didn't make it clear in the first place.
> > > It is the same cross boundary assertion.
> >
> > I see. After looking at the code it seems like sglist will coalesce
> > contiguous physical ranges without taking page boundaries into
> > account, which is not suitable for our purpose here. I guess I will
> > either have to modify sglist, or switch to using bus_dma. The main
> > problem with using bus_dma is that it will require bigger changes to
> > netfront I think.
>
> I have a crappy patch to use bus_dma. It's not yet ready for upstream
> but you might want to give it a try to see if it solves the cross page
> boundary issues.
>
I think this version is better.
It fixed the mbuf cross boundary issue and allowed me to boot from one
disk image successfully.
But seems like this patch is not stable enough yet and has its own
issue -- memory is not properly released?
The stack trace is likely not useful, but anyway...

Context:
pmap_growkernel: no memory to grow kernel

<118>Dec 30 22:55:47 nas kernel[2164]: Last message 'pid 1066
(python3.9)' repeated 1 times, suppressed by syslog-ng on nas.rglab.us
<118>Dec 30 22:55:47 nas kernel: pid 2086 (python3.9), jid 0, uid 0,
was killed: out of swap space
panic: pmap_growkernel: no memory to grow kernel
cpuid = 1
time = 1640876153
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe009b971210
vpanic() at vpanic+0x17b/frame 0xfffffe009b971260
panic() at panic+0x43/frame 0xfffffe009b9712c0
pmap_growkernel() at pmap_growkernel+0x2f1/frame 0xfffffe009b971300
vm_map_insert() at vm_map_insert+0x27b/frame 0xfffffe009b971390
vm_map_find() at vm_map_find+0x5ed/frame 0xfffffe009b971470
kva_import() at kva_import+0x3c/frame 0xfffffe009b9714b0
vmem_try_fetch() at vmem_try_fetch+0xde/frame 0xfffffe009b971500
vmem_xalloc() at vmem_xalloc+0x4db/frame 0xfffffe009b971580
kva_import_domain() at kva_import_domain+0x36/frame 0xfffffe009b9715b0
vmem_try_fetch() at vmem_try_fetch+0xde/frame 0xfffffe009b971600
vmem_xalloc() at vmem_xalloc+0x4db/frame 0xfffffe009b971680
vmem_alloc() at vmem_alloc+0x8a/frame 0xfffffe009b9716d0
kmem_malloc_domainset() at kmem_malloc_domainset+0x92/frame 0xfffffe009b971740
keg_alloc_slab() at keg_alloc_slab+0xfa/frame 0xfffffe009b9717a0
keg_fetch_slab() at keg_fetch_slab+0xfe/frame 0xfffffe009b971830
zone_fetch_slab() at zone_fetch_slab+0x61/frame 0xfffffe009b971870
zone_import() at zone_import+0x75/frame 0xfffffe009b9718f0
zone_alloc_item() at zone_alloc_item+0x56/frame 0xfffffe009b971930
abd_borrow_buf() at abd_borrow_buf+0x1f/frame 0xfffffe009b971950
vdev_geom_io_start() at vdev_geom_io_start+0x189/frame 0xfffffe009b971980
zio_vdev_io_start() at zio_vdev_io_start+0x1e4/frame 0xfffffe009b9719d0
zio_nowait() at zio_nowait+0x11a/frame 0xfffffe009b971a30
vdev_queue_io_done() at vdev_queue_io_done+0x1b8/frame 0xfffffe009b971a90
zio_vdev_io_done() at zio_vdev_io_done+0xe3/frame 0xfffffe009b971ad0
zio_execute() at zio_execute+0x6a/frame 0xfffffe009b971b20
taskqueue_run_locked() at taskqueue_run_locked+0x168/frame 0xfffffe009b971b80
taskqueue_thread_loop() at taskqueue_thread_loop+0x94/frame 0xfffffe009b971bb0
fork_exit() at fork_exit+0x80/frame 0xfffffe009b971bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009b971bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Roger Pau Monné]

On Thu, Dec 30, 2021 at 11:12:57PM +0800, G.R. wrote:
> On Thu, Dec 30, 2021 at 3:07 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Wed, Dec 29, 2021 at 11:27:50AM +0100, Roger Pau Monné wrote:
> > > On Wed, Dec 29, 2021 at 05:13:00PM +0800, G.R. wrote:
> > > > >
> > > > > I think this is hitting a KASSERT, could you paste the text printed as
> > > > > part of the panic (not just he backtrace)?
> > > > >
> > > > > Sorry this is taking a bit of time to solve.
> > > > >
> > > > > Thanks!
> > > > >
> > > > Sorry that I didn't make it clear in the first place.
> > > > It is the same cross boundary assertion.
> > >
> > > I see. After looking at the code it seems like sglist will coalesce
> > > contiguous physical ranges without taking page boundaries into
> > > account, which is not suitable for our purpose here. I guess I will
> > > either have to modify sglist, or switch to using bus_dma. The main
> > > problem with using bus_dma is that it will require bigger changes to
> > > netfront I think.
> >
> > I have a crappy patch to use bus_dma. It's not yet ready for upstream
> > but you might want to give it a try to see if it solves the cross page
> > boundary issues.
> >
> I think this version is better.

Thanks for all the testing.

> It fixed the mbuf cross boundary issue and allowed me to boot from one
> disk image successfully.

It's good to know it seems to handle splitting mbufs fragments at page
boundaries correctly.

> But seems like this patch is not stable enough yet and has its own
> issue -- memory is not properly released?

I know. I've been working on improving it this morning and I'm
attaching an updated version below.

Thanks, Roger.
---
diff --git a/sys/dev/xen/netfront/netfront.c b/sys/dev/xen/netfront/netfront.c
index 8dba5a8dc6d5..69528cc39b94 100644
--- a/sys/dev/xen/netfront/netfront.c
+++ b/sys/dev/xen/netfront/netfront.c
@@ -71,6 +71,8 @@ __FBSDID("$FreeBSD$");
 #include <xen/interface/io/netif.h>
 #include <xen/xenbus/xenbusvar.h>
 
+#include <machine/bus.h>
+
 #include "xenbus_if.h"
 
 /* Features supported by all backends.  TSO and LRO can be negotiated */
@@ -199,6 +201,17 @@ struct netfront_txq {
 	struct taskqueue 	*tq;
 	struct task       	defrtask;
 
+	bus_dma_segment_t	segs[MAX_TX_REQ_FRAGS];
+	struct mbuf_xennet {
+		struct m_tag 	tag;
+		bus_dma_tag_t	dma_tag;
+		bus_dmamap_t	dma_map;
+		struct netfront_txq *txq;
+		SLIST_ENTRY(mbuf_xennet) next;
+		u_int 		count;
+	}			xennet_tag[NET_TX_RING_SIZE + 1];
+	SLIST_HEAD(, mbuf_xennet) tags;
+
 	bool			full;
 };
 
@@ -221,6 +234,8 @@ struct netfront_info {
 
 	struct ifmedia		sc_media;
 
+	bus_dma_tag_t		dma_tag;
+
 	bool			xn_reset;
 };
 
@@ -301,6 +316,42 @@ xn_get_rx_ref(struct netfront_rxq *rxq, RING_IDX ri)
 	return (ref);
 }
 
+#define MTAG_COOKIE 1218492000
+#define MTAG_XENNET 0
+
+static void mbuf_grab(struct mbuf *m)
+{
+	struct mbuf_xennet *ref;
+
+	ref = (struct mbuf_xennet *)m_tag_locate(m, MTAG_COOKIE,
+	    MTAG_XENNET, NULL);
+	KASSERT(ref != NULL, ("Cannot find refcount"));
+	ref->count++;
+}
+
+static void mbuf_release(struct mbuf *m)
+{
+	struct mbuf_xennet *ref;
+
+	ref = (struct mbuf_xennet *)m_tag_locate(m, MTAG_COOKIE,
+	    MTAG_XENNET, NULL);
+	KASSERT(ref != NULL, ("Cannot find refcount"));
+	KASSERT(ref->count > 0, ("Invalid reference count"));
+
+	if (--ref->count == 0)
+		m_freem(m);
+}
+
+static void tag_free(struct m_tag *t)
+{
+	struct mbuf_xennet *ref = (struct mbuf_xennet *)t;
+
+	KASSERT(ref->count == 0, ("Free mbuf tag with pending refcnt"));
+	bus_dmamap_sync(ref->dma_tag, ref->dma_map, BUS_DMASYNC_POSTWRITE);
+	bus_dmamap_destroy(ref->dma_tag, ref->dma_map);
+	SLIST_INSERT_HEAD(&ref->txq->tags, ref, next);
+}
+
 #define IPRINTK(fmt, args...) \
     printf("[XEN] " fmt, ##args)
 #ifdef INVARIANTS
@@ -778,11 +829,18 @@ disconnect_txq(struct netfront_txq *txq)
 static void
 destroy_txq(struct netfront_txq *txq)
 {
+	unsigned int i;
 
 	free(txq->ring.sring, M_DEVBUF);
 	buf_ring_free(txq->br, M_DEVBUF);
 	taskqueue_drain_all(txq->tq);
 	taskqueue_free(txq->tq);
+
+	for (i = 0; i <= NET_TX_RING_SIZE; i++) {
+		bus_dmamap_destroy(txq->info->dma_tag,
+		    txq->xennet_tag[i].dma_map);
+		txq->xennet_tag[i].dma_map = NULL;
+	}
 }
 
 static void
@@ -822,10 +880,27 @@ setup_txqs(device_t dev, struct netfront_info *info,
 
 		mtx_init(&txq->lock, txq->name, "netfront transmit lock",
 		    MTX_DEF);
+		SLIST_INIT(&txq->tags);
 
 		for (i = 0; i <= NET_TX_RING_SIZE; i++) {
 			txq->mbufs[i] = (void *) ((u_long) i+1);
 			txq->grant_ref[i] = GRANT_REF_INVALID;
+			txq->xennet_tag[i].txq = txq;
+			txq->xennet_tag[i].dma_tag = info->dma_tag;
+			error = bus_dmamap_create(info->dma_tag, 0,
+			    &txq->xennet_tag[i].dma_map);
+			if (error != 0) {
+				device_printf(dev,
+				    "failed to allocate dma map\n");
+				goto fail;
+			}
+			m_tag_setup(&txq->xennet_tag[i].tag,
+			    MTAG_COOKIE, MTAG_XENNET,
+			    sizeof(txq->xennet_tag[i]) -
+			    sizeof(txq->xennet_tag[i].tag));
+			txq->xennet_tag[i].tag.m_tag_free = &tag_free;
+			SLIST_INSERT_HEAD(&txq->tags, &txq->xennet_tag[i],
+			    next);
 		}
 		txq->mbufs[NET_TX_RING_SIZE] = (void *)0;
 
@@ -1041,7 +1116,7 @@ xn_release_tx_bufs(struct netfront_txq *txq)
 		if (txq->mbufs_cnt < 0) {
 			panic("%s: tx_chain_cnt must be >= 0", __func__);
 		}
-		m_free(m);
+		mbuf_release(m);
 	}
 }
 
@@ -1311,7 +1386,7 @@ xn_txeof(struct netfront_txq *txq)
 			txq->mbufs[id] = NULL;
 			add_id_to_freelist(txq->mbufs, id);
 			txq->mbufs_cnt--;
-			m_free(m);
+			mbuf_release(m);
 			/* Only mark the txq active if we've freed up at least one slot to try */
 			ifp->if_drv_flags &= ~IFF_DRV_OACTIVE;
 		}
@@ -1530,46 +1605,51 @@ xn_count_frags(struct mbuf *m)
 static int
 xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 {
-	struct mbuf *m;
 	struct netfront_info *np = txq->info;
 	struct ifnet *ifp = np->xn_ifp;
-	u_int nfrags;
-	int otherend_id;
+	int otherend_id, error, nfrags;
+	bus_dma_segment_t *segs;
+	struct mbuf_xennet *tag;
+	bus_dmamap_t map;
+	unsigned int i;
 
-	/**
-	 * Defragment the mbuf if necessary.
-	 */
-	nfrags = xn_count_frags(m_head);
+	segs = txq->segs;
+	KASSERT(!SLIST_EMPTY(&txq->tags), ("no tags available"));
+	tag = SLIST_FIRST(&txq->tags);
+	SLIST_REMOVE_HEAD(&txq->tags, next);
+	KASSERT(tag->count == 0, ("tag already in-use"));
+	map = tag->dma_map;
+	error = bus_dmamap_load_mbuf_sg(np->dma_tag, map, m_head, segs,
+	    &nfrags, 0);
+	if (error == EFBIG || nfrags > np->maxfrags) {
+		struct mbuf *m;
 
-	/*
-	 * Check to see whether this request is longer than netback
-	 * can handle, and try to defrag it.
-	 */
-	/**
-	 * It is a bit lame, but the netback driver in Linux can't
-	 * deal with nfrags > MAX_TX_REQ_FRAGS, which is a quirk of
-	 * the Linux network stack.
-	 */
-	if (nfrags > np->maxfrags) {
+		bus_dmamap_unload(np->dma_tag, map);
 		m = m_defrag(m_head, M_NOWAIT);
 		if (!m) {
 			/*
 			 * Defrag failed, so free the mbuf and
 			 * therefore drop the packet.
 			 */
+			SLIST_INSERT_HEAD(&txq->tags, tag, next);
 			m_freem(m_head);
 			return (EMSGSIZE);
 		}
 		m_head = m;
+		error = bus_dmamap_load_mbuf_sg(np->dma_tag, map, m_head, segs,
+		    &nfrags, 0);
+		if (error != 0 || nfrags > np->maxfrags) {
+			bus_dmamap_unload(np->dma_tag, map);
+			SLIST_INSERT_HEAD(&txq->tags, tag, next);
+			m_freem(m_head);
+			return (error ?: EFBIG);
+		}
+	} else if (error != 0) {
+		SLIST_INSERT_HEAD(&txq->tags, tag, next);
+		m_freem(m_head);
+		return (error);
 	}
 
-	/* Determine how many fragments now exist */
-	nfrags = xn_count_frags(m_head);
-
-	/*
-	 * Check to see whether the defragmented packet has too many
-	 * segments for the Linux netback driver.
-	 */
 	/**
 	 * The FreeBSD TCP stack, with TSO enabled, can produce a chain
 	 * of mbufs longer than Linux can handle.  Make sure we don't
@@ -1583,6 +1663,8 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 		       "won't be able to handle it, dropping\n",
 		       __func__, nfrags, MAX_TX_REQ_FRAGS);
 #endif
+		SLIST_INSERT_HEAD(&txq->tags, tag, next);
+		bus_dmamap_unload(np->dma_tag, map);
 		m_freem(m_head);
 		return (EMSGSIZE);
 	}
@@ -1604,9 +1686,9 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 	 * the fragment pointers. Stop when we run out
 	 * of fragments or hit the end of the mbuf chain.
 	 */
-	m = m_head;
 	otherend_id = xenbus_get_otherend_id(np->xbdev);
-	for (m = m_head; m; m = m->m_next) {
+	m_tag_prepend(m_head, &tag->tag);
+	for (i = 0; i < nfrags; i++) {
 		netif_tx_request_t *tx;
 		uintptr_t id;
 		grant_ref_t ref;
@@ -1621,17 +1703,20 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 		if (txq->mbufs_cnt > NET_TX_RING_SIZE)
 			panic("%s: tx_chain_cnt must be <= NET_TX_RING_SIZE\n",
 			    __func__);
-		txq->mbufs[id] = m;
+		mbuf_grab(m_head);
+		txq->mbufs[id] = m_head;
 		tx->id = id;
 		ref = gnttab_claim_grant_reference(&txq->gref_head);
 		KASSERT((short)ref >= 0, ("Negative ref"));
-		mfn = virt_to_mfn(mtod(m, vm_offset_t));
+		mfn = atop(segs[i].ds_addr);
 		gnttab_grant_foreign_access_ref(ref, otherend_id,
 		    mfn, GNTMAP_readonly);
 		tx->gref = txq->grant_ref[id] = ref;
-		tx->offset = mtod(m, vm_offset_t) & (PAGE_SIZE - 1);
+		tx->offset = segs[i].ds_addr & PAGE_MASK;
+		KASSERT(tx->offset + segs[i].ds_len <= PAGE_SIZE,
+		    ("mbuf segment crosses a page boundary"));
 		tx->flags = 0;
-		if (m == m_head) {
+		if (i == 0) {
 			/*
 			 * The first fragment has the entire packet
 			 * size, subsequent fragments have just the
@@ -1640,7 +1725,7 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 			 * subtracting the sizes of the other
 			 * fragments.
 			 */
-			tx->size = m->m_pkthdr.len;
+			tx->size = m_head->m_pkthdr.len;
 
 			/*
 			 * The first fragment contains the checksum flags
@@ -1654,12 +1739,12 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 			 * so we have to test for CSUM_TSO
 			 * explicitly.
 			 */
-			if (m->m_pkthdr.csum_flags
+			if (m_head->m_pkthdr.csum_flags
 			    & (CSUM_DELAY_DATA | CSUM_TSO)) {
 				tx->flags |= (NETTXF_csum_blank
 				    | NETTXF_data_validated);
 			}
-			if (m->m_pkthdr.csum_flags & CSUM_TSO) {
+			if (m_head->m_pkthdr.csum_flags & CSUM_TSO) {
 				struct netif_extra_info *gso =
 					(struct netif_extra_info *)
 					RING_GET_REQUEST(&txq->ring,
@@ -1667,7 +1752,7 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 
 				tx->flags |= NETTXF_extra_info;
 
-				gso->u.gso.size = m->m_pkthdr.tso_segsz;
+				gso->u.gso.size = m_head->m_pkthdr.tso_segsz;
 				gso->u.gso.type =
 					XEN_NETIF_GSO_TYPE_TCPV4;
 				gso->u.gso.pad = 0;
@@ -1677,13 +1762,14 @@ xn_assemble_tx_request(struct netfront_txq *txq, struct mbuf *m_head)
 				gso->flags = 0;
 			}
 		} else {
-			tx->size = m->m_len;
+			tx->size = segs[i].ds_len;
 		}
-		if (m->m_next)
+		if (i != nfrags - 1)
 			tx->flags |= NETTXF_more_data;
 
 		txq->ring.req_prod_pvt++;
 	}
+	bus_dmamap_sync(np->dma_tag, map, BUS_DMASYNC_PREWRITE);
 	BPF_MTAP(ifp, m_head);
 
 	if_inc_counter(ifp, IFCOUNTER_OPACKETS, 1);
@@ -2244,7 +2330,20 @@ create_netdev(device_t dev)
     	ether_ifattach(ifp, np->mac);
 	netfront_carrier_off(np);
 
-	return (0);
+	err = bus_dma_tag_create(
+	    bus_get_dma_tag(dev),		/* parent */
+	    1, PAGE_SIZE,			/* algnmnt, boundary */
+	    BUS_SPACE_MAXADDR,			/* lowaddr */
+	    BUS_SPACE_MAXADDR,			/* highaddr */
+	    NULL, NULL,				/* filter, filterarg */
+	    PAGE_SIZE * MAX_TX_REQ_FRAGS,	/* max request size */
+	    MAX_TX_REQ_FRAGS,			/* max segments */
+	    PAGE_SIZE,				/* maxsegsize */
+	    BUS_DMA_ALLOCNOW,			/* flags */
+	    NULL, NULL,				/* lockfunc, lockarg */
+	    &np->dma_tag);
+
+	return (err);
 
 error:
 	KASSERT(err != 0, ("Error path with no error code specified"));
@@ -2277,6 +2376,7 @@ netif_free(struct netfront_info *np)
 	if_free(np->xn_ifp);
 	np->xn_ifp = NULL;
 	ifmedia_removeall(&np->sc_media);
+	bus_dma_tag_destroy(np->dma_tag);
 }
 
 static void

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

On Fri, Dec 31, 2021 at 2:52 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Thu, Dec 30, 2021 at 11:12:57PM +0800, G.R. wrote:
> > On Thu, Dec 30, 2021 at 3:07 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
> > >
> > > On Wed, Dec 29, 2021 at 11:27:50AM +0100, Roger Pau Monné wrote:
> > > > On Wed, Dec 29, 2021 at 05:13:00PM +0800, G.R. wrote:
> > > > > >
> > > > > > I think this is hitting a KASSERT, could you paste the text printed as
> > > > > > part of the panic (not just he backtrace)?
> > > > > >
> > > > > > Sorry this is taking a bit of time to solve.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > Sorry that I didn't make it clear in the first place.
> > > > > It is the same cross boundary assertion.
> > > >
> > > > I see. After looking at the code it seems like sglist will coalesce
> > > > contiguous physical ranges without taking page boundaries into
> > > > account, which is not suitable for our purpose here. I guess I will
> > > > either have to modify sglist, or switch to using bus_dma. The main
> > > > problem with using bus_dma is that it will require bigger changes to
> > > > netfront I think.
> > >
> > > I have a crappy patch to use bus_dma. It's not yet ready for upstream
> > > but you might want to give it a try to see if it solves the cross page
> > > boundary issues.
> > >
> > I think this version is better.
>
> Thanks for all the testing.
>
> > It fixed the mbuf cross boundary issue and allowed me to boot from one
> > disk image successfully.
>
> It's good to know it seems to handle splitting mbufs fragments at page
> boundaries correctly.
>
> > But seems like this patch is not stable enough yet and has its own
> > issue -- memory is not properly released?
>
> I know. I've been working on improving it this morning and I'm
> attaching an updated version below.
>
Good news.
With this  new patch, the NAS domU can serve iSCSI disk without OOM
panic, at least for a little while.
I'm going to keep it up and running for a while to see if it's stable over time.

BTW, an irrelevant question:
What's the current status of HVM domU on top of storage driver domain?
About 7 years ago, one user on the list was able to get this setup up
and running with your help (patch).[1]
When I attempted to reproduce a similar setup two years later, I
discovered that the patch was not submitted.
And even with that patch the setup cannot be reproduced successfully.
We spent some time debugging on the problem together[2], but didn't
bottom out the root cause at that time.
In case it's still broken and you still have the interest and time, I
can launch a separate thread on this topic and provide required
testing environment.

[1] https://lists.xenproject.org/archives/html/xen-users/2014-08/msg00003.html
[2] https://xen-users.narkive.com/9ihP0QG4/hvm-domu-on-storage-driver-domain

Thanks,
G.R.

> Thanks, Roger.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Roger Pau Monné]

On Fri, Dec 31, 2021 at 10:47:57PM +0800, G.R. wrote:
> On Fri, Dec 31, 2021 at 2:52 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Thu, Dec 30, 2021 at 11:12:57PM +0800, G.R. wrote:
> > > On Thu, Dec 30, 2021 at 3:07 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
> > > >
> > > > On Wed, Dec 29, 2021 at 11:27:50AM +0100, Roger Pau Monné wrote:
> > > > > On Wed, Dec 29, 2021 at 05:13:00PM +0800, G.R. wrote:
> > > > > > >
> > > > > > > I think this is hitting a KASSERT, could you paste the text printed as
> > > > > > > part of the panic (not just he backtrace)?
> > > > > > >
> > > > > > > Sorry this is taking a bit of time to solve.
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > Sorry that I didn't make it clear in the first place.
> > > > > > It is the same cross boundary assertion.
> > > > >
> > > > > I see. After looking at the code it seems like sglist will coalesce
> > > > > contiguous physical ranges without taking page boundaries into
> > > > > account, which is not suitable for our purpose here. I guess I will
> > > > > either have to modify sglist, or switch to using bus_dma. The main
> > > > > problem with using bus_dma is that it will require bigger changes to
> > > > > netfront I think.
> > > >
> > > > I have a crappy patch to use bus_dma. It's not yet ready for upstream
> > > > but you might want to give it a try to see if it solves the cross page
> > > > boundary issues.
> > > >
> > > I think this version is better.
> >
> > Thanks for all the testing.
> >
> > > It fixed the mbuf cross boundary issue and allowed me to boot from one
> > > disk image successfully.
> >
> > It's good to know it seems to handle splitting mbufs fragments at page
> > boundaries correctly.
> >
> > > But seems like this patch is not stable enough yet and has its own
> > > issue -- memory is not properly released?
> >
> > I know. I've been working on improving it this morning and I'm
> > attaching an updated version below.
> >
> Good news.
> With this  new patch, the NAS domU can serve iSCSI disk without OOM
> panic, at least for a little while.
> I'm going to keep it up and running for a while to see if it's stable over time.

Thanks again for all the testing. Do you see any difference
performance wise?

> BTW, an irrelevant question:
> What's the current status of HVM domU on top of storage driver domain?
> About 7 years ago, one user on the list was able to get this setup up
> and running with your help (patch).[1]
> When I attempted to reproduce a similar setup two years later, I
> discovered that the patch was not submitted.
> And even with that patch the setup cannot be reproduced successfully.
> We spent some time debugging on the problem together[2], but didn't
> bottom out the root cause at that time.
> In case it's still broken and you still have the interest and time, I
> can launch a separate thread on this topic and provide required
> testing environment.

Yes, better as a new thread please.

FWIW, I haven't looked at this since a long time, but I recall some
fixes in order to be able to use driver domains with HVM guests, which
require attaching the disk to dom0 in order for the device model
(QEMU) to access it.

I would give it a try without using stubdomains and see what you get.
You will need to run `xl devd` inside of the driver domain, so you
will need to install xen-tools on the domU. There's an init script to
launch `xl devd` at boot, it's called 'xendriverdomain'.

Thanks, Roger.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

> > > > But seems like this patch is not stable enough yet and has its own
> > > > issue -- memory is not properly released?
> > >
> > > I know. I've been working on improving it this morning and I'm
> > > attaching an updated version below.
> > >
> > Good news.
> > With this  new patch, the NAS domU can serve iSCSI disk without OOM
> > panic, at least for a little while.
> > I'm going to keep it up and running for a while to see if it's stable over time.
>
> Thanks again for all the testing. Do you see any difference
> performance wise?
I'm still on a *debug* kernel build to capture any potential panic --
none so far -- no performance testing yet.
Since I'm a home user with a relatively lightweight workload, so far I
didn't observe any difference in daily usage.

I did some quick iperf3 testing just now.
1. between nas domU <=> Linux dom0 running on an old i7-3770 based box.
The peak is roughly 12 Gbits/s when domU is the server.
But I do see regression down to ~8.5 Gbits/s when I repeat the test in
a short burst.
The regression can recover when I leave the system idle for a while.

When dom0 is the iperf3 server, the transfer rate is much lower, down
all the way to 1.x Gbits/s.
Sometimes, I can see the following kernel log repeats during the
testing, likely contributing to the slowdown.
             interrupt storm detected on "irq2328:"; throttling interrupt source
Another thing that looks alarming is the retransmission is high:
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   212 MBytes  1.78 Gbits/sec  110    231 KBytes
[  5]   1.00-2.00   sec   230 MBytes  1.92 Gbits/sec    1    439 KBytes
[  5]   2.00-3.00   sec   228 MBytes  1.92 Gbits/sec    3    335 KBytes
[  5]   3.00-4.00   sec   204 MBytes  1.71 Gbits/sec    1    486 KBytes
[  5]   4.00-5.00   sec   201 MBytes  1.69 Gbits/sec  812    258 KBytes
[  5]   5.00-6.00   sec   179 MBytes  1.51 Gbits/sec    1    372 KBytes
[  5]   6.00-7.00   sec  50.5 MBytes   423 Mbits/sec    2    154 KBytes
[  5]   7.00-8.00   sec   194 MBytes  1.63 Gbits/sec  339    172 KBytes
[  5]   8.00-9.00   sec   156 MBytes  1.30 Gbits/sec  854    215 KBytes
[  5]   9.00-10.00  sec   143 MBytes  1.20 Gbits/sec  997   93.8 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.76 GBytes  1.51 Gbits/sec  3120             sender
[  5]   0.00-10.45  sec  1.76 GBytes  1.44 Gbits/sec                  receiver

2. between a remote box <=> nas domU, through a 1Gbps ethernet cable.
Roughly saturate the link when domU is the server, without obvious perf drop
When domU running as a client, the achieved BW is ~30Mbps lower than the peak.
Retransmission sometimes also shows up in this scenario, more
seriously when domU is the client.

I cannot test with the stock kernel nor with your patch in release
mode immediately.
But according to the observed imbalance between inbounding and
outgoing path, non-trivial penalty applies I guess?

> > BTW, an irrelevant question:
> > What's the current status of HVM domU on top of storage driver domain?
> > About 7 years ago, one user on the list was able to get this setup up
> > and running with your help (patch).[1]
> > When I attempted to reproduce a similar setup two years later, I
> > discovered that the patch was not submitted.
> > And even with that patch the setup cannot be reproduced successfully.
> > We spent some time debugging on the problem together[2], but didn't
> > bottom out the root cause at that time.
> > In case it's still broken and you still have the interest and time, I
> > can launch a separate thread on this topic and provide required
> > testing environment.
>
> Yes, better as a new thread please.
>
> FWIW, I haven't looked at this since a long time, but I recall some
> fixes in order to be able to use driver domains with HVM guests, which
> require attaching the disk to dom0 in order for the device model
> (QEMU) to access it.
>
> I would give it a try without using stubdomains and see what you get.
> You will need to run `xl devd` inside of the driver domain, so you
> will need to install xen-tools on the domU. There's an init script to
> launch `xl devd` at boot, it's called 'xendriverdomain'.
Looks like I'm unlucky once again. Let's follow up in a separate thread.

> Thanks, Roger.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Roger Pau Monné]

On Wed, Jan 05, 2022 at 12:05:39AM +0800, G.R. wrote:
> > > > > But seems like this patch is not stable enough yet and has its own
> > > > > issue -- memory is not properly released?
> > > >
> > > > I know. I've been working on improving it this morning and I'm
> > > > attaching an updated version below.
> > > >
> > > Good news.
> > > With this  new patch, the NAS domU can serve iSCSI disk without OOM
> > > panic, at least for a little while.
> > > I'm going to keep it up and running for a while to see if it's stable over time.
> >
> > Thanks again for all the testing. Do you see any difference
> > performance wise?
> I'm still on a *debug* kernel build to capture any potential panic --
> none so far -- no performance testing yet.
> Since I'm a home user with a relatively lightweight workload, so far I
> didn't observe any difference in daily usage.
> 
> I did some quick iperf3 testing just now.

Thanks for doing this.

> 1. between nas domU <=> Linux dom0 running on an old i7-3770 based box.
> The peak is roughly 12 Gbits/s when domU is the server.
> But I do see regression down to ~8.5 Gbits/s when I repeat the test in
> a short burst.
> The regression can recover when I leave the system idle for a while.
> 
> When dom0 is the iperf3 server, the transfer rate is much lower, down
> all the way to 1.x Gbits/s.
> Sometimes, I can see the following kernel log repeats during the
> testing, likely contributing to the slowdown.
>              interrupt storm detected on "irq2328:"; throttling interrupt source

I assume the message is in the domU, not the dom0?

> Another thing that looks alarming is the retransmission is high:
> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> [  5]   0.00-1.00   sec   212 MBytes  1.78 Gbits/sec  110    231 KBytes
> [  5]   1.00-2.00   sec   230 MBytes  1.92 Gbits/sec    1    439 KBytes
> [  5]   2.00-3.00   sec   228 MBytes  1.92 Gbits/sec    3    335 KBytes
> [  5]   3.00-4.00   sec   204 MBytes  1.71 Gbits/sec    1    486 KBytes
> [  5]   4.00-5.00   sec   201 MBytes  1.69 Gbits/sec  812    258 KBytes
> [  5]   5.00-6.00   sec   179 MBytes  1.51 Gbits/sec    1    372 KBytes
> [  5]   6.00-7.00   sec  50.5 MBytes   423 Mbits/sec    2    154 KBytes
> [  5]   7.00-8.00   sec   194 MBytes  1.63 Gbits/sec  339    172 KBytes
> [  5]   8.00-9.00   sec   156 MBytes  1.30 Gbits/sec  854    215 KBytes
> [  5]   9.00-10.00  sec   143 MBytes  1.20 Gbits/sec  997   93.8 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate         Retr
> [  5]   0.00-10.00  sec  1.76 GBytes  1.51 Gbits/sec  3120             sender
> [  5]   0.00-10.45  sec  1.76 GBytes  1.44 Gbits/sec                  receiver

Do you see the same when running the same tests on a debug kernel
without my patch applied? (ie: a kernel build yourself from the same
baseline but just without my patch applied)

I'm mostly interested in knowing whether the patch itself causes any
regressions from the current state (which might not be great already).

> 
> 2. between a remote box <=> nas domU, through a 1Gbps ethernet cable.
> Roughly saturate the link when domU is the server, without obvious perf drop
> When domU running as a client, the achieved BW is ~30Mbps lower than the peak.
> Retransmission sometimes also shows up in this scenario, more
> seriously when domU is the client.
> 
> I cannot test with the stock kernel nor with your patch in release
> mode immediately.
> 
> But according to the observed imbalance between inbounding and
> outgoing path, non-trivial penalty applies I guess?

We should get a baseline using the same sources without my path
applied.

Thanks, Roger.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

On Wed, Jan 5, 2022 at 10:33 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Wed, Jan 05, 2022 at 12:05:39AM +0800, G.R. wrote:
> > > > > > But seems like this patch is not stable enough yet and has its own
> > > > > > issue -- memory is not properly released?
> > > > >
> > > > > I know. I've been working on improving it this morning and I'm
> > > > > attaching an updated version below.
> > > > >
> > > > Good news.
> > > > With this  new patch, the NAS domU can serve iSCSI disk without OOM
> > > > panic, at least for a little while.
> > > > I'm going to keep it up and running for a while to see if it's stable over time.
> > >
> > > Thanks again for all the testing. Do you see any difference
> > > performance wise?
> > I'm still on a *debug* kernel build to capture any potential panic --
> > none so far -- no performance testing yet.
> > Since I'm a home user with a relatively lightweight workload, so far I
> > didn't observe any difference in daily usage.
> >
> > I did some quick iperf3 testing just now.
>
> Thanks for doing this.
>
> > 1. between nas domU <=> Linux dom0 running on an old i7-3770 based box.
> > The peak is roughly 12 Gbits/s when domU is the server.
> > But I do see regression down to ~8.5 Gbits/s when I repeat the test in
> > a short burst.
> > The regression can recover when I leave the system idle for a while.
> >
> > When dom0 is the iperf3 server, the transfer rate is much lower, down
> > all the way to 1.x Gbits/s.
> > Sometimes, I can see the following kernel log repeats during the
> > testing, likely contributing to the slowdown.
> >              interrupt storm detected on "irq2328:"; throttling interrupt source
>
> I assume the message is in the domU, not the dom0?
Yes, in the TrueNAS domU.
BTW, I rebooted back to the stock kernel and the message is no longer observed.

With the stock kernel, the transfer rate from dom0 to nas domU can be
as high as 30Gbps.
The variation is still observed, sometimes down to ~19Gbps. There is
no retransmission in this direction.

For the reverse direction, the observed low transfer rate still exists.
It's still within the range of 1.x Gbps, but should still be better
than the previous test.
The huge number of re-transmission is still observed.
The same behavior can be observed on a stock FreeBSD 12.2 image, so
this is not specific to TrueNAS.

According to the packet capture, the re-transmission appears to be
caused by packet reorder.
Here is one example incident:
1. dom0 sees a sequence jump in the incoming stream and begins to send out SACKs
2. When SACK shows up at domU, it begins to re-transmit lost frames
   (the re-transmit looks weird since it show up as a mixed stream of
1448 bytes and 12 bytes packets, instead of always 1448 bytes)
3. Suddenly the packets that are believed to have lost show up, dom0
accept them as if they are re-transmission
4. The actual re-transmission finally shows up in dom0...
Should we expect packet reorder on a direct virtual link? Sounds fishy to me.
Any chance we can get this re-transmission fixed?

So looks like at least the imbalance between two directions are not
related to your patch.
Likely the debug build is a bigger contributor to the perf difference
in both directions.

I also tried your patch on a release build, and didn't observe any
major difference in iperf3 numbers.
Roughly match the 30Gbps and 1.xGbps number on the stock release kernel.

>
> > Another thing that looks alarming is the retransmission is high:
> > [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> > [  5]   0.00-1.00   sec   212 MBytes  1.78 Gbits/sec  110    231 KBytes
> > [  5]   1.00-2.00   sec   230 MBytes  1.92 Gbits/sec    1    439 KBytes
> > [  5]   2.00-3.00   sec   228 MBytes  1.92 Gbits/sec    3    335 KBytes
> > [  5]   3.00-4.00   sec   204 MBytes  1.71 Gbits/sec    1    486 KBytes
> > [  5]   4.00-5.00   sec   201 MBytes  1.69 Gbits/sec  812    258 KBytes
> > [  5]   5.00-6.00   sec   179 MBytes  1.51 Gbits/sec    1    372 KBytes
> > [  5]   6.00-7.00   sec  50.5 MBytes   423 Mbits/sec    2    154 KBytes
> > [  5]   7.00-8.00   sec   194 MBytes  1.63 Gbits/sec  339    172 KBytes
> > [  5]   8.00-9.00   sec   156 MBytes  1.30 Gbits/sec  854    215 KBytes
> > [  5]   9.00-10.00  sec   143 MBytes  1.20 Gbits/sec  997   93.8 KBytes
> > - - - - - - - - - - - - - - - - - - - - - - - - -
> > [ ID] Interval           Transfer     Bitrate         Retr
> > [  5]   0.00-10.00  sec  1.76 GBytes  1.51 Gbits/sec  3120             sender
> > [  5]   0.00-10.45  sec  1.76 GBytes  1.44 Gbits/sec                  receiver
>
> Do you see the same when running the same tests on a debug kernel
> without my patch applied? (ie: a kernel build yourself from the same
> baseline but just without my patch applied)
>
> I'm mostly interested in knowing whether the patch itself causes any
> regressions from the current state (which might not be great already).
>
> >
> > 2. between a remote box <=> nas domU, through a 1Gbps ethernet cable.
> > Roughly saturate the link when domU is the server, without obvious perf drop
> > When domU running as a client, the achieved BW is ~30Mbps lower than the peak.
> > Retransmission sometimes also shows up in this scenario, more
> > seriously when domU is the client.
> >
> > I cannot test with the stock kernel nor with your patch in release
> > mode immediately.
> >
> > But according to the observed imbalance between inbounding and
> > outgoing path, non-trivial penalty applies I guess?
>
> We should get a baseline using the same sources without my path
> applied.
>
> Thanks, Roger.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Roger Pau Monné]

On Sat, Jan 08, 2022 at 01:14:26AM +0800, G.R. wrote:
> On Wed, Jan 5, 2022 at 10:33 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
> >
> > On Wed, Jan 05, 2022 at 12:05:39AM +0800, G.R. wrote:
> > > > > > > But seems like this patch is not stable enough yet and has its own
> > > > > > > issue -- memory is not properly released?
> > > > > >
> > > > > > I know. I've been working on improving it this morning and I'm
> > > > > > attaching an updated version below.
> > > > > >
> > > > > Good news.
> > > > > With this  new patch, the NAS domU can serve iSCSI disk without OOM
> > > > > panic, at least for a little while.
> > > > > I'm going to keep it up and running for a while to see if it's stable over time.
> > > >
> > > > Thanks again for all the testing. Do you see any difference
> > > > performance wise?
> > > I'm still on a *debug* kernel build to capture any potential panic --
> > > none so far -- no performance testing yet.
> > > Since I'm a home user with a relatively lightweight workload, so far I
> > > didn't observe any difference in daily usage.
> > >
> > > I did some quick iperf3 testing just now.
> >
> > Thanks for doing this.
> >
> > > 1. between nas domU <=> Linux dom0 running on an old i7-3770 based box.
> > > The peak is roughly 12 Gbits/s when domU is the server.
> > > But I do see regression down to ~8.5 Gbits/s when I repeat the test in
> > > a short burst.
> > > The regression can recover when I leave the system idle for a while.
> > >
> > > When dom0 is the iperf3 server, the transfer rate is much lower, down
> > > all the way to 1.x Gbits/s.
> > > Sometimes, I can see the following kernel log repeats during the
> > > testing, likely contributing to the slowdown.
> > >              interrupt storm detected on "irq2328:"; throttling interrupt source
> >
> > I assume the message is in the domU, not the dom0?
> Yes, in the TrueNAS domU.
> BTW, I rebooted back to the stock kernel and the message is no longer observed.
> 
> With the stock kernel, the transfer rate from dom0 to nas domU can be
> as high as 30Gbps.
> The variation is still observed, sometimes down to ~19Gbps. There is
> no retransmission in this direction.
> 
> For the reverse direction, the observed low transfer rate still exists.
> It's still within the range of 1.x Gbps, but should still be better
> than the previous test.
> The huge number of re-transmission is still observed.
> The same behavior can be observed on a stock FreeBSD 12.2 image, so
> this is not specific to TrueNAS.

So that's domU sending the data, and dom0 receiving it.

> 
> According to the packet capture, the re-transmission appears to be
> caused by packet reorder.
> Here is one example incident:
> 1. dom0 sees a sequence jump in the incoming stream and begins to send out SACKs
> 2. When SACK shows up at domU, it begins to re-transmit lost frames
>    (the re-transmit looks weird since it show up as a mixed stream of
> 1448 bytes and 12 bytes packets, instead of always 1448 bytes)
> 3. Suddenly the packets that are believed to have lost show up, dom0
> accept them as if they are re-transmission

Hm, so there seems to be some kind of issue with ordering I would say.

> 4. The actual re-transmission finally shows up in dom0...
> Should we expect packet reorder on a direct virtual link? Sounds fishy to me.
> Any chance we can get this re-transmission fixed?

Does this still happen with all the extra features disabled? (-rxcsum
-txcsum -lro -tso)

> So looks like at least the imbalance between two directions are not
> related to your patch.
> Likely the debug build is a bigger contributor to the perf difference
> in both directions.
> 
> I also tried your patch on a release build, and didn't observe any
> major difference in iperf3 numbers.
> Roughly match the 30Gbps and 1.xGbps number on the stock release kernel.

Thanks a lot, will try to get this upstream then.

Roger.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

On Mon, Jan 10, 2022 at 10:54 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Sat, Jan 08, 2022 at 01:14:26AM +0800, G.R. wrote:
> > On Wed, Jan 5, 2022 at 10:33 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
> > >
> > > On Wed, Jan 05, 2022 at 12:05:39AM +0800, G.R. wrote:
> > > > > > > > But seems like this patch is not stable enough yet and has its own
> > > > > > > > issue -- memory is not properly released?
> > > > > > >
> > > > > > > I know. I've been working on improving it this morning and I'm
> > > > > > > attaching an updated version below.
> > > > > > >
> > > > > > Good news.
> > > > > > With this  new patch, the NAS domU can serve iSCSI disk without OOM
> > > > > > panic, at least for a little while.
> > > > > > I'm going to keep it up and running for a while to see if it's stable over time.
> > > > >
> > > > > Thanks again for all the testing. Do you see any difference
> > > > > performance wise?
> > > > I'm still on a *debug* kernel build to capture any potential panic --
> > > > none so far -- no performance testing yet.
> > > > Since I'm a home user with a relatively lightweight workload, so far I
> > > > didn't observe any difference in daily usage.
> > > >
> > > > I did some quick iperf3 testing just now.
> > >
> > > Thanks for doing this.
> > >
> > > > 1. between nas domU <=> Linux dom0 running on an old i7-3770 based box.
> > > > The peak is roughly 12 Gbits/s when domU is the server.
> > > > But I do see regression down to ~8.5 Gbits/s when I repeat the test in
> > > > a short burst.
> > > > The regression can recover when I leave the system idle for a while.
> > > >
> > > > When dom0 is the iperf3 server, the transfer rate is much lower, down
> > > > all the way to 1.x Gbits/s.
> > > > Sometimes, I can see the following kernel log repeats during the
> > > > testing, likely contributing to the slowdown.
> > > >              interrupt storm detected on "irq2328:"; throttling interrupt source
> > >
> > > I assume the message is in the domU, not the dom0?
> > Yes, in the TrueNAS domU.
> > BTW, I rebooted back to the stock kernel and the message is no longer observed.
> >
> > With the stock kernel, the transfer rate from dom0 to nas domU can be
> > as high as 30Gbps.
> > The variation is still observed, sometimes down to ~19Gbps. There is
> > no retransmission in this direction.
> >
> > For the reverse direction, the observed low transfer rate still exists.
> > It's still within the range of 1.x Gbps, but should still be better
> > than the previous test.
> > The huge number of re-transmission is still observed.
> > The same behavior can be observed on a stock FreeBSD 12.2 image, so
> > this is not specific to TrueNAS.
>
> So that's domU sending the data, and dom0 receiving it.
Correct.
>
> >
> > According to the packet capture, the re-transmission appears to be
> > caused by packet reorder.
> > Here is one example incident:
> > 1. dom0 sees a sequence jump in the incoming stream and begins to send out SACKs
> > 2. When SACK shows up at domU, it begins to re-transmit lost frames
> >    (the re-transmit looks weird since it show up as a mixed stream of
> > 1448 bytes and 12 bytes packets, instead of always 1448 bytes)
> > 3. Suddenly the packets that are believed to have lost show up, dom0
> > accept them as if they are re-transmission
>
> Hm, so there seems to be some kind of issue with ordering I would say.
Agree.

>
> > 4. The actual re-transmission finally shows up in dom0...
> > Should we expect packet reorder on a direct virtual link? Sounds fishy to me.
> > Any chance we can get this re-transmission fixed?
>
> Does this still happen with all the extra features disabled? (-rxcsum
> -txcsum -lro -tso)
No obvious impact I would say.
After disabling all extra features:
xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 00:18:3c:51:6e:4c
    inet 192.168.1.9 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet manual
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
The iperf3 result:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.04 GBytes  1.75 Gbits/sec  12674             sender
[  5]   0.00-10.14  sec  2.04 GBytes  1.73 Gbits/sec                  receiver
BTW, those extra features have huge impact on the dom0 => domU direction.
It goes all the way down from ~30 / 18 Gbps to 3.5 / 1.8 Gbps
(variation range) without those.
But there is no retransmission at all in both configs for this direction.
I wonder why such a huge difference since the nic is purely virtual
without any HW acceleration?

Any further suggestions on this retransmission issue?

>
> > So looks like at least the imbalance between two directions are not
> > related to your patch.
> > Likely the debug build is a bigger contributor to the perf difference
> > in both directions.
> >
> > I also tried your patch on a release build, and didn't observe any
> > major difference in iperf3 numbers.
> > Roughly match the 30Gbps and 1.xGbps number on the stock release kernel.
>
> Thanks a lot, will try to get this upstream then.
>
> Roger.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

On Mon, Jan 10, 2022 at 10:54 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
> > So looks like at least the imbalance between two directions are not
> > related to your patch.
> > Likely the debug build is a bigger contributor to the perf difference
> > in both directions.
> >
> > I also tried your patch on a release build, and didn't observe any
> > major difference in iperf3 numbers.
> > Roughly match the 30Gbps and 1.xGbps number on the stock release kernel.
>
> Thanks a lot, will try to get this upstream then.
>
> Roger.

Hi Roger, any news for the upstream fix? I haven't heard any news since...
The reason I came back to this thread is that I totally forgot about
this issue and upgraded to FreeNAS 13 only to rediscover this issue
once again :-(

Any chance the patch can apply on FreeBSD 13.1-RELEASE-p1 kernel?

Thanks,
G.R.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Paul Leiber]

Am 30.10.2022 um 17:36 schrieb G.R.:
> On Mon, Jan 10, 2022 at 10:54 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
>>> So looks like at least the imbalance between two directions are not
>>> related to your patch.
>>> Likely the debug build is a bigger contributor to the perf difference
>>> in both directions.
>>>
>>> I also tried your patch on a release build, and didn't observe any
>>> major difference in iperf3 numbers.
>>> Roughly match the 30Gbps and 1.xGbps number on the stock release kernel.
>> Thanks a lot, will try to get this upstream then.
>>
>> Roger.
> Hi Roger, any news for the upstream fix? I haven't heard any news since...
> The reason I came back to this thread is that I totally forgot about
> this issue and upgraded to FreeNAS 13 only to rediscover this issue
> once again :-(
>
> Any chance the patch can apply on FreeBSD 13.1-RELEASE-p1 kernel?
>
> Thanks,
> G.R.
>

Hi,

I want to confirm that the patch in an official release would make quite some people very happy. E.g. among OPNsense users, there are some who
suffer from the network issue [1]. FWIW, I compiled a kernel including Roger's patch, and it seems to be working without trouble in my OPNsense DomU.

Best regards,

Paul

[1] https://forum.opnsense.org/index.php?topic=28708.15

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Roger Pau Monné]

On Thu, Nov 03, 2022 at 07:58:52AM +0100, Paul Leiber wrote:
> 
> 
> Am 30.10.2022 um 17:36 schrieb G.R.:
> > On Mon, Jan 10, 2022 at 10:54 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
> > > > So looks like at least the imbalance between two directions are not
> > > > related to your patch.
> > > > Likely the debug build is a bigger contributor to the perf difference
> > > > in both directions.
> > > > 
> > > > I also tried your patch on a release build, and didn't observe any
> > > > major difference in iperf3 numbers.
> > > > Roughly match the 30Gbps and 1.xGbps number on the stock release kernel.
> > > Thanks a lot, will try to get this upstream then.
> > > 
> > > Roger.
> > Hi Roger, any news for the upstream fix? I haven't heard any news since...
> > The reason I came back to this thread is that I totally forgot about
> > this issue and upgraded to FreeNAS 13 only to rediscover this issue
> > once again :-(
> > 
> > Any chance the patch can apply on FreeBSD 13.1-RELEASE-p1 kernel?
> > 
> > Thanks,
> > G.R.
> > 
> 
> Hi,
> 
> I want to confirm that the patch in an official release would make quite some people very happy. E.g. among OPNsense users, there are some who
> suffer from the network issue [1]. FWIW, I compiled a kernel including Roger's patch, and it seems to be working without trouble in my OPNsense DomU.

Hello to both,

Sorry, I completely dropped the ball on that patch, didn't even
remember I had it pending :(.

Will do a build test with it and commit later today, I don't think I
will get any feedback, and it seems to improve the situation for your
use-cases.

Thanks, Roger.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

On Thu, Nov 3, 2022 at 8:37 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
> > > > Roger.
> > > Hi Roger, any news for the upstream fix? I haven't heard any news since...
> > > The reason I came back to this thread is that I totally forgot about
> > > this issue and upgraded to FreeNAS 13 only to rediscover this issue
> > > once again :-(
> > >
> > > Any chance the patch can apply on FreeBSD 13.1-RELEASE-p1 kernel?
> > >
> > > Thanks,
> > > G.R.
> > >
> >
> > Hi,
> >
> > I want to confirm that the patch in an official release would make quite some people very happy. E.g. among OPNsense users, there are some who
> > suffer from the network issue [1]. FWIW, I compiled a kernel including Roger's patch, and it seems to be working without trouble in my OPNsense DomU.
>
> Hello to both,
>
> Sorry, I completely dropped the ball on that patch, didn't even
> remember I had it pending :(.
>
> Will do a build test with it and commit later today, I don't think I
> will get any feedback, and it seems to improve the situation for your
> use-cases.

Hi Roger,
Just another query of the latest status. It'll be great if you can
share a link to the upstream commit.
I'm thinking of asking for a back-port of your fix to the FreeNAS
community, assuming it will take a long time to roll out otherwise.

Thanks,
G.R.

>
> Thanks, Roger.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Niklas Hallqvist]

> On 14 Dec 2022, at 07:16, G.R. <firemeteor@users.sourceforge.net> wrote:
> 
> On Thu, Nov 3, 2022 at 8:37 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
>>>>> Roger.
>>>> Hi Roger, any news for the upstream fix? I haven't heard any news since...
>>>> The reason I came back to this thread is that I totally forgot about
>>>> this issue and upgraded to FreeNAS 13 only to rediscover this issue
>>>> once again :-(
>>>> 
>>>> Any chance the patch can apply on FreeBSD 13.1-RELEASE-p1 kernel?
>>>> 
>>>> Thanks,
>>>> G.R.
>>>> 
>>> 
>>> Hi,
>>> 
>>> I want to confirm that the patch in an official release would make quite some people very happy. E.g. among OPNsense users, there are some who
>>> suffer from the network issue [1]. FWIW, I compiled a kernel including Roger's patch, and it seems to be working without trouble in my OPNsense DomU.
>> 
>> Hello to both,
>> 
>> Sorry, I completely dropped the ball on that patch, didn't even
>> remember I had it pending :(.
>> 
>> Will do a build test with it and commit later today, I don't think I
>> will get any feedback, and it seems to improve the situation for your
>> use-cases.
> 
> Hi Roger,
> Just another query of the latest status. It'll be great if you can
> share a link to the upstream commit.
> I'm thinking of asking for a back-port of your fix to the FreeNAS
> community, assuming it will take a long time to roll out otherwise.
> 
> Thanks,
> G.R.
> 
>> 
>> Thanks, Roger.
> 
> 

Hi everyone!

So did anything ever happen on this?  I find myself in the same situation with TrueNAS core 13, and can’t see any signs of changes in the FreeBSD 13 branches.

G.R: Did you ever build a kernel for TrueNAS 13?  Care to share it?  I don’t have a build environment setup, so I thought I’d just try to ask :-)

/Niklas

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Roger Pau Monné]

On Tue, Jan 09, 2024 at 12:13:04PM +0100, Niklas Hallqvist wrote:
> > On 14 Dec 2022, at 07:16, G.R. <firemeteor@users.sourceforge.net> wrote:
> > 
> > On Thu, Nov 3, 2022 at 8:37 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
> >>>>> Roger.
> >>>> Hi Roger, any news for the upstream fix? I haven't heard any news since...
> >>>> The reason I came back to this thread is that I totally forgot about
> >>>> this issue and upgraded to FreeNAS 13 only to rediscover this issue
> >>>> once again :-(
> >>>> 
> >>>> Any chance the patch can apply on FreeBSD 13.1-RELEASE-p1 kernel?
> >>>> 
> >>>> Thanks,
> >>>> G.R.
> >>>> 
> >>> 
> >>> Hi,
> >>> 
> >>> I want to confirm that the patch in an official release would make quite some people very happy. E.g. among OPNsense users, there are some who
> >>> suffer from the network issue [1]. FWIW, I compiled a kernel including Roger's patch, and it seems to be working without trouble in my OPNsense DomU.
> >> 
> >> Hello to both,
> >> 
> >> Sorry, I completely dropped the ball on that patch, didn't even
> >> remember I had it pending :(.
> >> 
> >> Will do a build test with it and commit later today, I don't think I
> >> will get any feedback, and it seems to improve the situation for your
> >> use-cases.
> > 
> > Hi Roger,
> > Just another query of the latest status. It'll be great if you can
> > share a link to the upstream commit.
> > I'm thinking of asking for a back-port of your fix to the FreeNAS
> > community, assuming it will take a long time to roll out otherwise.
> > 
> > Thanks,
> > G.R.
> > 
> >> 
> >> Thanks, Roger.
> > 
> > 
> 
> Hi everyone!
> 
> So did anything ever happen on this?  I find myself in the same situation with TrueNAS core 13, and can’t see any signs of changes in the FreeBSD 13 branches.

Hello,

I don't think the change is suitable to backport, it's IMO too
intrusive and risky.  It was committed late 2022, and it's in 14.0:

commit dabb3db7a817f003af3f89c965ba369c67fc4910
Author: Roger Pau Monné <royger@FreeBSD.org>
Date:   Thu Nov 3 13:29:22 2022 +0100

    xen/netfront: deal with mbuf data crossing a page boundary

    There's been a report recently of mbufs with data that crosses a page
    boundary. It seems those mbufs are generated by the iSCSI target
    system:

    https://lists.xenproject.org/archives/html/xen-devel/2021-12/msg01581.html

    In order to handle those mbufs correctly on netfront use the bus_dma
    interface and explicitly request that segments must not cross a page
    boundary. No other requirements are necessary, so it's expected that
    bus_dma won't need to bounce the data and hence it shouldn't
    introduce a too big performance penalty.

    Using bus_dma requires some changes to netfront, mainly in order to
    accommodate for the fact that now ring slots no longer have a 1:1
    match with mbufs, as a single mbuf can use two ring slots if the data
    buffer crosses a page boundary. Store the first packet of the mbuf
    chain in every ring slot that's used, and use a mbuf tag in order to
    store the bus_dma related structures and a refcount to keep track of
    the pending slots before the mbuf chain can be freed.

    Reported by: G.R.
    Tested by: G.R.
    MFC: 1 week
    Differential revision: https://reviews.freebsd.org/D33876

TrueNAS/OOPNsense might consider picking it up themselves.

Thanks, Roger.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[G.R.]

On Tue, Jan 9, 2024 at 11:28 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Tue, Jan 09, 2024 at 12:13:04PM +0100, Niklas Hallqvist wrote:
> > > On 14 Dec 2022, at 07:16, G.R. <firemeteor@users.sourceforge.net> wrote:
...
> > > Hi Roger,
> > > Just another query of the latest status. It'll be great if you can
> > > share a link to the upstream commit.
> > > I'm thinking of asking for a back-port of your fix to the FreeNAS
> > > community, assuming it will take a long time to roll out otherwise.
> > >
> > > Thanks,
> > > G.R.
> > >
> > >>
> > >> Thanks, Roger.
> > >
> > >
> >
> > Hi everyone!
> >
> > So did anything ever happen on this?  I find myself in the same situation with TrueNAS core 13, and can’t see any signs of changes in the FreeBSD 13 branches.
>
> Hello,
>
> I don't think the change is suitable to backport, it's IMO too
> intrusive and risky.  It was committed late 2022, and it's in 14.0:
>
...
> TrueNAS/OOPNsense might consider picking it up themselves.
>
> Thanks, Roger.

Just FYI: I was able to locate that commit in 14.0 tree and
cherry-picked it into TrueNas 13.
I did it last November and the build has been running stably without
issue since.
The fix was fairly standalone and didn't cause any trouble during the
cherry-picking so it could be reasonable to raise a request in the
TrueNAS forum.

Thanks,
G.R.

Re: Possible bug? DOM-U network stopped working after fatal error reported in DOM0

[Roger Pau Monné]

On Sun, Dec 19, 2021 at 02:35:56AM +0800, G.R. wrote:
> Hi all,
> 
> I ran into the following error report in the DOM0 kernel after a recent upgrade:
> [  501.840816] vif vif-1-0 vif1.0: Cross page boundary, txp->offset:
> 2872, size: 1460
> [  501.840828] vif vif-1-0 vif1.0: fatal error; disabling device
> [  501.841076] xenbr0: port 2(vif1.0) entered disabled state
> Once this error happens, the DOM-U behind this vif is no-longer
> accessible. And recreating the same DOM-U does not fix the problem.
> Only a reboot on the physical host machine helps.
> 
> The problem showed up after a recent upgrade on the DOM-U OS from
> FreeNAS 11.3 to TrueNAS 12.0U7 and breaks the iSCSI service while
> leaving other services like NFS intact.
> The underlying OS for the NAS is FreeBSD, version 11.3 and 12.2 respectively.
> So far I have tried the following combos:
> - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 11.3 DOM-U: Good
> - Linux 4.19 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed
> - Linux 5.10 DOM0 + XEN 4.8 + FreeBSD 12.2 DOM-U: Regressed
> - Linux 5.10 DOM0 + XEN 4.11 + FreeBSD 12.2 DOM-U: Regressed
> 
> I plan to try out the XEN 4.14 version which is the latest I can get
> from the distro (Debian).
> If that still does not fix the problem, I would build the 4.16 version
> from source as my last resort.
> 
> I have to admit that this trial process is blind as I have no idea
> which component in the combo is to be blamed. Is it a bug in the
> backend-driver, frontend-driver or the hypervisor itself? Or due to
> incompatible versions? Any suggestion on other diagnose ideas (e.g.
> debug logs) will be welcome, while I work on the planned experiments.

This is a bug in FreeBSD netfront, so no matter which Linux or Xen
version you use.

Does it make a difference if you disable TSO and LRO from netfront?

$ ifconfig xn0 -tso -lro

Do you have instructions I can follow in order to try to reproduce the
issue?

Thanks, Roger.