[Qemu-devel] [Bug 1402755] [NEW] qemu-kvm: e1000 RX ring is filled with partial-pkt of size 0

* [Qemu-devel] [Bug 1402755] [NEW] qemu-kvm: e1000 RX ring is filled with partial-pkt of size 0
@ 2014-12-15 16:59 Moti
  2015-01-07 15:57 ` Stefan Hajnoczi
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Moti @ 2014-12-15 16:59 UTC (permalink / raw)
  To: qemu-devel

Public bug reported:

Hello,
We are using CentOS 6.5 with qemu-kvm-0.12.1.2-2.415 as a host of or VMs.
In the VM we use e1000 as the NIC emulation.
We've modified the e1000 driver to our needs. This modification start the RX engine while the RX ring is empty (RDH == RDT)
and at a later stage we fill the RX descriptors with buffers. This scheme works well on intel chips and VMware.
What we observe in this setup is that from time to time the RX ring is filled with "partial packets" of size 0 (meaning, DD bit is set,
No other status bits are set and packet size is also 0).

Looking at the e1000 RX routine in qemu-kvm you can observe the following flow:
1. A packet is avail for receive:
2. The routine checks for RCTL_EN - it is enabled
3. The routine checks that the RDH equal RDT (they are as the ring is empty) but also checks if rxov is on (it is still off) so it doesn’t
Exit as it is supposed to.
4. The routine now updates the descriptor status with the DD bit (and vlan which we don’t care)
5. The routine checks if a buffer address is not NULL (it is as NULL since we haven’t filled it yet) – so is logs something.
6. The routine now updates the guest memory with this value (DD is 1) 
7. The routine updates the check_rxov flag in order to allow ovf check the next time around. 
(but ovf will not occur since in the next iteration RDH != RDT)
8. The routine loops over all the descriptors with the NULL buffer (which is all our ring) and writes the DD bit
9. We get this endless partial packet problem we see.

qemu-kvm-0.12.1.2-2.415.el6_5.3/qemu-kvm-0.12.1.2/hw/e1000.c
static ssize_t
e1000_receive(VLANClientState *nc, const uint8_t *buf, size_t size)
{
: : :
if (!(s->mac_reg[RCTL] & E1000_RCTL_EN))
return -1;

: : :
do {
if (s->mac_reg[RDH] == s->mac_reg[RDT] && s->check_rxov) {
set_ics(s, 0, E1000_ICS_RXO);
return -1;
}
base = ((uint64_t)s->mac_reg[RDBAH] << 32) + s->mac_reg[RDBAL] +
sizeof(desc) * s->mac_reg[RDH];
cpu_physical_memory_read(base, (void *)&desc, sizeof(desc));
desc.special = vlan_special;
desc.status |= (vlan_status | E1000_RXD_STAT_DD);
if (desc.buffer_addr) {
cpu_physical_memory_write(le64_to_cpu(desc.buffer_addr),
(void *)(buf + vlan_offset), size);
desc.length = cpu_to_le16(size);
desc.status |= E1000_RXD_STAT_EOP|E1000_RXD_STAT_IXSM;
} else // as per intel docs; skip descriptors with null buf addr
DBGOUT(RX, "Null RX descriptor!!\n");
cpu_physical_memory_write(base, (void *)&desc, sizeof(desc));

: : :
if (++s->mac_reg[RDH] * sizeof(desc) >= s->mac_reg[RDLEN])
s->mac_reg[RDH] = 0;
s->check_rxov = 1;
: : :
} while (desc.buffer_addr == 0);
}

A workaround is to enable the RX machine only after the descriptor ring is filled for the first time.

Moti

** Affects: qemu
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1402755

Title:
  qemu-kvm: e1000 RX ring is filled with partial-pkt of size 0

Status in QEMU:
  New

Bug description:
  Hello,
  We are using CentOS 6.5 with qemu-kvm-0.12.1.2-2.415 as a host of or VMs.
  In the VM we use e1000 as the NIC emulation.
  We've modified the e1000 driver to our needs. This modification start the RX engine while the RX ring is empty (RDH == RDT)
  and at a later stage we fill the RX descriptors with buffers. This scheme works well on intel chips and VMware.
  What we observe in this setup is that from time to time the RX ring is filled with "partial packets" of size 0 (meaning, DD bit is set,
  No other status bits are set and packet size is also 0).

  Looking at the e1000 RX routine in qemu-kvm you can observe the following flow:
  1. A packet is avail for receive:
  2. The routine checks for RCTL_EN - it is enabled
  3. The routine checks that the RDH equal RDT (they are as the ring is empty) but also checks if rxov is on (it is still off) so it doesn’t
  Exit as it is supposed to.
  4. The routine now updates the descriptor status with the DD bit (and vlan which we don’t care)
  5. The routine checks if a buffer address is not NULL (it is as NULL since we haven’t filled it yet) – so is logs something.
  6. The routine now updates the guest memory with this value (DD is 1) 
  7. The routine updates the check_rxov flag in order to allow ovf check the next time around. 
  (but ovf will not occur since in the next iteration RDH != RDT)
  8. The routine loops over all the descriptors with the NULL buffer (which is all our ring) and writes the DD bit
  9. We get this endless partial packet problem we see.

  qemu-kvm-0.12.1.2-2.415.el6_5.3/qemu-kvm-0.12.1.2/hw/e1000.c
  static ssize_t
  e1000_receive(VLANClientState *nc, const uint8_t *buf, size_t size)
  {
  : : :
  if (!(s->mac_reg[RCTL] & E1000_RCTL_EN))
  return -1;

  : : :
  do {
  if (s->mac_reg[RDH] == s->mac_reg[RDT] && s->check_rxov) {
  set_ics(s, 0, E1000_ICS_RXO);
  return -1;
  }
  base = ((uint64_t)s->mac_reg[RDBAH] << 32) + s->mac_reg[RDBAL] +
  sizeof(desc) * s->mac_reg[RDH];
  cpu_physical_memory_read(base, (void *)&desc, sizeof(desc));
  desc.special = vlan_special;
  desc.status |= (vlan_status | E1000_RXD_STAT_DD);
  if (desc.buffer_addr) {
  cpu_physical_memory_write(le64_to_cpu(desc.buffer_addr),
  (void *)(buf + vlan_offset), size);
  desc.length = cpu_to_le16(size);
  desc.status |= E1000_RXD_STAT_EOP|E1000_RXD_STAT_IXSM;
  } else // as per intel docs; skip descriptors with null buf addr
  DBGOUT(RX, "Null RX descriptor!!\n");
  cpu_physical_memory_write(base, (void *)&desc, sizeof(desc));

  : : :
  if (++s->mac_reg[RDH] * sizeof(desc) >= s->mac_reg[RDLEN])
  s->mac_reg[RDH] = 0;
  s->check_rxov = 1;
  : : :
  } while (desc.buffer_addr == 0);
  }

  A workaround is to enable the RX machine only after the descriptor ring is filled for the first time.

  Moti

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1402755/+subscriptions

^ permalink raw reply	[flat|nested] 4+ messages in thread