From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753164AbcEIR3L (ORCPT ); Mon, 9 May 2016 13:29:11 -0400 Received: from mail-am1on0092.outbound.protection.outlook.com ([157.56.112.92]:30131 "EHLO emea01-am1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750986AbcEIR3D (ORCPT ); Mon, 9 May 2016 13:29:03 -0400 Authentication-Results: spf=pass (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; synopsys.com; dkim=none (message not signed) header.d=none;synopsys.com; dmarc=pass action=none header.from=mellanox.com; From: Elad Kanfi To: CC: , , , , Subject: [PATCH v3 1/2] net: nps_enet: Tx handler synchronization Date: Mon, 9 May 2016 20:13:19 +0300 Message-ID: <1462814000-16331-2-git-send-email-eladkan@mellanox.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1462814000-16331-1-git-send-email-eladkan@mellanox.com> References: <1462814000-16331-1-git-send-email-eladkan@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain X-TM-AS-Product-Ver: SMEX-11.0.0.1191-8.000.1202-22312.001 X-TM-AS-Result: No--4.718500-8.000000-31 X-TM-AS-User-Approved-Sender: No X-TM-AS-User-Blocked-Sender: No X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:193.47.165.134;IPV:NLI;CTRY:IL;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(2980300002)(438002)(189002)(199003)(9170700002)(33646002)(11100500001)(2906002)(575784001)(586003)(86362001)(92566002)(5003940100001)(87936001)(49486002)(101416001)(50986999)(76176999)(2950100001)(19580395003)(19580405001)(110136002)(5008740100001)(6806005)(50226002)(551934003)(36756003)(4326007)(8936002)(77096005)(2351001)(1220700001)(189998001)(106466001)(229853001)(47776003)(48376002)(81166005)(50466002);DIR:OUT;SFP:1101;SCL:1;SRVR:AM2PR05MB0691;H:mtlcas13.mtl.com;FPR:;SPF:Pass;MLV:sfv;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;DB3FFO11FD012;1:0YYd0q9vvnqqbi35govU1JP6sQIYTaFiZb9c284o17xOh2YXwOpu47BiQdD3VbdT5NnkY+t08Q4mcsPRkekE8vFmb/QwTeArPX33y5qbbie838GSvZEbeIRLvQvjxPt7YuVGCuR7raPwMnjZQ7wwL+NyhIoG1Yhh3jl2z0uL7jvNLF3z33yKYzYVQs1kZg2xJcRZQHiAwRO62cJQlD8FfysCvlACU4Dmdb5VQ6G3kEAJxkKwsu9EFagK8oQ/JuGhpEZRN1EqYvFS5SMqPfgz60EH8PVDx9lQYTG5SdM3y9OPR773tjBZM1ItBHUWVzPrc1rlbK6HO5MpzVqKMX65eg3A9IEJ1tKNSfLipxPDj3zhF2n6rUnelxZ4Nls6E44/562Guf512GTaFEpn20GhmHSqxFkmfdBZhjxp9ITgGXCo9MMmZFhEZyCekmkbVj8KZHQcR5n0S7/I8CIhHEZ9o4Of3wQ3lOOCaPEBPYVTJllUvmtc5lobcMKGKNTZ2zBtxxKpVFeLF4da1VAng1jzmU29cm3LpOhTPt8ep0Phs4Y= X-MS-Office365-Filtering-Correlation-Id: 3bc56cc6-4c7c-421c-0d7b-08d3782d4589 X-Microsoft-Exchange-Diagnostics: 1;AM2PR05MB0691;2:BKsT7xktb9cJSNynBxEc9vcldat8edzQZ2neEDmmTClHSx8BGWeqUEuHeyv0cfbbmiWiwJVjmO4Jb0F8ModCTSeXRAazFIH9fdhJN31VPvjxwRQpc0f3hdpQO712MIVenejQ+5T4Zd8LZV7uQYfWLscz8Rp455lXCj/CO3tvb5r1QFAsE3uA6tiyO45pRWSJ;3:V3+N3sr6SAfmNMlOlo7Ig0Qi3LCbpwQzdFCQ4/BsKz+gyGOhwjWn2yj2CDRuZyvDKuCNHzPzIFycTroZjBH34x2L5BWpBVVksgMA7Ihks3dTCUkcYF8FZNyTlrCWRQkmBmx7sRXA8fWLnrP8tikJcdVHv6HniIn7Fa/b9UKG7Hey6Qi0LG3yCVy3Q4X9Ivs/j0w9aRqTv9+71jXgB3BqOWJ/94krhyPBFvP9y/gO0uu8PK8D/tAkyKm+MDKnNoks2Lrr3FdbAmeUponJenXxeA== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(8251501002);SRVR:AM2PR05MB0691; X-Microsoft-Exchange-Diagnostics: 1;AM2PR05MB0691;25:HSSzJg6IRmbQM2cevm/7ZG3wY+/68uAhZSQreT0D5SveWvIa7ZQLgGhtcgWiNqE8QifAdqGJ5QeuF9X1bhMQSkXPe+VZFW36E6bOrV4/Lhdacjy0sjHF4Q0cywqZi+gAyK+ky4sPzfVFGi1Pogn9OC0BR0mDm9MYEmYC5bufccMkKzsy+ZMgiZW7F6bQprhsOvm3O9ExsX8CrDEVaw6kARtryzm6O92LyXQUlhJqvGsOpax91vhCD+iaVf6D7k6dU3EmsiU25W8ED7xduD/BTDLy4akrXR6xzLYzS2bWkSmRpRdZtTJUHtK91x+Kbe+qA/iircJlwhVABMJarXUdnU9938iWKOVm6+4FBDIz5XNoLWu6Y674v6TOImMc6I3pS0M1OnEiLQ20DEDsJcK1sS1r2AqLcadzdevjRv5/qR+sldLoPpcNdIVozJxpCuJVzd1UMfTNA41oUt9XS3B8qvziViYYdbisO8UkBfx2jYbSDfYcz36uatPY2RMKRUruELxLT+AV01Er+XXJYIAvAH6MvI7sKnxLiXhjqL06dbxDAlRRFH+O+cT+B182PkOV/r+sQ5/Q8VEczzvzlxUZ9DGfXfe0nzA8KPb6bEHf0U0w/X3Z7+w9x4j/KJLO6EXkW+EJ7NOf0fpD0FZvNnkLGn/UTZ7i+/3kdba1/kwOxBvP4pqhoqc/eOTDYBrETxehDCuo4kCJjZDMVgt8cZpA65Xu4mT3L7+26vJVXfembgg1pw1I5RsElQtB9b9H3plb0Bxx1iPIy3hbZdT5cHPbvANsAGyAYKljr6O2WrFchAMiDEdDJE3rWM9ChokrBZ7D X-Microsoft-Exchange-Diagnostics: 1;AM2PR05MB0691;20:AjPqcdwTc0/9BHvoi0FPfh5NuiswIuglMapUQuzHok2JRmfDlNVSpCcp+2SYzI3VKLNpB0e93qbbKqqkK0jcaVxm5lnuXPlcOb5uFXsPDADTi0/yqp/rN6nfIc28p+IRHVwBwU/6+I2bEcGah19Jtp89nuLKVt1wVPNqDEnv1w9D7Xud8TP2y5Pq/dc9KWj+teliOIHgp6nihLoLD0RSwWz46sted7y43a23oTPEIpBOFMy6Yg46iYcIhSNBFR1oyQfgnExZbv+yPR3BgR+RsJ1DDAJICeCiA1k6Ec0aJjmsTGgO5beIAgPze/5ImmteI53NTNaAIKaJ4Yf9yB942crQzcmvpbmb/J0nh3D7c7BiTmvjbtlv+MxZRMeSUCbcFLR/9Er3xt67XbOZnifiKKFkYy3T7i3+CV04DPPijwAmphXrdsv1ZVsi5rQ55eSzmBCiy+gXLvgaXaY96+lm9kKXhIBFC3ORoSFdeO6pSm9sTlh84P2u6A4n5OYuOZI9 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(13024025)(13023025)(13015025)(13018025)(13017025)(5005006)(8121501046)(3002001)(10201501046)(6055026);SRVR:AM2PR05MB0691;BCL:0;PCL:0;RULEID:;SRVR:AM2PR05MB0691; X-Microsoft-Exchange-Diagnostics: 1;AM2PR05MB0691;4:qPJIALU6VZWhJPoGeYj0w67GREALowlKTWKchDfXupGrfQNS9R2o1IJ9a4mTlaEX/GY3H77goI4WX9ZPE/TUAJrvbEqjxRFZB1zsyibBFE/fhdJuFbs+UvKus1gArSjiSuQFKN9hBGZrJzNZvBw3Pki9pL+2AaWFeaCqDTZaUqfMZKE2euixP3ipTarhuDhAWyO46zU88hdZQvhPzOyvE2/8OFz5M+xR7JE8a2o3XywB+Ok4OU6cDYF/ou8u4KFtWQkr2uWow9z9j/RsCG4k/CqhcSqGr828wxablFo8qZ5TYh8ynqVmmIQxNx95OxOi5TGP3WHIbI3ykDrFH4lnGyHc39ReGZ/jB5neZM3+nHba9XDVffghtfstIDDEZZ95uVbXh04e4AA+w0//F+5gp/6Z63ZWn1VpGHsW1JbdeV6KhsbAC59bSR3pu+c6sdSbGrVDyzMzWXB+Bm72Y0sTCXiO8MSamP4GPsZSi5+3qQ0= X-Forefront-PRVS: 0937FB07C5 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;AM2PR05MB0691;23:c9V1lvFgyRoKYuvl63m8aaoJmO1rQdtyVs78IZU90?= =?us-ascii?Q?R74qwvC+z6rB6EDlJuCXi3gP//gDp5hTMWOhK76GsOqYN/2rnTCVBBjnqZDC?= =?us-ascii?Q?ZE4j5/NtKPcHAGO/7OBEEnaGhM7wRgKdZsP+ppSC4Q6C42iXXn0kOGLiQdhi?= =?us-ascii?Q?7BYIn/kZ1QzyTdflPBKIkXkjfg9uXUV5eJLeIVjgrRwYPxd3YgSBRNHsckFr?= =?us-ascii?Q?WZ8SHILaLI12ePu6pRkOMJmQHmrDcmDYZN8DD+hQ2hEM2IlsejDJqE71oL2B?= =?us-ascii?Q?5MJms6Zq/tOGyLcLSySHJdGddgacch0jcVu/lpnBhPVArBjCvaK97LlqTTuV?= =?us-ascii?Q?gYIGMlnTtBpouVM9rqpc/CsQe1Q9HSstTS5uFLYF8BN7hxsB+tWAfvHGaSLu?= =?us-ascii?Q?uLU18VzI0NPIX0zMWOouBGRz6gNvXQn6NWSX+UbACwRPM9YTKemALTVxG/i6?= =?us-ascii?Q?29TDdpsqDK6qPk0DfWwWYYB2rbpHaleTU4iZcpTHShwtWhpguLb3l1ejTRzd?= =?us-ascii?Q?eVxCjkCC4/qFvIShPEIvyKAh1L7wr9C+3kZ+J0txQuFRfmSeOCouG57/mEYa?= =?us-ascii?Q?DQs55xc7o7Dzv5YVOHXz7Kmd824OwKHqBdS1s9oTl7a/Sw1tb1GFRvdO3Sdb?= =?us-ascii?Q?o4C5T2vdvVRPP0WFuFBb/Mnid46EKuc5+pUzvmwwxfZPpIHW0JxRv/G/dveE?= =?us-ascii?Q?+ghwDpAYElk9kR7R1GxfCVA4wy/4EDFTHHjfSZa8HTdUcbiQF3/yyjy1t2fp?= =?us-ascii?Q?uQNl3syvMw6FvnYVSIiD9+lO/5lhgPnVBukjv9GKsALOq/t/wisyAY1j3dF+?= =?us-ascii?Q?psUKDg0A8lhDDcpPiPOR8KOorjTUxyEHDvSZHckdCen+2a1TnB0Ke8w0Bf7q?= =?us-ascii?Q?tC9Pes5/LYGtW7F0HwI9sDC2ebq9HzEiH+cguxQkryajXbanEfE5fPc2gd/9?= =?us-ascii?Q?WsjMKfhYvN6qpUMQWUQhqs8ZN/+izk/o9N1JrL9FgZms73Q7Ywr++paE5SN5?= =?us-ascii?Q?loqowR4J7IOERKD+0lZUJ28JwbWDrg2XEdldiPwPvJ4XMr4/W0WI6ctIHHg+?= =?us-ascii?Q?ziSIfg=3D?= X-Microsoft-Exchange-Diagnostics: 1;AM2PR05MB0691;5:9Co5qKx8K9o+6bRKuB6nY4AzsN+aj0V3JDEkzBzGzSD97cKxn27XyJgUoyzHwhxuzbnc/4QDfZjsRJBpEVMwI2qdMrE1j3x/ovWrgrDW0038FdarF1XLZX5Z/+LbFsEy1/COfkWx7X47h1RDFOxQXg==;24:5z2j5e6f1HglslAHF1GQmaUK8freD/ylLmJEpBrEi6a+w9N8gGk/84NOJ7lLIyYKOIQ4PZzT8gwu/XHu8W5kdaThhCUxQfYpfpGVVV8pBUU=;7:vENiTDFqrk0l8PK8M3fmkB9wotu3iMj2bXT0u0lzJFxxtu0iTcMAtm973ff2eVj9xkW8NYbY56C5dYSK0R+uC0LrES3RxYTSZHO4VtdxpNqzLPB1hNDqT++ast3jJQ/OChZ7R2u5tglFRgQMP9RdRrG+rpcH0GeYgSTrivRYKY3vPc78zltGAKqsn1zW/K0x SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2016 17:13:42.2659 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b;Ip=[193.47.165.134];Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM2PR05MB0691 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Elad Kanfi Below is a description of a possible problematic sequence. CPU-A is sending a frame and CPU-B handles the interrupt that indicates the frame was sent. CPU-B reads an invalid value of tx_packet_sent. CPU-A CPU-B ----- ----- nps_enet_send_frame . . tx_skb = skb tx_packet_sent = true order HW to start tx . . HW complete tx ------> get tx complete interrupt . . if(tx_packet_sent == true) handle tx_skb end memory transaction (tx_packet_sent actually written) Furthermore there is a dependency between tx_skb and tx_packet_sent. There is no assurance that tx_skb contains a valid pointer at CPU B when it sees tx_packet_sent == true. Solution: Initialize tx_skb to NULL and use it to indicate that packet was sent, in this way tx_packet_sent can be removed. Add a write memory barrier after setting tx_skb in order to make sure that it is valid before HW is informed and IRQ is fired. Fixed sequence will be: CPU-A CPU-B ----- ----- tx_skb = skb wmb() . . order HW to start tx . . HW complete tx ------> get tx complete interrupt . . if(tx_skb != NULL) handle tx_skb tx_skb = NULL Signed-off-by: Elad Kanfi Acked-by: Noam Camus Acked-by: Gilad Ben-Yossef --- drivers/net/ethernet/ezchip/nps_enet.c | 15 +++++++++------ drivers/net/ethernet/ezchip/nps_enet.h | 2 -- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/drivers/net/ethernet/ezchip/nps_enet.c b/drivers/net/ethernet/ezchip/nps_enet.c index 1f23845..25ac2de 100644 --- a/drivers/net/ethernet/ezchip/nps_enet.c +++ b/drivers/net/ethernet/ezchip/nps_enet.c @@ -145,7 +145,7 @@ static void nps_enet_tx_handler(struct net_device *ndev) u32 tx_ctrl_nt = (tx_ctrl_value & TX_CTL_NT_MASK) >> TX_CTL_NT_SHIFT; /* Check if we got TX */ - if (!priv->tx_packet_sent || tx_ctrl_ct) + if (!priv->tx_skb || tx_ctrl_ct) return; /* Ack Tx ctrl register */ @@ -160,7 +160,7 @@ static void nps_enet_tx_handler(struct net_device *ndev) } dev_kfree_skb(priv->tx_skb); - priv->tx_packet_sent = false; + priv->tx_skb = NULL; if (netif_queue_stopped(ndev)) netif_wake_queue(ndev); @@ -217,7 +217,7 @@ static irqreturn_t nps_enet_irq_handler(s32 irq, void *dev_instance) u32 tx_ctrl_ct = (tx_ctrl_value & TX_CTL_CT_MASK) >> TX_CTL_CT_SHIFT; u32 rx_ctrl_cr = (rx_ctrl_value & RX_CTL_CR_MASK) >> RX_CTL_CR_SHIFT; - if ((!tx_ctrl_ct && priv->tx_packet_sent) || rx_ctrl_cr) + if ((!tx_ctrl_ct && priv->tx_skb) || rx_ctrl_cr) if (likely(napi_schedule_prep(&priv->napi))) { nps_enet_reg_set(priv, NPS_ENET_REG_BUF_INT_ENABLE, 0); __napi_schedule(&priv->napi); @@ -387,8 +387,6 @@ static void nps_enet_send_frame(struct net_device *ndev, /* Write the length of the Frame */ tx_ctrl_value |= length << TX_CTL_NT_SHIFT; - /* Indicate SW is done */ - priv->tx_packet_sent = true; tx_ctrl_value |= NPS_ENET_ENABLE << TX_CTL_CT_SHIFT; /* Send Frame */ nps_enet_reg_set(priv, NPS_ENET_REG_TX_CTL, tx_ctrl_value); @@ -465,7 +463,7 @@ static s32 nps_enet_open(struct net_device *ndev) s32 err; /* Reset private variables */ - priv->tx_packet_sent = false; + priv->tx_skb = NULL; priv->ge_mac_cfg_2_value = 0; priv->ge_mac_cfg_3_value = 0; @@ -534,6 +532,11 @@ static netdev_tx_t nps_enet_start_xmit(struct sk_buff *skb, priv->tx_skb = skb; + /* make sure tx_skb is actually written to the memory + * before the HW is informed and the IRQ is fired. + */ + wmb(); + nps_enet_send_frame(ndev, skb); return NETDEV_TX_OK; diff --git a/drivers/net/ethernet/ezchip/nps_enet.h b/drivers/net/ethernet/ezchip/nps_enet.h index d0cab60..3939ca2 100644 --- a/drivers/net/ethernet/ezchip/nps_enet.h +++ b/drivers/net/ethernet/ezchip/nps_enet.h @@ -165,14 +165,12 @@ * struct nps_enet_priv - Storage of ENET's private information. * @regs_base: Base address of ENET memory-mapped control registers. * @irq: For RX/TX IRQ number. - * @tx_packet_sent: SW indication if frame is being sent. * @tx_skb: socket buffer of sent frame. * @napi: Structure for NAPI. */ struct nps_enet_priv { void __iomem *regs_base; s32 irq; - bool tx_packet_sent; struct sk_buff *tx_skb; struct napi_struct napi; u32 ge_mac_cfg_2_value; -- 1.7.1