From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1757633AbbLBJFw (ORCPT <rfc822;w@1wt.eu>);
	Wed, 2 Dec 2015 04:05:52 -0500
Received: from mailout1.w1.samsung.com ([210.118.77.11]:19654 "EHLO
	mailout1.w1.samsung.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1756855AbbLBJFp convert rfc822-to-8bit (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Wed, 2 Dec 2015 04:05:45 -0500
X-AuditID: cbfec7f4-f79026d00000418a-98-565eb466172d
From: Pavel Fedin <p.fedin@samsung.com>
To: "'Sunil Kovvuri'" <sunil.kovvuri@gmail.com>,
        "'Eric Dumazet'" <eric.dumazet@gmail.com>
Cc: "'Linux Netdev List'" <netdev@vger.kernel.org>,
        "'LKML'" <linux-kernel@vger.kernel.org>,
        "'LAKML'" <linux-arm-kernel@lists.infradead.org>,
        "'Sunil Goutham'" <Sunil.Goutham@caviumnetworks.com>,
        "'Sunil Goutham'" <sgoutham@cavium.com>
References: <y@samsung.com>
 <1448961223-41888-4-git-send-email-sunil.kovvuri@gmail.com>
 <01d101d12c46$2de43b40$89acb1c0$@samsung.com>
 <1448983985.25582.35.camel@edumazet-glaptop2.roam.corp.google.com>
 <CA+sq2Ce9oH-kCDO8hK_wX+ZZdXyyBuT4V2bt5Ab+2=pEX+BO+w@mail.gmail.com>
In-reply-to: <CA+sq2Ce9oH-kCDO8hK_wX+ZZdXyyBuT4V2bt5Ab+2=pEX+BO+w@mail.gmail.com>
Subject: RE: [PATCH 3/6] net: thunderx: Increase transmit queue length
Date: Wed, 02 Dec 2015 12:05:41 +0300
Message-id: <00cb01d12ce0$9f090540$dd1b0fc0$@samsung.com>
MIME-version: 1.0
Content-type: text/plain; charset=UTF-8
Content-transfer-encoding: 8BIT
X-Mailer: Microsoft Outlook 14.0
Thread-index: AQGScnHROc58U50eD5J6ppH35rF/UQG2SVlqAT5pB6wB6Z4d1QJ5t4arnvoVzLA=
Content-language: ru
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrLLMWRmVeSWpSXmKPExsVy+t/xy7ppW+LCDFpfSljse3+WzWLT42us
	Fpd3zWGzOLZAzOLy5R/MFgc+7mWxWNT0jsWB3WPG70UsHhtO9LN67Jx1l91j85J6j8+b5AJY
	o7hsUlJzMstSi/TtErgyPjXcZy6YJVhxYPs+9gbGLr4uRk4OCQETidfH/7JD2GISF+6tZ+ti
	5OIQEljKKLFm/Uoo5zujxLvFrWBVbALqEqe/fmABsUUEIiSeH/gKFmcWeMUoca1DEsQWEpjK
	JHGgrbqLkYODUyBY4tkMW5CwsICbxJytq9lAbBYBVYmG/7/AWnkFLCX+fnjMBGELSvyYfI8F
	YqS6xKR5i5ghbG2JJ+8usEIcqiCx4+xrRogT/CTeHP8AVS8iMe3fPeYJjEKzkIyahWTULCSj
	ZiFpWcDIsopRNLU0uaA4KT3XUK84Mbe4NC9dLzk/dxMjJFK+7GBcfMzqEKMAB6MSD+8Knrgw
	IdbEsuLK3EOMEhzMSiK8XjJAId6UxMqq1KL8+KLSnNTiQ4zSHCxK4rxzd70PERJITyxJzU5N
	LUgtgskycXBKNTCufiQ9tSKzJOtcineB8zThU9ujprsXP2duuse+JHhZd7P/dW3L1mfC2saf
	6z6s97OO/q9pseRryGTVQ+12H+cK65gyGUcHOMzpajzEfcWpQm7njVVPdB1OX4s8oirQeiNG
	/tysyVfjH7dnTdx2pKJA6ozJ3meRVc+mqGllcUfPuThh9sdLC/YpsRRnJBpqMRcVJwIA1aMo
	O5ACAAA=
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

 Hello!

> Probably you might have to set "coherent_pool" size in bootargs to a
> higher value.
> Can you please check.

 I have tried to do this. I was able to enlarge the pool up to 4MB, and still got allocation failures. At 8MB pool preallocation stops working:
--- cut ---
Call trace:
[<ffffffc00012ddb8>] __alloc_pages_nodemask+0x4f4/0x7d4
[<ffffffc0007be370>] atomic_pool_init+0x60/0x1a4
[<ffffffc0007be4d4>] arm64_dma_init+0x20/0x28
[<ffffffc000082848>] do_one_initcall+0x8c/0x1a4
[<ffffffc0007baac0>] kernel_init_freeable+0x154/0x1f4
[<ffffffc0005c2b14>] kernel_init+0x10/0xd8
DMA: failed to allocate 8192 KiB pool for atomic coherent allocation
--- cut ---
 and i get even worse faults in the driver.

 I know that it is possible to allocate larger pools by setting CONFIG_FORCE_MAX_ZONEORDER, but:
a) This is done on per-platform basis. For ThunderX we used to have a patch (http://www.spinics.net/lists/arm-kernel/msg415457.html), which never made it upstream, because vGIC fixes stopped requiring it at some point. And also we may want to use the nicvf driver not only on actual hardware, but also inside virtual machine in KVM. So do we need to set CONFIG_FORCE_MAX_ZONEORDER for virt too? And what if at some point qemu emulates not only "virt", but some other machine (let's say AMD X-Gene), and we run it on ThunderX and want to use nicvf with this model?
b) IMHO it's not good to have a driver which simply does not work without some obscure option in boot arguments.

 So, i see several possible ways to solve this:

1. Introduce some mechanism which would allow the driver to tell the kernel that it needs coherent pool of large size. Can be problematic because the driver can be a module, and pool allocation happens early.
2. Can we use some other method for allocating queues, which would not require such a huge coherent pool?
3. The driver could check value of atomic_pool_size and adjust own memory requirements accordingly. This indeed looks like a quick hack, but would at least make things running quickly.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia