From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Figo.zhang" Subject: Re: Page allocator bottleneck Date: Thu, 9 Nov 2017 11:51:02 +0800 Message-ID: References: <20170915102320.zqceocmvvkyybekj@techsingularity.net> <1c218381-067e-7757-ccc2-4e5befd2bfc3@mellanox.com> <20171103134020.3hwquerifnc6k6qw@techsingularity.net> <20171108093547.ctsjv4a42xjvfsf7@techsingularity.net> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="001a113e55ec8eb87c055d84b66e" Cc: Tariq Toukan , Linux Kernel Network Developers , linux-mm , David Miller , Jesper Dangaard Brouer , Eric Dumazet , Alexei Starovoitov , Saeed Mahameed , Eran Ben Elisha , Andrew Morton , Michal Hocko To: Mel Gorman Return-path: In-Reply-To: <20171108093547.ctsjv4a42xjvfsf7@techsingularity.net> Sender: owner-linux-mm@kvack.org List-Id: netdev.vger.kernel.org --001a113e55ec8eb87c055d84b66e Content-Type: text/plain; charset="UTF-8" @Tariq, some ideas would steal from DPDK to improve the high speed network card? such as a physical CPU dedicated for the RX and TX thread (no context switch and interrupt latency), and the memory has prepared and allocated. 2017-11-08 17:35 GMT+08:00 Mel Gorman : > On Wed, Nov 08, 2017 at 02:42:04PM +0900, Tariq Toukan wrote: > > > > Hi all, > > > > > > > > After leaving this task for a while doing other tasks, I got back to > it now > > > > and see that the good behavior I observed earlier was not stable. > > > > > > > > Recall: I work with a modified driver that allocates a page (4K) per > packet > > > > (MTU=1500), in order to simulate the stress on page-allocator in > 200Gbps > > > > NICs. > > > > > > > > > > There is almost new in the data that hasn't been discussed before. The > > > suggestion to free on a remote per-cpu list would be expensive as it > would > > > require per-cpu lists to have a lock for safe remote access. > > > > That's right, but each such lock will be significantly less congested > than > > the buddy allocator lock. > > That is not necessarily true if all the allocations and frees always happen > on the same CPUs. The contention will be equivalent to the zone lock. > Your point will only hold true if there are also heavy allocation streams > from other CPUs that are unrelated. > > > In the flow in subject two cores need to > > synchronize (one allocates, one frees). > > We also need to evaluate the cost of acquiring and releasing the lock in > the > > case of no congestion at all. > > > > If the per-cpu structures have a lock, there will be a light amount of > overhead. Nothing too severe, but it shouldn't be done lightly either. > > > > However, > > > I'd be curious if you could test the mm-pagealloc-irqpvec-v1r4 branch > > > ttps://git.kernel.org/pub/scm/linux/kernel/git/mel/linux.git . It's > an > > > unfinished prototype I worked on a few weeks ago. I was going to > revisit > > > in about a months time when 4.15-rc1 was out. I'd be interested in > seeing > > > if it has a postive gain in normal page allocations without destroying > > > the performance of interrupt and softirq allocation contexts. The > > > interrupt/softirq context testing is crucial as that is something that > > > hurt us before when trying to improve page allocator performance. > > > > > Yes, I will test that once I get back in office (after netdev conference > and > > vacation). > > Thanks. > > > Can you please elaborate in a few words about the idea behind the > prototype? > > Does it address page-allocator scalability issues, or only the rate of > > single core page allocations? > > Short answer -- maybe. All scalability issues or rates of allocation are > context and workload dependant so the question is impossible to answer > for the general case. > > Broadly speaking, the patch reintroduces the per-cpu lists being for !irq > context allocations again. The last time we did this, hard and soft IRQ > allocations went through the buddy allocator which couldn't scale and > the patch was reverted. With this patch, it goes through a very large > pagevec-like structure that is protected by a lock but the fast paths > for alloc/free are extremely simple operations so the lock hold times are > very small. Potentially, a development path is that the current per-cpu > allocator is replaced with pagevec-like structures that are dynamically > allocated which would also allow pages to be freed to remote CPU lists > (if we could detect when that is appropriate which is unclear). We could > also drain remote lists without using IPIs. The downside is that the memory > footprint of the allocator would be higher and the size could no longer > be tuned so there would need to be excellent justification for such a move. > > I haven't posted the patches properly yet because mmotm is carrying too > many patches as it is and this patch indirectly depends on the contents. I > also didn't write memory hot-remove support which would be a requirement > before merging. I hadn't intended to put further effort into it until I > had some evidence the approach had promise. My own testing indicated it > worked but the drivers I was using for network tests did not allocate > intensely enough to show any major gain/loss. > > -- > Mel Gorman > SUSE Labs > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: email@kvack.org > --001a113e55ec8eb87c055d84b66e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
@Tariq, some=C2=A0ideas=C2=A0would=C2=A0steal=C2=A0from DP= DK to improve the high=C2=A0speed=C2=A0network=C2=A0card?=C2=A0
= such=C2=A0as a physical CPU dedicated=C2=A0for=C2=A0the RX and TX=C2=A0thr= ead (no context=C2=A0switch and interrupt latency), and the=C2=A0
memory has prepared=C2=A0and allocated.
=

2017-11-08 = 17:35 GMT+08:00 Mel Gorman <mgorman@techsingularity.net><= /span>:
On Wed, Nov 08, = 2017 at 02:42:04PM +0900, Tariq Toukan wrote:
> > > Hi all,
> > >
> > > After leaving this task for a while doing other tasks, I got= back to it now
> > > and see that the good behavior I observed earlier was not st= able.
> > >
> > > Recall: I work with a modified driver that allocates a page = (4K) per packet
> > > (MTU=3D1500), in order to simulate the stress on page-alloca= tor in 200Gbps
> > > NICs.
> > >
> >
> > There is almost new in the data that hasn't been discussed be= fore. The
> > suggestion to free on a remote per-cpu list would be expensive as= it would
> > require per-cpu lists to have a lock for safe remote access.
>
> That's right, but each such lock will be significantly less conges= ted than
> the buddy allocator lock.

That is not necessarily true if all the allocations and frees always= happen
on the same CPUs. The contention will be equivalent to the zone lock.
Your point will only hold true if there are also heavy allocation streams from other CPUs that are unrelated.

> In the flow in subject two cores need to
> synchronize (one allocates, one frees).
> We also need to evaluate the cost of acquiring and releasing the lock = in the
> case of no congestion at all.
>

If the per-cpu structures have a lock, there will be a light amount = of
overhead. Nothing too severe, but it shouldn't be done lightly either.<= br>
> >=C2=A0 However,
> > I'd be curious if you could test the mm-pagealloc-irqpvec-v1r= 4 branch
> > ttps://git.kernel.org/pub/scm/= linux/kernel/git/mel/linux.git .=C2=A0 It's an
> > unfinished prototype I worked on a few weeks ago. I was going to = revisit
> > in about a months time when 4.15-rc1 was out. I'd be interest= ed in seeing
> > if it has a postive gain in normal page allocations without destr= oying
> > the performance of interrupt and softirq allocation contexts. The=
> > interrupt/softirq context testing is crucial as that is something= that
> > hurt us before when trying to improve page allocator performance.=
> >
> Yes, I will test that once I get back in office (after netdev conferen= ce and
> vacation).

Thanks.

> Can you please elaborate in a few words about the idea behind the prot= otype?
> Does it address page-allocator scalability issues, or only the rate of=
> single core page allocations?

Short answer -- maybe. All scalability issues or rates of allocation= are
context and workload dependant so the question is impossible to answer
for the general case.

Broadly speaking, the patch reintroduces the per-cpu lists being for !irq context allocations again. The last time we did this, hard and soft IRQ
allocations went through the buddy allocator which couldn't scale and the patch was reverted. With this patch, it goes through a very large
pagevec-like structure that is protected by a lock but the fast paths
for alloc/free are extremely simple operations so the lock hold times are very small. Potentially, a development path is that the current per-cpu
allocator is replaced with pagevec-like structures that are dynamically
allocated which would also allow pages to be freed to remote CPU lists
(if we could detect when that is appropriate which is unclear). We could also drain remote lists without using IPIs. The downside is that the memory=
footprint of the allocator would be higher and the size could no longer
be tuned so there would need to be excellent justification for such a move.=

I haven't posted the patches properly yet because mmotm is carrying too=
many patches as it is and this patch indirectly depends on the contents. I<= br> also didn't write memory hot-remove support which would be a requiremen= t
before merging. I hadn't intended to put further effort into it until I=
had some evidence the approach had promise. My own testing indicated it
worked but the drivers I was using for network tests did not allocate
intensely enough to show any major gain/loss.

--
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.= =C2=A0 For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=3Dmailto:"dont@kvack.org"> email@kva= ck.org </a>

--001a113e55ec8eb87c055d84b66e-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org