From: Mel Gorman <mgorman@techsingularity.net> To: Dave Chinner <david@fromorbit.com> Cc: Linus Torvalds <torvalds@linux-foundation.org>, Michal Hocko <mhocko@suse.cz>, Minchan Kim <minchan@kernel.org>, Vladimir Davydov <vdavydov@virtuozzo.com>, Johannes Weiner <hannes@cmpxchg.org>, Vlastimil Babka <vbabka@suse.cz>, Andrew Morton <akpm@linux-foundation.org>, Bob Peterson <rpeterso@redhat.com>, "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>, "Huang, Ying" <ying.huang@intel.com>, Christoph Hellwig <hch@lst.de>, Wu Fengguang <fengguang.wu@intel.com>, LKP <lkp@01.org>, Tejun Heo <tj@kernel.org>, LKML <linux-kernel@vger.kernel.org> Subject: Re: [LKP] [lkp] [xfs] 68a9f5e700: aim7.jobs-per-min -13.6% regression Date: Thu, 18 Aug 2016 14:24:14 +0100 [thread overview] Message-ID: <20160818132414.GK8119@techsingularity.net> (raw) In-Reply-To: <20160818071111.GD22388@dastard> On Thu, Aug 18, 2016 at 05:11:11PM +1000, Dave Chinner wrote: > On Thu, Aug 18, 2016 at 01:45:17AM +0100, Mel Gorman wrote: > > On Wed, Aug 17, 2016 at 04:49:07PM +0100, Mel Gorman wrote: > > > > Yes, we could try to batch the locking like DaveC already suggested > > > > (ie we could move the locking to the caller, and then make > > > > shrink_page_list() just try to keep the lock held for a few pages if > > > > the mapping doesn't change), and that might result in fewer crazy > > > > cacheline ping-pongs overall. But that feels like exactly the wrong > > > > kind of workaround. > > > > > > > > > > Even if such batching was implemented, it would be very specific to the > > > case of a single large file filling LRUs on multiple nodes. > > > > > > > The latest Jason Bourne movie was sufficiently bad that I spent time > > thinking how the tree_lock could be batched during reclaim. It's not > > straight-forward but this prototype did not blow up on UMA and may be > > worth considering if Dave can test either approach has a positive impact. > > SO, I just did a couple of tests. I'll call the two patches "sleepy" > for the contention backoff patch and "bourney" for the Jason Bourne > inspired batching patch. This is an average of 3 runs, overwriting > a 47GB file on a machine with 16GB RAM: > > IO throughput wall time __pv_queued_spin_lock_slowpath > vanilla 470MB/s 1m42s 25-30% > sleepy 295MB/s 2m43s <1% > bourney 425MB/s 1m53s 25-30% > Thanks. I updated the tests today and reran them trying to reproduce what you saw but I'm simply not seeing it on bare metal with a spinning disk. xfsio Throughput 4.8.0-rc2 4.8.0-rc2 4.8.0-rc2 vanilla sleepy bourney Min tput 147.4450 ( 0.00%) 147.2580 ( 0.13%) 147.3900 ( 0.04%) Hmean tput 147.5853 ( 0.00%) 147.5101 ( 0.05%) 147.6121 ( -0.02%) Stddev tput 0.1041 ( 0.00%) 0.1785 (-71.47%) 0.2036 (-95.63%) CoeffVar tput 0.0705 ( 0.00%) 0.1210 (-71.56%) 0.1379 (-95.59%) Max tput 147.6940 ( 0.00%) 147.6420 ( 0.04%) 147.8820 ( -0.13%) I'm currently setting up a KVM instance that may fare better. Due to quirks of where machines are, I have to setup the KVM instance on real NUMA hardware but maybe that'll make the problem even more obvious. > The overall CPU usage of sleepy was much lower than the others, but > it was also much slower. Too much sleeping and not enough reclaim > work being done, I think. > Looks like it. On my initial test, there was barely any sleeping. > As for bourney, it's not immediately clear as to why it's nearly as > bad as the movie. At worst I would have expected it to have not > noticable impact, but maybe we are delaying freeing of pages too > long and so stalling allocation of new pages? It also doesn't do > much to reduce contention, especially considering the reduction in > throughput. > > On a hunch that the batch list isn't all one mapping, I sorted it. > Patch is below if you're curious. > The fact that sorting makes such a difference makes me think that it's the wrong direction. It's far too specific to this test case and does nothing to throttle a reclaimer. It's also fairly complex and I expected that normal users of remove_mapping such as truncation would take a hit. The hit of bouncing the lock around just hurts too much. > FWIW, I just remembered about /proc/sys/vm/zone_reclaim_mode. > That is a terrifying "fix" for this problem. It just happens to work because there is no spillover to other nodes so only one kswapd instance is potentially active. > Anyway, I've burnt enough erase cycles on this SSD for today.... > I'll continue looking at getting KVM up and running and then consider other possibilities for throttling. -- Mel Gorman SUSE Labs
WARNING: multiple messages have this Message-ID (diff)
From: Mel Gorman <mgorman@techsingularity.net> To: lkp@lists.01.org Subject: Re: [xfs] 68a9f5e700: aim7.jobs-per-min -13.6% regression Date: Thu, 18 Aug 2016 14:24:14 +0100 [thread overview] Message-ID: <20160818132414.GK8119@techsingularity.net> (raw) In-Reply-To: <20160818071111.GD22388@dastard> [-- Attachment #1: Type: text/plain, Size: 3983 bytes --] On Thu, Aug 18, 2016 at 05:11:11PM +1000, Dave Chinner wrote: > On Thu, Aug 18, 2016 at 01:45:17AM +0100, Mel Gorman wrote: > > On Wed, Aug 17, 2016 at 04:49:07PM +0100, Mel Gorman wrote: > > > > Yes, we could try to batch the locking like DaveC already suggested > > > > (ie we could move the locking to the caller, and then make > > > > shrink_page_list() just try to keep the lock held for a few pages if > > > > the mapping doesn't change), and that might result in fewer crazy > > > > cacheline ping-pongs overall. But that feels like exactly the wrong > > > > kind of workaround. > > > > > > > > > > Even if such batching was implemented, it would be very specific to the > > > case of a single large file filling LRUs on multiple nodes. > > > > > > > The latest Jason Bourne movie was sufficiently bad that I spent time > > thinking how the tree_lock could be batched during reclaim. It's not > > straight-forward but this prototype did not blow up on UMA and may be > > worth considering if Dave can test either approach has a positive impact. > > SO, I just did a couple of tests. I'll call the two patches "sleepy" > for the contention backoff patch and "bourney" for the Jason Bourne > inspired batching patch. This is an average of 3 runs, overwriting > a 47GB file on a machine with 16GB RAM: > > IO throughput wall time __pv_queued_spin_lock_slowpath > vanilla 470MB/s 1m42s 25-30% > sleepy 295MB/s 2m43s <1% > bourney 425MB/s 1m53s 25-30% > Thanks. I updated the tests today and reran them trying to reproduce what you saw but I'm simply not seeing it on bare metal with a spinning disk. xfsio Throughput 4.8.0-rc2 4.8.0-rc2 4.8.0-rc2 vanilla sleepy bourney Min tput 147.4450 ( 0.00%) 147.2580 ( 0.13%) 147.3900 ( 0.04%) Hmean tput 147.5853 ( 0.00%) 147.5101 ( 0.05%) 147.6121 ( -0.02%) Stddev tput 0.1041 ( 0.00%) 0.1785 (-71.47%) 0.2036 (-95.63%) CoeffVar tput 0.0705 ( 0.00%) 0.1210 (-71.56%) 0.1379 (-95.59%) Max tput 147.6940 ( 0.00%) 147.6420 ( 0.04%) 147.8820 ( -0.13%) I'm currently setting up a KVM instance that may fare better. Due to quirks of where machines are, I have to setup the KVM instance on real NUMA hardware but maybe that'll make the problem even more obvious. > The overall CPU usage of sleepy was much lower than the others, but > it was also much slower. Too much sleeping and not enough reclaim > work being done, I think. > Looks like it. On my initial test, there was barely any sleeping. > As for bourney, it's not immediately clear as to why it's nearly as > bad as the movie. At worst I would have expected it to have not > noticable impact, but maybe we are delaying freeing of pages too > long and so stalling allocation of new pages? It also doesn't do > much to reduce contention, especially considering the reduction in > throughput. > > On a hunch that the batch list isn't all one mapping, I sorted it. > Patch is below if you're curious. > The fact that sorting makes such a difference makes me think that it's the wrong direction. It's far too specific to this test case and does nothing to throttle a reclaimer. It's also fairly complex and I expected that normal users of remove_mapping such as truncation would take a hit. The hit of bouncing the lock around just hurts too much. > FWIW, I just remembered about /proc/sys/vm/zone_reclaim_mode. > That is a terrifying "fix" for this problem. It just happens to work because there is no spillover to other nodes so only one kswapd instance is potentially active. > Anyway, I've burnt enough erase cycles on this SSD for today.... > I'll continue looking at getting KVM up and running and then consider other possibilities for throttling. -- Mel Gorman SUSE Labs
next prev parent reply other threads:[~2016-08-18 13:24 UTC|newest] Thread overview: 219+ messages / expand[flat|nested] mbox.gz Atom feed top 2016-08-09 14:33 [lkp] [xfs] 68a9f5e700: aim7.jobs-per-min -13.6% regression kernel test robot 2016-08-09 14:33 ` kernel test robot 2016-08-10 18:24 ` [lkp] " Linus Torvalds 2016-08-10 18:24 ` Linus Torvalds 2016-08-10 23:08 ` [lkp] " Dave Chinner 2016-08-10 23:08 ` Dave Chinner 2016-08-10 23:51 ` [lkp] " Linus Torvalds 2016-08-10 23:51 ` Linus Torvalds 2016-08-10 23:58 ` [LKP] [lkp] " Huang, Ying 2016-08-10 23:58 ` Huang, Ying 2016-08-11 0:11 ` [LKP] [lkp] " Huang, Ying 2016-08-11 0:11 ` Huang, Ying 2016-08-11 0:23 ` [LKP] [lkp] " Linus Torvalds 2016-08-11 0:23 ` Linus Torvalds 2016-08-11 0:33 ` [LKP] [lkp] " Huang, Ying 2016-08-11 0:33 ` Huang, Ying 2016-08-11 1:00 ` [LKP] [lkp] " Linus Torvalds 2016-08-11 1:00 ` Linus Torvalds 2016-08-11 4:46 ` [LKP] [lkp] " Dave Chinner 2016-08-11 4:46 ` Dave Chinner 2016-08-15 17:22 ` [LKP] [lkp] " Huang, Ying 2016-08-15 17:22 ` Huang, Ying 2016-08-16 0:08 ` [LKP] [lkp] " Dave Chinner 2016-08-16 0:08 ` Dave Chinner 2016-08-11 15:57 ` [LKP] [lkp] " Christoph Hellwig 2016-08-11 15:57 ` Christoph Hellwig 2016-08-11 16:55 ` [LKP] [lkp] " Linus Torvalds 2016-08-11 16:55 ` Linus Torvalds 2016-08-11 17:51 ` [LKP] [lkp] " Huang, Ying 2016-08-11 17:51 ` Huang, Ying 2016-08-11 19:51 ` [LKP] [lkp] " Linus Torvalds 2016-08-11 19:51 ` Linus Torvalds 2016-08-11 20:00 ` [LKP] [lkp] " Christoph Hellwig 2016-08-11 20:00 ` Christoph Hellwig 2016-08-11 20:35 ` [LKP] [lkp] " Linus Torvalds 2016-08-11 20:35 ` Linus Torvalds 2016-08-11 22:16 ` [LKP] [lkp] " Al Viro 2016-08-11 22:16 ` Al Viro 2016-08-11 22:30 ` [LKP] [lkp] " Linus Torvalds 2016-08-11 22:30 ` Linus Torvalds 2016-08-11 21:16 ` [LKP] [lkp] " Huang, Ying 2016-08-11 21:16 ` Huang, Ying 2016-08-11 21:40 ` [LKP] [lkp] " Linus Torvalds 2016-08-11 21:40 ` Linus Torvalds 2016-08-11 22:08 ` [LKP] [lkp] " Christoph Hellwig 2016-08-11 22:08 ` Christoph Hellwig 2016-08-12 0:54 ` [LKP] [lkp] " Dave Chinner 2016-08-12 0:54 ` Dave Chinner 2016-08-12 2:23 ` [LKP] [lkp] " Dave Chinner 2016-08-12 2:23 ` Dave Chinner 2016-08-12 2:32 ` [LKP] [lkp] " Linus Torvalds 2016-08-12 2:32 ` Linus Torvalds 2016-08-12 2:52 ` [LKP] [lkp] " Christoph Hellwig 2016-08-12 2:52 ` Christoph Hellwig 2016-08-12 3:20 ` [LKP] [lkp] " Linus Torvalds 2016-08-12 3:20 ` Linus Torvalds 2016-08-12 4:16 ` [LKP] [lkp] " Dave Chinner 2016-08-12 4:16 ` Dave Chinner 2016-08-12 5:02 ` [LKP] [lkp] " Linus Torvalds 2016-08-12 5:02 ` Linus Torvalds 2016-08-12 6:04 ` [LKP] [lkp] " Dave Chinner 2016-08-12 6:04 ` Dave Chinner 2016-08-12 6:29 ` [LKP] [lkp] " Ye Xiaolong 2016-08-12 6:29 ` Ye Xiaolong 2016-08-12 8:51 ` [LKP] [lkp] " Ye Xiaolong 2016-08-12 8:51 ` Ye Xiaolong 2016-08-12 10:02 ` [LKP] [lkp] " Dave Chinner 2016-08-12 10:02 ` Dave Chinner 2016-08-12 10:43 ` Fengguang Wu 2016-08-12 10:43 ` Fengguang Wu 2016-08-13 0:30 ` [LKP] [lkp] " Christoph Hellwig 2016-08-13 0:30 ` Christoph Hellwig 2016-08-13 21:48 ` [LKP] [lkp] " Christoph Hellwig 2016-08-13 21:48 ` Christoph Hellwig 2016-08-13 22:07 ` [LKP] [lkp] " Fengguang Wu 2016-08-13 22:07 ` Fengguang Wu 2016-08-13 22:15 ` [LKP] [lkp] " Christoph Hellwig 2016-08-13 22:15 ` Christoph Hellwig 2016-08-13 22:51 ` [LKP] [lkp] " Fengguang Wu 2016-08-13 22:51 ` Fengguang Wu 2016-08-14 14:50 ` [LKP] [lkp] " Fengguang Wu 2016-08-14 14:50 ` Fengguang Wu 2016-08-14 16:17 ` [LKP] [lkp] " Christoph Hellwig 2016-08-14 16:17 ` Christoph Hellwig 2016-08-14 23:46 ` [LKP] [lkp] " Dave Chinner 2016-08-14 23:46 ` Dave Chinner 2016-08-14 23:57 ` [LKP] [lkp] " Fengguang Wu 2016-08-14 23:57 ` Fengguang Wu 2016-08-15 14:14 ` [LKP] [lkp] " Fengguang Wu 2016-08-15 14:14 ` Fengguang Wu 2016-08-15 21:22 ` [LKP] [lkp] " Dave Chinner 2016-08-15 21:22 ` Dave Chinner 2016-08-16 12:20 ` [LKP] [lkp] " Fengguang Wu 2016-08-16 12:20 ` Fengguang Wu 2016-08-15 20:30 ` [LKP] [lkp] " Huang, Ying 2016-08-15 20:30 ` Huang, Ying 2016-08-22 22:09 ` [LKP] [lkp] " Huang, Ying 2016-08-22 22:09 ` Huang, Ying 2016-09-26 6:25 ` [LKP] [lkp] " Huang, Ying 2016-09-26 6:25 ` Huang, Ying 2016-09-26 14:55 ` [LKP] [lkp] " Christoph Hellwig 2016-09-26 14:55 ` Christoph Hellwig 2016-09-27 0:52 ` [LKP] [lkp] " Huang, Ying 2016-09-27 0:52 ` Huang, Ying 2016-08-16 13:25 ` [LKP] [lkp] " Fengguang Wu 2016-08-16 13:25 ` Fengguang Wu 2016-08-13 23:32 ` [LKP] [lkp] " Dave Chinner 2016-08-13 23:32 ` Dave Chinner 2016-08-12 2:27 ` [LKP] [lkp] " Linus Torvalds 2016-08-12 2:27 ` Linus Torvalds 2016-08-12 3:56 ` [LKP] [lkp] " Dave Chinner 2016-08-12 3:56 ` Dave Chinner 2016-08-12 18:03 ` [LKP] [lkp] " Linus Torvalds 2016-08-12 18:03 ` Linus Torvalds 2016-08-13 23:58 ` [LKP] [lkp] " Fengguang Wu 2016-08-13 23:58 ` Fengguang Wu 2016-08-15 0:48 ` [LKP] [lkp] " Dave Chinner 2016-08-15 0:48 ` Dave Chinner 2016-08-15 1:37 ` [LKP] [lkp] " Linus Torvalds 2016-08-15 1:37 ` Linus Torvalds 2016-08-15 2:28 ` [LKP] [lkp] " Dave Chinner 2016-08-15 2:28 ` Dave Chinner 2016-08-15 2:53 ` [LKP] [lkp] " Linus Torvalds 2016-08-15 2:53 ` Linus Torvalds 2016-08-15 5:00 ` [LKP] [lkp] " Dave Chinner 2016-08-15 5:00 ` Dave Chinner [not found] ` <CA+55aFwva2Xffai+Eqv1Jn_NGryk3YJ2i5JoHOQnbQv6qVPAsw@mail.gmail.com> [not found] ` <CA+55aFy14nUnJQ_GdF=j8Fa9xiH70c6fY2G3q5HQ01+8z1z3qQ@mail.gmail.com> 2016-08-15 5:12 ` Linus Torvalds 2016-08-15 22:22 ` [LKP] [lkp] " Dave Chinner 2016-08-15 22:22 ` Dave Chinner 2016-08-15 22:42 ` [LKP] [lkp] " Dave Chinner 2016-08-15 22:42 ` Dave Chinner 2016-08-15 23:20 ` [LKP] [lkp] " Linus Torvalds 2016-08-15 23:20 ` Linus Torvalds 2016-08-15 23:48 ` [LKP] [lkp] " Linus Torvalds 2016-08-15 23:48 ` Linus Torvalds 2016-08-16 0:44 ` [LKP] [lkp] " Dave Chinner 2016-08-16 0:44 ` Dave Chinner 2016-08-16 15:05 ` [LKP] [lkp] " Mel Gorman 2016-08-16 15:05 ` Mel Gorman 2016-08-16 17:47 ` [LKP] [lkp] " Linus Torvalds 2016-08-16 17:47 ` Linus Torvalds 2016-08-17 15:48 ` [LKP] [lkp] " Michal Hocko 2016-08-17 15:48 ` Michal Hocko 2016-08-17 16:42 ` [LKP] [lkp] " Michal Hocko 2016-08-17 16:42 ` Michal Hocko 2016-08-17 15:49 ` [LKP] [lkp] " Mel Gorman 2016-08-17 15:49 ` Mel Gorman 2016-08-18 0:45 ` [LKP] [lkp] " Mel Gorman 2016-08-18 0:45 ` Mel Gorman 2016-08-18 7:11 ` [LKP] [lkp] " Dave Chinner 2016-08-18 7:11 ` Dave Chinner 2016-08-18 13:24 ` Mel Gorman [this message] 2016-08-18 13:24 ` Mel Gorman 2016-08-18 17:55 ` [LKP] [lkp] " Linus Torvalds 2016-08-18 17:55 ` Linus Torvalds 2016-08-18 21:19 ` [LKP] [lkp] " Dave Chinner 2016-08-18 21:19 ` Dave Chinner 2016-08-18 22:25 ` [LKP] [lkp] " Linus Torvalds 2016-08-18 22:25 ` Linus Torvalds 2016-08-19 9:00 ` [LKP] [lkp] " Michal Hocko 2016-08-19 9:00 ` Michal Hocko 2016-08-19 10:49 ` [LKP] [lkp] " Mel Gorman 2016-08-19 10:49 ` Mel Gorman 2016-08-19 23:48 ` [LKP] [lkp] " Dave Chinner 2016-08-19 23:48 ` Dave Chinner 2016-08-20 1:08 ` [LKP] [lkp] " Linus Torvalds 2016-08-20 1:08 ` Linus Torvalds 2016-08-20 12:16 ` [LKP] [lkp] " Mel Gorman 2016-08-20 12:16 ` Mel Gorman 2016-08-19 15:08 ` [LKP] [lkp] " Mel Gorman 2016-08-19 15:08 ` Mel Gorman 2016-09-01 23:32 ` [LKP] [lkp] " Dave Chinner 2016-09-01 23:32 ` Dave Chinner 2016-09-06 15:37 ` [LKP] [lkp] " Mel Gorman 2016-09-06 15:37 ` Mel Gorman 2016-09-06 15:52 ` [LKP] [lkp] " Huang, Ying 2016-09-06 15:52 ` Huang, Ying 2016-08-24 15:40 ` [LKP] [lkp] " Huang, Ying 2016-08-24 15:40 ` Huang, Ying 2016-08-25 9:37 ` [LKP] [lkp] " Mel Gorman 2016-08-25 9:37 ` Mel Gorman 2016-08-18 2:44 ` [LKP] [lkp] " Dave Chinner 2016-08-18 2:44 ` Dave Chinner 2016-08-16 0:15 ` [LKP] [lkp] " Linus Torvalds 2016-08-16 0:15 ` Linus Torvalds 2016-08-16 0:38 ` [LKP] [lkp] " Dave Chinner 2016-08-16 0:38 ` Dave Chinner 2016-08-16 0:50 ` [LKP] [lkp] " Linus Torvalds 2016-08-16 0:50 ` Linus Torvalds 2016-08-16 0:19 ` [LKP] [lkp] " Dave Chinner 2016-08-16 0:19 ` Dave Chinner 2016-08-16 1:51 ` [LKP] [lkp] " Linus Torvalds 2016-08-16 1:51 ` Linus Torvalds 2016-08-16 22:02 ` [LKP] [lkp] " Dave Chinner 2016-08-16 22:02 ` Dave Chinner 2016-08-16 23:23 ` [LKP] [lkp] " Linus Torvalds 2016-08-16 23:23 ` Linus Torvalds 2016-08-15 23:01 ` [LKP] [lkp] " Linus Torvalds 2016-08-15 23:01 ` Linus Torvalds 2016-08-16 0:17 ` [LKP] [lkp] " Dave Chinner 2016-08-16 0:17 ` Dave Chinner 2016-08-16 0:45 ` [LKP] [lkp] " Linus Torvalds 2016-08-16 0:45 ` Linus Torvalds 2016-08-15 5:03 ` [LKP] [lkp] " Ingo Molnar 2016-08-15 5:03 ` Ingo Molnar 2016-08-17 16:24 ` [LKP] [lkp] " Peter Zijlstra 2016-08-17 16:24 ` Peter Zijlstra 2016-08-15 12:58 ` [LKP] [lkp] " Fengguang Wu 2016-08-15 12:58 ` Fengguang Wu 2016-08-11 1:16 ` [LKP] [lkp] " Dave Chinner 2016-08-11 1:16 ` Dave Chinner 2016-08-11 1:32 ` [LKP] [lkp] " Dave Chinner 2016-08-11 1:32 ` Dave Chinner 2016-08-11 2:36 ` [LKP] [lkp] " Ye Xiaolong 2016-08-11 2:36 ` Ye Xiaolong 2016-08-11 3:05 ` [LKP] [lkp] " Dave Chinner 2016-08-11 3:05 ` Dave Chinner 2016-08-12 1:26 ` [LKP] [lkp] " Dave Chinner 2016-08-12 1:26 ` Dave Chinner
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20160818132414.GK8119@techsingularity.net \ --to=mgorman@techsingularity.net \ --cc=akpm@linux-foundation.org \ --cc=david@fromorbit.com \ --cc=fengguang.wu@intel.com \ --cc=hannes@cmpxchg.org \ --cc=hch@lst.de \ --cc=kirill.shutemov@linux.intel.com \ --cc=linux-kernel@vger.kernel.org \ --cc=lkp@01.org \ --cc=mhocko@suse.cz \ --cc=minchan@kernel.org \ --cc=rpeterso@redhat.com \ --cc=tj@kernel.org \ --cc=torvalds@linux-foundation.org \ --cc=vbabka@suse.cz \ --cc=vdavydov@virtuozzo.com \ --cc=ying.huang@intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.