From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?iso-8859-1?Q?Carsten_Schiers?= Subject: Re: =?iso-8859-1?q?Load_increase_after_memory_upgrade_=28?= =?iso-8859-1?q?part2=29?= Date: Sun, 4 Dec 2011 13:09:28 +0100 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20111202152339.GA2687@andromeda.dapyr.net> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: =?iso-8859-1?Q?Konrad_Rzeszutek_Wilk?= , =?iso-8859-1?Q?Konrad_Rzeszutek_Wilk?= Cc: =?iso-8859-1?Q?xen-devel?= , =?iso-8859-1?Q?lersek=40redhat=2Ecom?= , =?iso-8859-1?Q?Ian_Campbell?= , =?iso-8859-1?Q?zhenzhong=2Eduan=40oracle=2Ecom?= List-Id: xen-devel@lists.xenproject.org Here with two cards enabled and creating a bit "work" by watching TV with o= ne oft hem: [ 23.842720] Starting SWIOTLB debug thread. [ 23.842750] swiotlb_start_thread: Go! [ 23.842838] xen_swiotlb_start_thread: Go! [ 28.841451] 0 [budget_av 0000:00:01.0] bounce: from:435596(slow:0)to:0 m= ap:658 unmap:0 sync:435596 [ 28.841592] SWIOTLB is 4% full [ 33.840147] 0 [budget_av 0000:00:01.0] bounce: from:127652(slow:0)to:0 m= ap:0 unmap:0 sync:127652 [ 33.840283] SWIOTLB is 4% full [ 33.844222] 0 budget_av 0000:00:01.0 alloc coherent: 8, free: 0 [ 38.840227] 0 [budget_av 0000:00:01.0] bounce: from:128310(slow:0)to:0 m= ap:0 unmap:0 sync:128310 [ 38.840361] SWIOTLB is 4% full [ 43.840182] 0 [budget_av 0000:00:01.0] bounce: from:128310(slow:0)to:0 m= ap:0 unmap:0 sync:128310 [ 43.840323] SWIOTLB is 4% full [ 48.840094] 0 [budget_av 0000:00:01.0] bounce: from:127652(slow:0)to:0 m= ap:0 unmap:0 sync:127652 [ 48.840154] SWIOTLB is 4% full [ 53.840160] 0 [budget_av 0000:00:01.0] bounce: from:119756(slow:0)to:0 m= ap:0 unmap:0 sync:119756 [ 53.840301] SWIOTLB is 4% full [ 58.840202] 0 [budget_av 0000:00:01.0] bounce: from:128310(slow:0)to:0 m= ap:0 unmap:0 sync:128310 [ 58.840339] SWIOTLB is 4% full [ 63.840626] 0 [budget_av 0000:00:01.0] bounce: from:128310(slow:0)to:0 m= ap:0 unmap:0 sync:128310 [ 63.840686] SWIOTLB is 4% full [ 68.840122] 0 [budget_av 0000:00:01.0] bounce: from:127323(slow:0)to:0 m= ap:0 unmap:0 sync:127323 [ 68.840180] SWIOTLB is 4% full [ 73.840647] 0 [budget_av 0000:00:01.0] bounce: from:211547(slow:0)to:0 m= ap:0 unmap:0 sync:211547 [ 73.840784] SWIOTLB is 4% full [ 78.840204] 0 [budget_av 0000:00:01.0] bounce: from:255962(slow:0)to:0 m= ap:0 unmap:0 sync:255962 [ 78.840344] SWIOTLB is 4% full [ 83.840114] 0 [budget_av 0000:00:01.0] bounce: from:255304(slow:0)to:0 m= ap:0 unmap:0 sync:255304 [ 83.840178] SWIOTLB is 4% full [ 88.840158] 0 [budget_av 0000:00:01.0] bounce: from:256620(slow:0)to:0 m= ap:0 unmap:0 sync:256620 [ 88.840302] SWIOTLB is 4% full [ 93.840185] 0 [budget_av 0000:00:00.0] bounce: from:250040(slow:0)to:0 m= ap:0 unmap:0 sync:250040 [ 93.840319] SWIOTLB is 4% full [ 98.840181] 0 [budget_av 0000:00:00.0] bounce: from:255962(slow:0)to:0 m= ap:0 unmap:0 sync:255962 [ 98.841563] SWIOTLB is 4% full [ 103.841221] 0 [budget_av 0000:00:00.0] bounce: from:255962(slow:0)to:0 m= ap:0 unmap:0 sync:255962 [ 103.841361] SWIOTLB is 4% full [ 108.840247] 0 [budget_av 0000:00:00.0] bounce: from:255962(slow:0)to:0 m= ap:0 unmap:0 sync:255962 [ 108.840389] SWIOTLB is 4% full [ 113.840157] 0 [budget_av 0000:00:00.0] bounce: from:261555(slow:0)to:0 m= ap:0 unmap:0 sync:261555 [ 113.840298] SWIOTLB is 4% full [ 118.840119] 0 [budget_av 0000:00:00.0] bounce: from:295442(slow:0)to:0 m= ap:0 unmap:0 sync:295442 [ 118.840259] SWIOTLB is 4% full [ 123.841025] 0 [budget_av 0000:00:00.0] bounce: from:295113(slow:0)to:0 m= ap:0 unmap:0 sync:295113 [ 123.841164] SWIOTLB is 4% full [ 128.840175] 0 [budget_av 0000:00:00.0] bounce: from:294784(slow:0)to:0 m= ap:0 unmap:0 sync:294784 [ 128.840310] SWIOTLB is 4% full [ 133.840194] 0 [budget_av 0000:00:00.0] bounce: from:293797(slow:0)to:0 m= ap:0 unmap:0 sync:293797 [ 133.840330] SWIOTLB is 4% full [ 138.840498] 0 [budget_av 0000:00:00.0] bounce: from:341502(slow:0)to:0 m= ap:0 unmap:0 sync:341502 [ 138.840637] SWIOTLB is 4% full [ 143.840173] 0 [budget_av 0000:00:00.0] bounce: from:341502(slow:0)to:0 m= ap:0 unmap:0 sync:341502 [ 143.840313] SWIOTLB is 4% full [ 148.840215] 0 [budget_av 0000:00:00.0] bounce: from:341831(slow:0)to:0 m= ap:0 unmap:0 sync:341831 [ 148.840355] SWIOTLB is 4% full [ 153.840205] 0 [budget_av 0000:00:01.0] bounce: from:329658(slow:0)to:0 m= ap:0 unmap:0 sync:329658 [ 153.840341] SWIOTLB is 4% full [ 158.840137] 0 [budget_av 0000:00:00.0] bounce: from:342160(slow:0)to:0 m= ap:0 unmap:0 sync:342160 [ 158.840277] SWIOTLB is 4% full [ 163.841288] 0 [budget_av 0000:00:00.0] bounce: from:341502(slow:0)to:0 m= ap:0 unmap:0 sync:341502 [ 163.841424] SWIOTLB is 4% full [ 168.840198] 0 [budget_av 0000:00:00.0] bounce: from:341502(slow:0)to:0 m= ap:0 unmap:0 sync:341502 [ 168.840339] SWIOTLB is 4% full [ 173.840167] 0 [budget_av 0000:00:00.0] bounce: from:341502(slow:0)to:0 m= ap:0 unmap:0 sync:341502 [ 173.840304] SWIOTLB is 4% full [ 178.840184] 0 [budget_av 0000:00:00.0] bounce: from:328013(slow:0)to:0 m= ap:0 unmap:0 sync:328013 [ 178.840324] SWIOTLB is 4% full [ 183.840129] 0 [budget_av 0000:00:00.0] bounce: from:341831(slow:0)to:0 m= ap:0 unmap:0 sync:341831 [ 183.840269] SWIOTLB is 4% full [ 188.840123] 0 [budget_av 0000:00:01.0] bounce: from:340515(slow:0)to:0 m= ap:0 unmap:0 sync:340515 [ 188.841647] SWIOTLB is 4% full [ 193.840192] 0 [budget_av 0000:00:00.0] bounce: from:338541(slow:0)to:0 m= ap:0 unmap:0 sync:338541 [ 193.840329] SWIOTLB is 4% full [ 198.840148] 0 [budget_av 0000:00:01.0] bounce: from:330316(slow:0)to:0 m= ap:0 unmap:0 sync:330316 [ 198.840230] SWIOTLB is 4% full [ 203.840860] 0 [budget_av 0000:00:00.0] bounce: from:341831(slow:0)to:0 m= ap:0 unmap:0 sync:341831 [ 203.841000] SWIOTLB is 4% full [ 208.840562] 0 [budget_av 0000:00:01.0] bounce: from:337883(slow:0)to:0 m= ap:0 unmap:0 sync:337883 [ 208.840698] SWIOTLB is 4% full [ 213.840171] 0 [budget_av 0000:00:00.0] bounce: from:341502(slow:0)to:0 m= ap:0 unmap:0 sync:341502 [ 213.840311] SWIOTLB is 4% full [ 218.840214] 0 [budget_av 0000:00:01.0] bounce: from:320117(slow:0)to:0 m= ap:0 unmap:0 sync:320117 [ 218.840354] SWIOTLB is 4% full [ 223.840238] 0 [budget_av 0000:00:01.0] bounce: from:299390(slow:0)to:0 m= ap:0 unmap:0 sync:299390 [ 223.840373] SWIOTLB is 4% full [ 228.841415] 0 [budget_av 0000:00:01.0] bounce: from:298732(slow:0)to:0 m= ap:0 unmap:0 sync:298732 [ 228.841560] SWIOTLB is 4% full [ 233.840705] 0 [budget_av 0000:00:00.0] bounce: from:299061(slow:0)to:0 m= ap:0 unmap:0 sync:299061 [ 233.840844] SWIOTLB is 4% full [ 238.840145] 0 [budget_av 0000:00:01.0] bounce: from:293468(slow:0)to:0 m= ap:0 unmap:0 sync:293468 [ 238.840280] SWIOTLB is 4% full -----Urspr=FCngliche Nachricht----- Von: Konrad Rzeszutek Wilk [mailto:konrad@darnok.org] = Gesendet: Freitag, 2. Dezember 2011 16:24 An: Konrad Rzeszutek Wilk Cc: Ian Campbell; xen-devel; Carsten Schiers; zhenzhong.duan@oracle.com; le= rsek@redhat.com Betreff: Re: [Xen-devel] Load increase after memory upgrade (part2) > > > > > That is a puzzle. It should not. The code is very much the same -= both > > > > > use the generic SWIOTLB which has not changed for years. > > > > = > > > > The swiotlb-xen used by classic-xen kernels (which I assume is what > > > > Carsten means by "Xenified") isn't exactly the same as the stuff in > > > > mainline Linux, it's been heavily refactored for one thing. It's not > > > > impossible that mainline is bouncing something it doesn't really ne= ed > > > > to. > > > = > > > The usage, at least with 'pci_alloc_coherent' is that there is no bou= ncing > > > being done. The alloc_coherent will allocate a nice page, underneath = the 4GB > > > mark and give it to the driver. The driver can use it as it wishes an= d there > > > is no need to bounce buffer. > > = > > Oh, I didn't realise dma_alloc_coherent was part of swiotlb now. Only a > > subset of swiotlb is in use then, all the bouncing stuff _should_ be > > idle/unused -- but has that been confirmed? > = > Nope. I hope that the diagnostic patch I have in mind will prove/disprove= that. > Now I just need to find a moment to write it :-) Done! Carsten, can you please patch your kernel with this hacky patch and when you have booted the new kernel, just do modprobe dump_swiotlb it should give an idea of how many bounces are happening, coherent allocations, syncs, and so on.. along with the last driver that did those operations.