linux-scsi.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
       [not found] <307581a490b610c3025ee80f79a465a89d68ed19.camel@unipv.it>
@ 2019-08-20 17:13 ` Alan Stern
  2019-08-23 10:39   ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Alan Stern @ 2019-08-20 17:13 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH

On Mon, 19 Aug 2019, Andrea Vai wrote:

> Hi Alan,
>   I attach the two traces, collected as follows:
> 
> - start the trace;
> - wait 10 seconds;
> - plug the drive;
> - wait 5 seconds;
> - mount the drive;
> - wait 5 seconds;
> - copy a 500 byte file;
> - wait 5 seconds;
> - unmount the drive;
> - wait 5 seconds;
> - stop the trace.

Still no noticeable differences between the two traces.  They both 
include a 1.2 second delay shortly after the writing starts, and the 
initialization sequences are the same.

I really don't know where to look for this.  The only thing I can think
of at this point is to repeat this test, but using a file large enough
for the difference in writing speed to show up plainly.

By the way, it would be best to run the tests with the smallest
possible number of other USB devices plugged in.  None at all, if you
can arrange it.

Alan Stern


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-08-20 17:13 ` Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Alan Stern
@ 2019-08-23 10:39   ` Andrea Vai
  2019-08-23 20:42     ` Alan Stern
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-08-23 10:39 UTC (permalink / raw)
  To: Alan Stern
  Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH

[-- Attachment #1: Type: text/plain, Size: 4851 bytes --]

Il giorno mar, 20/08/2019 alle 13.13 -0400, Alan Stern ha scritto:
> On Mon, 19 Aug 2019, Andrea Vai wrote:
> 
> > Hi Alan,
> >   I attach the two traces, collected as follows:
> > 
> > - start the trace;
> > - wait 10 seconds;
> > - plug the drive;
> > - wait 5 seconds;
> > - mount the drive;
> > - wait 5 seconds;
> > - copy a 500 byte file;
> > - wait 5 seconds;
> > - unmount the drive;
> > - wait 5 seconds;
> > - stop the trace.
> 
> Still no noticeable differences between the two traces.  They both 
> include a 1.2 second delay shortly after the writing starts, and
> the 
> initialization sequences are the same.
> 
> I really don't know where to look for this.  The only thing I can
> think
> of at this point is to repeat this test, but using a file large
> enough
> for the difference in writing speed to show up plainly.
> 
> By the way, it would be best to run the tests with the smallest
> possible number of other USB devices plugged in.  None at all, if
> you
> can arrange it.

Thanks, I went some steps further on this.
The following considerations all apply to the "bad" kernel.

Increasing the filesize lead me to find out that using a file sized
less than roughly 10MB the problem does not happen.

I found these results by making sets of 10 tries for each filesize,
using a filesize of 1kB, 10kB, 100kB, 1MB, 10MB, 100MB, 500MB (so, we
have 70 usbmon logs on these). If we define "fast" a copy that takes
(roughly(*)) no more time to complete than all the other tries in its
set, and "slow" elsewhere (=one or more tries in its set are
(sensibly(*)) faster), I noticed that in each set with a filesize of
10MB or more the behavior can be very different: sometimes the copy is
still "fast", sometimes is "slow". The frequency of the "slow" copies
increases with the filesize. Also, among the "slow" copies in a set,
the time can be very different.

Also, I found that if the file is not present on the target location
(i.e. the USB pendrive), the problem does not happen (I have ten
usbmon logs here, taken in the worst scenario (500MB filesize)).

Tell me which log(s) would you like me to send you: I can sum up here
all the sets of tries, and the time their copies took to complete (in
seconds):

1kB: 26, 27, 26, 26, 27, 26, 26, 27, 26, 27
10kB: 27, 27, 26, 26, 27, 26, 27, 26, 27, 27
100kB: 26, 26, 26, 27, 26, 26, 26, 27, 27, 27
1MB: 26, 27, 27, 27, 27, 27, 27, 27, 27, 26
10MB: 27, 31, 37, 27, 38, 27, 39, 27, 30, 28
100MB: 32, 32, 144, 32, 145, 32, 123, 32, 153, 123
500MB: 56, 1396, 747, 131, 795, 764, 292, 1021, 807, 516

Also, note that the first copy is always "fast", because each file was
initially not present on the pendrive. As said, I did one test of 10
tries by deleting the file on the pendrive before copying it again,
and the results are

500MB: 56, 56, 57, 57, 56, 56, 60, 25***, 55, 56 (***Note the "fake"
25s, doesn't matter because I forgot to plug the pendrive :-/ )

I have made a script to semi-automate all the tests I have done. I
attach the script here, so anyone interested could check it for any
mistake (remember I am not very skilled so I may have wrote buggy
code, done wrong assumptions, etc.). Please note that I decreased the
time between the trace start and the drive plugging from 10s to 5s
(simply to reduce the time needed to me to look at the countdown). Of
course I can do again the test(s) you need with a bigger amount of
$wait.

The script has been run with the command

# for k in {1..10}; do size=1000; ./test_usbmon $size && ping -a -c 5 8.8.8.8 ; done
(example for 1kB filesize)

or, in the set of "delete before copy",

# for k in {1..10}; do size=500000000; ./cancellaTestFile $size && ./test_usbmon $size && ping -a -c 5 8.8.8.8 ; done

The ping command is there just to have a sound alarm when finished.

I also attach the script to delete the file ("cancellaTestFile").

I took care to plug the pendrive exactly at the end of the countdown,
to keep the times in the logs more simple to detect and manage by you.

I have also logged all the terminal output log of the script.

Last note: I ran all the tests without any other USB device connected
but the pendrive (well, actually there is a card reader connected to
the internal USB connector, but on another bus. I didn't want to open
the case and disconnect it but of course I can do it if needed).
Thanks for pointing it out.

Thanks, and bye
Andrea

(*) as an example, on a set that shows the total elapsed time in
seconds being

26, 27, 27, 27, 27, 27, 27, 27, 27, 26

I have assumed all of the copies to be "fast", while in the set

32, 32, 144, 32, 145, 32, 123, 32, 153, 123

I have assumed 5 of the copies as "fast" (the ones that took 32
seconds) and the other "slow". Not going to deepen in some standard
deviation evaluation, etc., but if you'd like to I can provide some
more scientific detailed data :-)


[-- Attachment #2: test_usbmon --]
[-- Type: application/x-shellscript, Size: 1607 bytes --]

[-- Attachment #3: cancellaTestFile --]
[-- Type: application/x-shellscript, Size: 296 bytes --]

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-08-23 10:39   ` Andrea Vai
@ 2019-08-23 20:42     ` Alan Stern
  2019-08-26  6:09       ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Alan Stern @ 2019-08-23 20:42 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH

On Fri, 23 Aug 2019, Andrea Vai wrote:

> Il giorno mar, 20/08/2019 alle 13.13 -0400, Alan Stern ha scritto:
> > On Mon, 19 Aug 2019, Andrea Vai wrote:
> > 
> > > Hi Alan,
> > >   I attach the two traces, collected as follows:
> > > 
> > > - start the trace;
> > > - wait 10 seconds;
> > > - plug the drive;
> > > - wait 5 seconds;
> > > - mount the drive;
> > > - wait 5 seconds;
> > > - copy a 500 byte file;
> > > - wait 5 seconds;
> > > - unmount the drive;
> > > - wait 5 seconds;
> > > - stop the trace.
> > 
> > Still no noticeable differences between the two traces.  They both 
> > include a 1.2 second delay shortly after the writing starts, and
> > the 
> > initialization sequences are the same.
> > 
> > I really don't know where to look for this.  The only thing I can
> > think
> > of at this point is to repeat this test, but using a file large
> > enough
> > for the difference in writing speed to show up plainly.
> > 
> > By the way, it would be best to run the tests with the smallest
> > possible number of other USB devices plugged in.  None at all, if
> > you
> > can arrange it.
> 
> Thanks, I went some steps further on this.
> The following considerations all apply to the "bad" kernel.
> 
> Increasing the filesize lead me to find out that using a file sized
> less than roughly 10MB the problem does not happen.
> 
> I found these results by making sets of 10 tries for each filesize,
> using a filesize of 1kB, 10kB, 100kB, 1MB, 10MB, 100MB, 500MB (so, we
> have 70 usbmon logs on these). If we define "fast" a copy that takes
> (roughly(*)) no more time to complete than all the other tries in its
> set, and "slow" elsewhere (=one or more tries in its set are
> (sensibly(*)) faster), I noticed that in each set with a filesize of
> 10MB or more the behavior can be very different: sometimes the copy is
> still "fast", sometimes is "slow". The frequency of the "slow" copies
> increases with the filesize. Also, among the "slow" copies in a set,
> the time can be very different.
> 
> Also, I found that if the file is not present on the target location
> (i.e. the USB pendrive), the problem does not happen (I have ten
> usbmon logs here, taken in the worst scenario (500MB filesize)).
> 
> Tell me which log(s) would you like me to send you: I can sum up here
> all the sets of tries, and the time their copies took to complete (in
> seconds):
> 
> 1kB: 26, 27, 26, 26, 27, 26, 26, 27, 26, 27
> 10kB: 27, 27, 26, 26, 27, 26, 27, 26, 27, 27
> 100kB: 26, 26, 26, 27, 26, 26, 26, 27, 27, 27
> 1MB: 26, 27, 27, 27, 27, 27, 27, 27, 27, 26
> 10MB: 27, 31, 37, 27, 38, 27, 39, 27, 30, 28
> 100MB: 32, 32, 144, 32, 145, 32, 123, 32, 153, 123
> 500MB: 56, 1396, 747, 131, 795, 764, 292, 1021, 807, 516
> 
> Also, note that the first copy is always "fast", because each file was
> initially not present on the pendrive. As said, I did one test of 10
> tries by deleting the file on the pendrive before copying it again,
> and the results are
> 
> 500MB: 56, 56, 57, 57, 56, 56, 60, 25***, 55, 56 (***Note the "fake"
> 25s, doesn't matter because I forgot to plug the pendrive :-/ )
> 
> I have made a script to semi-automate all the tests I have done. I
> attach the script here, so anyone interested could check it for any
> mistake (remember I am not very skilled so I may have wrote buggy
> code, done wrong assumptions, etc.). Please note that I decreased the
> time between the trace start and the drive plugging from 10s to 5s
> (simply to reduce the time needed to me to look at the countdown). Of
> course I can do again the test(s) you need with a bigger amount of
> $wait.
> 
> The script has been run with the command
> 
> # for k in {1..10}; do size=1000; ./test_usbmon $size && ping -a -c 5 8.8.8.8 ; done
> (example for 1kB filesize)
> 
> or, in the set of "delete before copy",
> 
> # for k in {1..10}; do size=500000000; ./cancellaTestFile $size && ./test_usbmon $size && ping -a -c 5 8.8.8.8 ; done
> 
> The ping command is there just to have a sound alarm when finished.
> 
> I also attach the script to delete the file ("cancellaTestFile").
> 
> I took care to plug the pendrive exactly at the end of the countdown,
> to keep the times in the logs more simple to detect and manage by you.
> 
> I have also logged all the terminal output log of the script.
> 
> Last note: I ran all the tests without any other USB device connected
> but the pendrive (well, actually there is a card reader connected to
> the internal USB connector, but on another bus. I didn't want to open
> the case and disconnect it but of course I can do it if needed).
> Thanks for pointing it out.
> 
> Thanks, and bye
> Andrea
> 
> (*) as an example, on a set that shows the total elapsed time in
> seconds being
> 
> 26, 27, 27, 27, 27, 27, 27, 27, 27, 26
> 
> I have assumed all of the copies to be "fast", while in the set
> 
> 32, 32, 144, 32, 145, 32, 123, 32, 153, 123
> 
> I have assumed 5 of the copies as "fast" (the ones that took 32
> seconds) and the other "slow". Not going to deepen in some standard
> deviation evaluation, etc., but if you'd like to I can provide some
> more scientific detailed data :-)

Wow, that sounds like a lot of work.

Let's start with the 39-second run for the 10-MB file.  If you can put 
the trace files on a server somewhere, available for downloading, that 
would avoid sending a lot of uninteresting data to the mailing list.

Odd that the delays never occur when you're writing a new file.  (If
nothing else, that gives you a way to work around the problem!)  It's
hard to say what it means, though.  Maybe the flash drive doesn't like 
overwriting used blocks.

Alan Stern


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-08-23 20:42     ` Alan Stern
@ 2019-08-26  6:09       ` Andrea Vai
  2019-08-26 16:33         ` Alan Stern
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-08-26  6:09 UTC (permalink / raw)
  To: Alan Stern
  Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH

[Sorry, I had previously sent the message to the list but has been
rejected. Sorry for any duplicate]

Il giorno ven, 23/08/2019 alle 16.42 -0400, Alan Stern ha scritto:
> On Fri, 23 Aug 2019, Andrea Vai wrote:
> 
> > Il giorno mar, 20/08/2019 alle 13.13 -0400, Alan Stern ha scritto:
> > > On Mon, 19 Aug 2019, Andrea Vai wrote:
> > > 
> > > > Hi Alan,
> > > >   I attach the two traces, collected as follows:
> > > > 
> > > > - start the trace;
> > > > - wait 10 seconds;
> > > > - plug the drive;
> > > > - wait 5 seconds;
> > > > - mount the drive;
> > > > - wait 5 seconds;
> > > > - copy a 500 byte file;
> > > > - wait 5 seconds;
> > > > - unmount the drive;
> > > > - wait 5 seconds;
> > > > - stop the trace.
> > > 
> > > Still no noticeable differences between the two traces.  They
> both 
> > > include a 1.2 second delay shortly after the writing starts, and
> > > the 
> > > initialization sequences are the same.
> > > 
> > > I really don't know where to look for this.  The only thing I
> can
> > > think
> > > of at this point is to repeat this test, but using a file large
> > > enough
> > > for the difference in writing speed to show up plainly.
> > > 
> > > By the way, it would be best to run the tests with the smallest
> > > possible number of other USB devices plugged in.  None at all,
> if
> > > you
> > > can arrange it.
> > 
> > Thanks, I went some steps further on this.
> > The following considerations all apply to the "bad" kernel.
> > 
> > Increasing the filesize lead me to find out that using a file
> sized
> > less than roughly 10MB the problem does not happen.
> > 
> > I found these results by making sets of 10 tries for each
> filesize,
> > using a filesize of 1kB, 10kB, 100kB, 1MB, 10MB, 100MB, 500MB (so,
> we
> > have 70 usbmon logs on these). If we define "fast" a copy that
> takes
> > (roughly(*)) no more time to complete than all the other tries in
> its
> > set, and "slow" elsewhere (=one or more tries in its set are
> > (sensibly(*)) faster), I noticed that in each set with a filesize
> of
> > 10MB or more the behavior can be very different: sometimes the
> copy is
> > still "fast", sometimes is "slow". The frequency of the "slow"
> copies
> > increases with the filesize. Also, among the "slow" copies in a
> set,
> > the time can be very different.
> > 
> > Also, I found that if the file is not present on the target
> location
> > (i.e. the USB pendrive), the problem does not happen (I have ten
> > usbmon logs here, taken in the worst scenario (500MB filesize)).
> > 
> > Tell me which log(s) would you like me to send you: I can sum up
> here
> > all the sets of tries, and the time their copies took to complete
> (in
> > seconds):
> > 
> > 1kB: 26, 27, 26, 26, 27, 26, 26, 27, 26, 27
> > 10kB: 27, 27, 26, 26, 27, 26, 27, 26, 27, 27
> > 100kB: 26, 26, 26, 27, 26, 26, 26, 27, 27, 27
> > 1MB: 26, 27, 27, 27, 27, 27, 27, 27, 27, 26
> > 10MB: 27, 31, 37, 27, 38, 27, 39, 27, 30, 28
> > 100MB: 32, 32, 144, 32, 145, 32, 123, 32, 153, 123
> > 500MB: 56, 1396, 747, 131, 795, 764, 292, 1021, 807, 516
> > 
> > Also, note that the first copy is always "fast", because each file
> was
> > initially not present on the pendrive. As said, I did one test of
> 10
> > tries by deleting the file on the pendrive before copying it
> again,
> > and the results are
> > 
> > 500MB: 56, 56, 57, 57, 56, 56, 60, 25***, 55, 56 (***Note the
> "fake"
> > 25s, doesn't matter because I forgot to plug the pendrive :-/ )
> > 
> > I have made a script to semi-automate all the tests I have done. I
> > attach the script here, so anyone interested could check it for
> any
> > mistake (remember I am not very skilled so I may have wrote buggy
> > code, done wrong assumptions, etc.). Please note that I decreased
> the
> > time between the trace start and the drive plugging from 10s to 5s
> > (simply to reduce the time needed to me to look at the countdown).
> Of
> > course I can do again the test(s) you need with a bigger amount of
> > $wait.
> > 
> > The script has been run with the command
> > 
> > # for k in {1..10}; do size=1000; ./test_usbmon $size && ping -a
> -c 5 8.8.8.8 ; done
> > (example for 1kB filesize)
> > 
> > or, in the set of "delete before copy",
> > 
> > # for k in {1..10}; do size=500000000; ./cancellaTestFile $size &&
> ./test_usbmon $size && ping -a -c 5 8.8.8.8 ; done
> > 
> > The ping command is there just to have a sound alarm when
> finished.
> > 
> > I also attach the script to delete the file ("cancellaTestFile").
> > 
> > I took care to plug the pendrive exactly at the end of the
> countdown,
> > to keep the times in the logs more simple to detect and manage by
> you.
> > 
> > I have also logged all the terminal output log of the script.
> > 
> > Last note: I ran all the tests without any other USB device
> connected
> > but the pendrive (well, actually there is a card reader connected
> to
> > the internal USB connector, but on another bus. I didn't want to
> open
> > the case and disconnect it but of course I can do it if needed).
> > Thanks for pointing it out.
> > 
> > Thanks, and bye
> > Andrea
> > 
> > (*) as an example, on a set that shows the total elapsed time in
> > seconds being
> > 
> > 26, 27, 27, 27, 27, 27, 27, 27, 27, 26
> > 
> > I have assumed all of the copies to be "fast", while in the set
> > 
> > 32, 32, 144, 32, 145, 32, 123, 32, 153, 123
> > 
> > I have assumed 5 of the copies as "fast" (the ones that took 32
> > seconds) and the other "slow". Not going to deepen in some
> standard
> > deviation evaluation, etc., but if you'd like to I can provide
> some
> > more scientific detailed data :-)
> 
> Wow, that sounds like a lot of work.

just a drop in the ocean, compared to yours :-)

> Let's start with the 39-second run for the 10-MB file.  If you can
> put 
> the trace files on a server somewhere, available for downloading,
> that 
> would avoid sending a lot of uninteresting data to the mailing list.

ok, so you can grab them at

http://fisica.unipv.it/transfer/usbmon_logs.zip

(they will be automatically removed from there in a couple of weeks).

For each size there is a .txt file (which contains the terminal
output) and 10 bad.mon.out_.... trace files. The file suffix "NonCanc"
means there has not been file deletion before copy; while "Canc" means
the opposite.

Each trace file name is identified by a timestamp that is also
referenced inside the txt, so if you want to get i.e. the 39-sec trial
for the 10MB filesize you have to open the ...10MB....txt, search for
the 39 seconds total time string ("Dopo stop trace: 39"), look at the
beginning of that trial, a dozen rows before, take note of the
timestamp, and open the corresponding bad.mon.out file (of course, if
there are more trials with the same time, you have to identify it by
counting its position (7th in the example above)).

To make it more simple:

$ seconds=39; size=10MB; grep -B14 "Dopo stop trace: $seconds" log_10trials_"$size"_NonCanc.txt

should show you more straightly the part(s) you need.

> Odd that the delays never occur when you're writing a new file.  (If
> nothing else, that gives you a way to work around the problem!) 

Thank you, didn't realize that :-) I will try it.

Thanks, and bye
Andrea


-- 
Andrea Vai
University of Pavia
Department of Physics
Via Bassi, 6
27100 Pavia PV
Tel. +39 0382 987489
Mob. +39 328 3354086

http://fisica.unipv.it
http://www.andreavai.it


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-08-26  6:09       ` Andrea Vai
@ 2019-08-26 16:33         ` Alan Stern
  2019-09-18 15:25           ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Alan Stern @ 2019-08-26 16:33 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH

On Mon, 26 Aug 2019, Andrea Vai wrote:

> ok, so you can grab them at
> 
> http://fisica.unipv.it/transfer/usbmon_logs.zip
> 
> (they will be automatically removed from there in a couple of weeks).
> 
> For each size there is a .txt file (which contains the terminal
> output) and 10 bad.mon.out_.... trace files. The file suffix "NonCanc"
> means there has not been file deletion before copy; while "Canc" means
> the opposite.
> 
> Each trace file name is identified by a timestamp that is also
> referenced inside the txt, so if you want to get i.e. the 39-sec trial
> for the 10MB filesize you have to open the ...10MB....txt, search for
> the 39 seconds total time string ("Dopo stop trace: 39"), look at the
> beginning of that trial, a dozen rows before, take note of the
> timestamp, and open the corresponding bad.mon.out file (of course, if
> there are more trials with the same time, you have to identify it by
> counting its position (7th in the example above)).
> 
> To make it more simple:
> 
> $ seconds=39; size=10MB; grep -B14 "Dopo stop trace: $seconds" log_10trials_"$size"_NonCanc.txt
> 
> should show you more straightly the part(s) you need.
> 
> > Odd that the delays never occur when you're writing a new file.  (If
> > nothing else, that gives you a way to work around the problem!) 
> 
> Thank you, didn't realize that :-) I will try it.

In fact, even the traces where the file doesn't exist beforehand show 
some delays.  Just not as many delays as the traces where the file does 
exist.  And again, each delay is in the middle of a write command, not 
between commands.

I suppose changes to the upper software layers could affect which
blocks are assigned when a new file is written.  Perhaps one kernel
re-uses the same old blocks that had been previously occupied and the
other kernel allocates a completely new set of blocks.  That might
change the drive's behavior.  The quick way to tell is to record two
usbmon traces, one under the "good" kernel and one under the "bad"  
kernel, where each test involves writing over a file that already
exists (say, 50 MB) -- the same file for both tests.  The block numbers
will appear in the traces.

Also, I wonder if the changing the size of the data transfers would
make any difference.  This is easy to try; just write "64" to
/sys/block/sd?/queue/max_sectors_kb (where the ? is the appropriate
drive letter) after the drive is plugged in but before the test starts.

Alan Stern


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-08-26 16:33         ` Alan Stern
@ 2019-09-18 15:25           ` Andrea Vai
  2019-09-18 16:30             ` Alan Stern
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-09-18 15:25 UTC (permalink / raw)
  To: Alan Stern
  Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH

Il giorno lun 26 ago 2019 alle ore 18:33 Alan Stern <
stern@rowland.harvard.edu> ha scritto:
> [...]
> In fact, even the traces where the file doesn't exist beforehand
> show 
> some delays.  Just not as many delays as the traces where the file
> does 
> exist.  And again, each delay is in the middle of a write command,
> not 
> between commands.
> 
> I suppose changes to the upper software layers could affect which
> blocks are assigned when a new file is written.  Perhaps one kernel
> re-uses the same old blocks that had been previously occupied and
> the
> other kernel allocates a completely new set of blocks.  That might
> change the drive's behavior.  The quick way to tell is to record two
> usbmon traces, one under the "good" kernel and one under the "bad"  
> kernel, where each test involves writing over a file that already
> exists (say, 50 MB) -- the same file for both tests.  The block
> numbers
> will appear in the traces.

ok, I performed 10 tests for each kernel, so we have 20 traces.
 
> Also, I wonder if the changing the size of the data transfers would
> make any difference.  This is easy to try; just write "64" to
> /sys/block/sd?/queue/max_sectors_kb (where the ? is the appropriate
> drive letter) after the drive is plugged in but before the test
> starts.

ok, so I duplicated the tests above for the "64" case (it was
initially set as "120", if it is relevant to know), leading to 40 tests named as

bad.mon.out_50000000_64_TIMESTAMP
bad.mon.out_50000000_non64_TIMESTAMP
good.mon.out_50000000_64_TIMESTAMP
good.mon.out_50000000_non64_TIMESTAMP

where "64" denotes the ones done with that value in max_sectors_kb,
and "not64" the ones without it (as far as I can tell, it has been
always "120").

So, we have 40 traces total. Each set of 10 trials is identified by
a text file, which contains the output log of the test script (and the
timestamps), also available in the download zipfile.

Just to summarize here the times, they are respectively (number
expressed  in seconds):

BAD:
  Logs: log_10trials_50MB_BAD_NonCanc_64.txt,
log_10trials_50MB_BAD_NonCanc_non64.txt
  64: 34, 34, 35, 39, 37, 32, 42, 44, 43, 40
  not64: 61, 71, 59, 71, 62, 75, 62, 70, 62, 68
GOOD:
  Logs: log_10trials_50MB_GOOD_NonCanc_64.txt,
log_10trials_50MB_GOOD_NonCanc_non64.txt
  64: 34, 32, 35, 34, 35, 33, 34, 33, 33, 33
  not64: 32, 30, 32, 31, 31, 30, 32, 30, 32, 31

Finally, one note about the workaround proposed by Alan, "delete the
file before copying". My original problem occurred while using a
backup software (dar - see http://dar.linux.free.fr/). So, I tried now
to do the backup by deleting the existing file beforehand, and it
still takes a lot of time with bad kernel: a 900 file backup takes
~160sec with GOOD kernel, and >40min with BAD kernel. I also tried the
"64" tweak in the BAD kernel and it becomes ~300s. Then, I also tried
the "64" case with good kernel, and became ~140s. Detailed data:

GOOD (not "64): 155s, 151s
GOOD ("64"): 142s, 141s

BAD (not "64"): 47minutes, 43minutes
BAD ("64"): 315s, 288s, 268s, 239s, 302s

The command ran is:
$ SECONDS=0; rm /run/media/andrea/BAK_ANDVAI/aero.1.dar && dar -c /run/media/andrea/BAK_ANDVAI/aero -R /home/andrea/Musica/MP3/Aerosmith && umount /run/media/andrea/BAK_ANDVAI; echo "Ci ho messo: $SECONDS secondi."

Speculations:
- It seems the "64" value plays a role, more evident on "bad" kernels
(~halves the time) and less (but still existing?) on "good" kernels;
- dar with the bad kernel, with the "delete beforehand" action, is
still an order of magnitude slower than with the good kernel (so, it
behaves the same way as in the "overwrite" case). Maybe it depends on
the way dar itself writes data... I don't know if you can understand
it, or we should ask for a light to the dar developer(s) about it.

You can grab the traces at

http://fisica.unipv.it/transfer/usbmon_logs_2.zip

Thanks, and bye
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-18 15:25           ` Andrea Vai
@ 2019-09-18 16:30             ` Alan Stern
  2019-09-19  7:33               ` Andrea Vai
  2019-09-19  8:26               ` Damien Le Moal
  0 siblings, 2 replies; 104+ messages in thread
From: Alan Stern @ 2019-09-18 16:30 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Johannes Thumshirn, Jens Axboe, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH

On Wed, 18 Sep 2019, Andrea Vai wrote:

> > Also, I wonder if the changing the size of the data transfers would
> > make any difference.  This is easy to try; just write "64" to
> > /sys/block/sd?/queue/max_sectors_kb (where the ? is the appropriate
> > drive letter) after the drive is plugged in but before the test
> > starts.
> 
> ok, so I duplicated the tests above for the "64" case (it was
> initially set as "120", if it is relevant to know), leading to 40 tests named as
> 
> bad.mon.out_50000000_64_TIMESTAMP
> bad.mon.out_50000000_non64_TIMESTAMP
> good.mon.out_50000000_64_TIMESTAMP
> good.mon.out_50000000_non64_TIMESTAMP
> 
> where "64" denotes the ones done with that value in max_sectors_kb,
> and "not64" the ones without it (as far as I can tell, it has been
> always "120").
> 
> So, we have 40 traces total. Each set of 10 trials is identified by
> a text file, which contains the output log of the test script (and the
> timestamps), also available in the download zipfile.
> 
> Just to summarize here the times, they are respectively (number
> expressed  in seconds):
> 
> BAD:
>   Logs: log_10trials_50MB_BAD_NonCanc_64.txt,
> log_10trials_50MB_BAD_NonCanc_non64.txt
>   64: 34, 34, 35, 39, 37, 32, 42, 44, 43, 40
>   not64: 61, 71, 59, 71, 62, 75, 62, 70, 62, 68
> GOOD:
>   Logs: log_10trials_50MB_GOOD_NonCanc_64.txt,
> log_10trials_50MB_GOOD_NonCanc_non64.txt
>   64: 34, 32, 35, 34, 35, 33, 34, 33, 33, 33
>   not64: 32, 30, 32, 31, 31, 30, 32, 30, 32, 31

The improvement from using "64" with the bad kernel is quite large.  
That alone would be a big help for you.

However, I did see what appears to be a very significant difference 
between the bad and good kernel traces.  It has to do with the order in 
which the blocks are accessed.

Here is an extract from one of the bad traces.  I have erased all the 
information except for the columns containing the block numbers to be 
written:

00019628 00
00019667 00
00019628 80
00019667 80
00019629 00
00019668 00
00019629 80
00019668 80

Here is the equivalent portion from one of the good traces:

00019628 00
00019628 80
00019629 00
00019629 80
0001962a 00
0001962a 80
0001962b 00
0001962b 80

Notice that under the good kernel, the block numbers increase
monotonically in a single sequence.  But under the bad kernel, the
block numbers are not monotonic -- it looks like there are two separate
threads each with its own strictly increasing sequence.

This is exactly the sort of difference one might expect to see from
the commit f664a3cc17b7 ("scsi: kill off the legacy IO path") you
identified as the cause of the problem.  With multiqueue I/O, it's not 
surprising to see multiple sequences of block numbers.

Add it's not at all surprising that a consumer-grade USB storage device 
might do a much worse job of handling non-sequential writes than 
sequential ones.

Which leads to a simple question for the SCSI or block-layer 
maintainers:  Is there a sysfs setting Andrea can tweak which will 
effectively restrict a particular disk device down to a single I/O
queue, forcing sequential access?

Alan Stern


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-18 16:30             ` Alan Stern
@ 2019-09-19  7:33               ` Andrea Vai
  2019-09-19 17:54                 ` Alan Stern
  2019-09-19  8:26               ` Damien Le Moal
  1 sibling, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-09-19  7:33 UTC (permalink / raw)
  To: Alan Stern
  Cc: Johannes Thumshirn, Jens Axboe, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH

Il giorno mer, 18/09/2019 alle 12.30 -0400, Alan Stern ha scritto:
> On Wed, 18 Sep 2019, Andrea Vai wrote:
> [...]
> > BAD:
> >   Logs: log_10trials_50MB_BAD_NonCanc_64.txt,
> > log_10trials_50MB_BAD_NonCanc_non64.txt
> >   64: 34, 34, 35, 39, 37, 32, 42, 44, 43, 40
> >   not64: 61, 71, 59, 71, 62, 75, 62, 70, 62, 68
> > GOOD:
> >   Logs: log_10trials_50MB_GOOD_NonCanc_64.txt,
> > log_10trials_50MB_GOOD_NonCanc_non64.txt
> >   64: 34, 32, 35, 34, 35, 33, 34, 33, 33, 33
> >   not64: 32, 30, 32, 31, 31, 30, 32, 30, 32, 31
> 
> The improvement from using "64" with the bad kernel is quite
> large.  
> That alone would be a big help for you.

Well, not so much, actually, because the backup would take twice the
time, that is quite annoying for me. But, apart from that, and from
the efforts of Alan and other people following this issue (thanks), I
would like to point out what I am not sure to have ever made clear
about my support request: I have understood that my problem is quite
specific, and don't want anyone to waste their time to help
specifically *me* (I can buy another media, use the "64" tweak, or
find any other workaround). But since we have identified the problem
as kernel-related, I am worried for other users, maybe new to linux,
that can have the same problem, and the evidence for them would be
that linux is extremely slow to copy file over some USB media. So,
among all the technical comments, I would like to make clear (if it's
not already clear) that in my opinion it would be important to solve
the problem without the need of user workarounds. Does it make sense?
Are we moving towards that goal?

BTW, another question: Alan refers to the slow media as a "consumer-
grade USB storage device". What could I do to identify and buy a "good
media"? Are there any features to look for?

Many thanks, and sorry if I ask anything obvious.
Bye,
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-18 16:30             ` Alan Stern
  2019-09-19  7:33               ` Andrea Vai
@ 2019-09-19  8:26               ` Damien Le Moal
  2019-09-19  8:55                 ` Ming Lei
  2019-09-19 14:01                 ` Alan Stern
  1 sibling, 2 replies; 104+ messages in thread
From: Damien Le Moal @ 2019-09-19  8:26 UTC (permalink / raw)
  To: Alan Stern, Andrea Vai
  Cc: Johannes Thumshirn, Jens Axboe, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg

On 2019/09/18 18:30, Alan Stern wrote:
> On Wed, 18 Sep 2019, Andrea Vai wrote:
> 
>>> Also, I wonder if the changing the size of the data transfers would
>>> make any difference.  This is easy to try; just write "64" to
>>> /sys/block/sd?/queue/max_sectors_kb (where the ? is the appropriate
>>> drive letter) after the drive is plugged in but before the test
>>> starts.
>>
>> ok, so I duplicated the tests above for the "64" case (it was
>> initially set as "120", if it is relevant to know), leading to 40 tests named as
>>
>> bad.mon.out_50000000_64_TIMESTAMP
>> bad.mon.out_50000000_non64_TIMESTAMP
>> good.mon.out_50000000_64_TIMESTAMP
>> good.mon.out_50000000_non64_TIMESTAMP
>>
>> where "64" denotes the ones done with that value in max_sectors_kb,
>> and "not64" the ones without it (as far as I can tell, it has been
>> always "120").
>>
>> So, we have 40 traces total. Each set of 10 trials is identified by
>> a text file, which contains the output log of the test script (and the
>> timestamps), also available in the download zipfile.
>>
>> Just to summarize here the times, they are respectively (number
>> expressed  in seconds):
>>
>> BAD:
>>   Logs: log_10trials_50MB_BAD_NonCanc_64.txt,
>> log_10trials_50MB_BAD_NonCanc_non64.txt
>>   64: 34, 34, 35, 39, 37, 32, 42, 44, 43, 40
>>   not64: 61, 71, 59, 71, 62, 75, 62, 70, 62, 68
>> GOOD:
>>   Logs: log_10trials_50MB_GOOD_NonCanc_64.txt,
>> log_10trials_50MB_GOOD_NonCanc_non64.txt
>>   64: 34, 32, 35, 34, 35, 33, 34, 33, 33, 33
>>   not64: 32, 30, 32, 31, 31, 30, 32, 30, 32, 31
> 
> The improvement from using "64" with the bad kernel is quite large.  
> That alone would be a big help for you.
> 
> However, I did see what appears to be a very significant difference 
> between the bad and good kernel traces.  It has to do with the order in 
> which the blocks are accessed.
> 
> Here is an extract from one of the bad traces.  I have erased all the 
> information except for the columns containing the block numbers to be 
> written:
> 
> 00019628 00
> 00019667 00
> 00019628 80
> 00019667 80
> 00019629 00
> 00019668 00
> 00019629 80
> 00019668 80
> 
> Here is the equivalent portion from one of the good traces:
> 
> 00019628 00
> 00019628 80
> 00019629 00
> 00019629 80
> 0001962a 00
> 0001962a 80
> 0001962b 00
> 0001962b 80
> 
> Notice that under the good kernel, the block numbers increase
> monotonically in a single sequence.  But under the bad kernel, the
> block numbers are not monotonic -- it looks like there are two separate
> threads each with its own strictly increasing sequence.
> 
> This is exactly the sort of difference one might expect to see from
> the commit f664a3cc17b7 ("scsi: kill off the legacy IO path") you
> identified as the cause of the problem.  With multiqueue I/O, it's not 
> surprising to see multiple sequences of block numbers.
> 
> Add it's not at all surprising that a consumer-grade USB storage device 
> might do a much worse job of handling non-sequential writes than 
> sequential ones.
> 
> Which leads to a simple question for the SCSI or block-layer 
> maintainers:  Is there a sysfs setting Andrea can tweak which will 
> effectively restrict a particular disk device down to a single I/O
> queue, forcing sequential access?

The scheduling inefficiency you are seeing may be coming from the fact that the
block layer does a direct issue of requests, bypassing the elevator, under some
conditions. One of these is sync requests on a multiqueue device. We hit this
problem on Zoned disks which can easily return an error for write requests
without the elevator throttling writes per zones (zone write locking). This
problem was discovered by Hans (on CC).

I discussed this with Hannes yesterday and we think we have a fix, but we'll
need to do a lot of testing as all block devices are potentially impacted by the
change, including stacked drivers (DM). Performance regression is scary with any
change in that area (see blk_mq_make_request() and use of
blk_mq_try_issue_directly() vs blk_mq_sched_insert_request()).


-- 
Damien Le Moal
Western Digital Research

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-19  8:26               ` Damien Le Moal
@ 2019-09-19  8:55                 ` Ming Lei
  2019-09-19  9:09                   ` Damien Le Moal
  2019-09-19 14:01                 ` Alan Stern
  1 sibling, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-09-19  8:55 UTC (permalink / raw)
  To: Damien Le Moal
  Cc: Alan Stern, Andrea Vai, Johannes Thumshirn, Jens Axboe, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg

On Thu, Sep 19, 2019 at 08:26:32AM +0000, Damien Le Moal wrote:
> On 2019/09/18 18:30, Alan Stern wrote:
> > On Wed, 18 Sep 2019, Andrea Vai wrote:
> > 
> >>> Also, I wonder if the changing the size of the data transfers would
> >>> make any difference.  This is easy to try; just write "64" to
> >>> /sys/block/sd?/queue/max_sectors_kb (where the ? is the appropriate
> >>> drive letter) after the drive is plugged in but before the test
> >>> starts.
> >>
> >> ok, so I duplicated the tests above for the "64" case (it was
> >> initially set as "120", if it is relevant to know), leading to 40 tests named as
> >>
> >> bad.mon.out_50000000_64_TIMESTAMP
> >> bad.mon.out_50000000_non64_TIMESTAMP
> >> good.mon.out_50000000_64_TIMESTAMP
> >> good.mon.out_50000000_non64_TIMESTAMP
> >>
> >> where "64" denotes the ones done with that value in max_sectors_kb,
> >> and "not64" the ones without it (as far as I can tell, it has been
> >> always "120").
> >>
> >> So, we have 40 traces total. Each set of 10 trials is identified by
> >> a text file, which contains the output log of the test script (and the
> >> timestamps), also available in the download zipfile.
> >>
> >> Just to summarize here the times, they are respectively (number
> >> expressed  in seconds):
> >>
> >> BAD:
> >>   Logs: log_10trials_50MB_BAD_NonCanc_64.txt,
> >> log_10trials_50MB_BAD_NonCanc_non64.txt
> >>   64: 34, 34, 35, 39, 37, 32, 42, 44, 43, 40
> >>   not64: 61, 71, 59, 71, 62, 75, 62, 70, 62, 68
> >> GOOD:
> >>   Logs: log_10trials_50MB_GOOD_NonCanc_64.txt,
> >> log_10trials_50MB_GOOD_NonCanc_non64.txt
> >>   64: 34, 32, 35, 34, 35, 33, 34, 33, 33, 33
> >>   not64: 32, 30, 32, 31, 31, 30, 32, 30, 32, 31
> > 
> > The improvement from using "64" with the bad kernel is quite large.  
> > That alone would be a big help for you.
> > 
> > However, I did see what appears to be a very significant difference 
> > between the bad and good kernel traces.  It has to do with the order in 
> > which the blocks are accessed.
> > 
> > Here is an extract from one of the bad traces.  I have erased all the 
> > information except for the columns containing the block numbers to be 
> > written:
> > 
> > 00019628 00
> > 00019667 00
> > 00019628 80
> > 00019667 80
> > 00019629 00
> > 00019668 00
> > 00019629 80
> > 00019668 80
> > 
> > Here is the equivalent portion from one of the good traces:
> > 
> > 00019628 00
> > 00019628 80
> > 00019629 00
> > 00019629 80
> > 0001962a 00
> > 0001962a 80
> > 0001962b 00
> > 0001962b 80
> > 
> > Notice that under the good kernel, the block numbers increase
> > monotonically in a single sequence.  But under the bad kernel, the
> > block numbers are not monotonic -- it looks like there are two separate
> > threads each with its own strictly increasing sequence.
> > 
> > This is exactly the sort of difference one might expect to see from
> > the commit f664a3cc17b7 ("scsi: kill off the legacy IO path") you
> > identified as the cause of the problem.  With multiqueue I/O, it's not 
> > surprising to see multiple sequences of block numbers.
> > 
> > Add it's not at all surprising that a consumer-grade USB storage device 
> > might do a much worse job of handling non-sequential writes than 
> > sequential ones.
> > 
> > Which leads to a simple question for the SCSI or block-layer 
> > maintainers:  Is there a sysfs setting Andrea can tweak which will 
> > effectively restrict a particular disk device down to a single I/O
> > queue, forcing sequential access?
> 
> The scheduling inefficiency you are seeing may be coming from the fact that the
> block layer does a direct issue of requests, bypassing the elevator, under some
> conditions. One of these is sync requests on a multiqueue device. We hit this
> problem on Zoned disks which can easily return an error for write requests
> without the elevator throttling writes per zones (zone write locking). This
> problem was discovered by Hans (on CC).
> 
> I discussed this with Hannes yesterday and we think we have a fix, but we'll
> need to do a lot of testing as all block devices are potentially impacted by the
> change, including stacked drivers (DM). Performance regression is scary with any
> change in that area (see blk_mq_make_request() and use of
> blk_mq_try_issue_directly() vs blk_mq_sched_insert_request()).

Not sure this one is same with yours, for USB, mq-deadline is used at
default, and direct issue won't be possible. Direct issue is only used
in case of none or underlying queues of DM multipath.

thanks, 
Ming

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-19  8:55                 ` Ming Lei
@ 2019-09-19  9:09                   ` Damien Le Moal
  2019-09-19  9:21                     ` Ming Lei
  0 siblings, 1 reply; 104+ messages in thread
From: Damien Le Moal @ 2019-09-19  9:09 UTC (permalink / raw)
  To: Ming Lei
  Cc: Alan Stern, Andrea Vai, Johannes Thumshirn, Jens Axboe, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg

On 2019/09/19 10:56, Ming Lei wrote:
> On Thu, Sep 19, 2019 at 08:26:32AM +0000, Damien Le Moal wrote:
>> On 2019/09/18 18:30, Alan Stern wrote:
>>> On Wed, 18 Sep 2019, Andrea Vai wrote:
>>>
>>>>> Also, I wonder if the changing the size of the data transfers would
>>>>> make any difference.  This is easy to try; just write "64" to
>>>>> /sys/block/sd?/queue/max_sectors_kb (where the ? is the appropriate
>>>>> drive letter) after the drive is plugged in but before the test
>>>>> starts.
>>>>
>>>> ok, so I duplicated the tests above for the "64" case (it was
>>>> initially set as "120", if it is relevant to know), leading to 40 tests named as
>>>>
>>>> bad.mon.out_50000000_64_TIMESTAMP
>>>> bad.mon.out_50000000_non64_TIMESTAMP
>>>> good.mon.out_50000000_64_TIMESTAMP
>>>> good.mon.out_50000000_non64_TIMESTAMP
>>>>
>>>> where "64" denotes the ones done with that value in max_sectors_kb,
>>>> and "not64" the ones without it (as far as I can tell, it has been
>>>> always "120").
>>>>
>>>> So, we have 40 traces total. Each set of 10 trials is identified by
>>>> a text file, which contains the output log of the test script (and the
>>>> timestamps), also available in the download zipfile.
>>>>
>>>> Just to summarize here the times, they are respectively (number
>>>> expressed  in seconds):
>>>>
>>>> BAD:
>>>>   Logs: log_10trials_50MB_BAD_NonCanc_64.txt,
>>>> log_10trials_50MB_BAD_NonCanc_non64.txt
>>>>   64: 34, 34, 35, 39, 37, 32, 42, 44, 43, 40
>>>>   not64: 61, 71, 59, 71, 62, 75, 62, 70, 62, 68
>>>> GOOD:
>>>>   Logs: log_10trials_50MB_GOOD_NonCanc_64.txt,
>>>> log_10trials_50MB_GOOD_NonCanc_non64.txt
>>>>   64: 34, 32, 35, 34, 35, 33, 34, 33, 33, 33
>>>>   not64: 32, 30, 32, 31, 31, 30, 32, 30, 32, 31
>>>
>>> The improvement from using "64" with the bad kernel is quite large.  
>>> That alone would be a big help for you.
>>>
>>> However, I did see what appears to be a very significant difference 
>>> between the bad and good kernel traces.  It has to do with the order in 
>>> which the blocks are accessed.
>>>
>>> Here is an extract from one of the bad traces.  I have erased all the 
>>> information except for the columns containing the block numbers to be 
>>> written:
>>>
>>> 00019628 00
>>> 00019667 00
>>> 00019628 80
>>> 00019667 80
>>> 00019629 00
>>> 00019668 00
>>> 00019629 80
>>> 00019668 80
>>>
>>> Here is the equivalent portion from one of the good traces:
>>>
>>> 00019628 00
>>> 00019628 80
>>> 00019629 00
>>> 00019629 80
>>> 0001962a 00
>>> 0001962a 80
>>> 0001962b 00
>>> 0001962b 80
>>>
>>> Notice that under the good kernel, the block numbers increase
>>> monotonically in a single sequence.  But under the bad kernel, the
>>> block numbers are not monotonic -- it looks like there are two separate
>>> threads each with its own strictly increasing sequence.
>>>
>>> This is exactly the sort of difference one might expect to see from
>>> the commit f664a3cc17b7 ("scsi: kill off the legacy IO path") you
>>> identified as the cause of the problem.  With multiqueue I/O, it's not 
>>> surprising to see multiple sequences of block numbers.
>>>
>>> Add it's not at all surprising that a consumer-grade USB storage device 
>>> might do a much worse job of handling non-sequential writes than 
>>> sequential ones.
>>>
>>> Which leads to a simple question for the SCSI or block-layer 
>>> maintainers:  Is there a sysfs setting Andrea can tweak which will 
>>> effectively restrict a particular disk device down to a single I/O
>>> queue, forcing sequential access?
>>
>> The scheduling inefficiency you are seeing may be coming from the fact that the
>> block layer does a direct issue of requests, bypassing the elevator, under some
>> conditions. One of these is sync requests on a multiqueue device. We hit this
>> problem on Zoned disks which can easily return an error for write requests
>> without the elevator throttling writes per zones (zone write locking). This
>> problem was discovered by Hans (on CC).
>>
>> I discussed this with Hannes yesterday and we think we have a fix, but we'll
>> need to do a lot of testing as all block devices are potentially impacted by the
>> change, including stacked drivers (DM). Performance regression is scary with any
>> change in that area (see blk_mq_make_request() and use of
>> blk_mq_try_issue_directly() vs blk_mq_sched_insert_request()).
> 
> Not sure this one is same with yours, for USB, mq-deadline is used at
> default, and direct issue won't be possible. Direct issue is only used
> in case of none or underlying queues of DM multipath.

For a multi-queue zoned disk, mq-deadline is also set, but we have observed
unaligned write IO errors for sync writes because of mq-deadline being bypassed
and as a result zones not being write-locked.

In blk_mq_make_request(), at the end, you have:

	} else if ((q->nr_hw_queues > 1 && is_sync) || (!q->elevator &&
			!data.hctx->dispatch_busy)) {
		blk_mq_try_issue_directly(data.hctx, rq, &cookie);
	} else {
		blk_mq_sched_insert_request(rq, false, true, true);
	}

Which I read as "for a sync req on a multi-queue device, direct issue",
regardless of the elevator being none or something else.

The correct test should probably be:

	} else if (!q->elevator &&
		   ((q->nr_hw_queues > 1 && is_sync) || 	
		     !data.hctx->dispatch_busy))) {
		blk_mq_try_issue_directly(data.hctx, rq, &cookie);
	} else {
		blk_mq_sched_insert_request(rq, false, true, true);
	}

That is, never bypass the elevator if one is set. Thoughts ?

-- 
Damien Le Moal
Western Digital Research

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-19  9:09                   ` Damien Le Moal
@ 2019-09-19  9:21                     ` Ming Lei
  0 siblings, 0 replies; 104+ messages in thread
From: Ming Lei @ 2019-09-19  9:21 UTC (permalink / raw)
  To: Damien Le Moal
  Cc: Alan Stern, Andrea Vai, Johannes Thumshirn, Jens Axboe, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg

On Thu, Sep 19, 2019 at 09:09:33AM +0000, Damien Le Moal wrote:
> On 2019/09/19 10:56, Ming Lei wrote:
> > On Thu, Sep 19, 2019 at 08:26:32AM +0000, Damien Le Moal wrote:
> >> On 2019/09/18 18:30, Alan Stern wrote:
> >>> On Wed, 18 Sep 2019, Andrea Vai wrote:
> >>>
> >>>>> Also, I wonder if the changing the size of the data transfers would
> >>>>> make any difference.  This is easy to try; just write "64" to
> >>>>> /sys/block/sd?/queue/max_sectors_kb (where the ? is the appropriate
> >>>>> drive letter) after the drive is plugged in but before the test
> >>>>> starts.
> >>>>
> >>>> ok, so I duplicated the tests above for the "64" case (it was
> >>>> initially set as "120", if it is relevant to know), leading to 40 tests named as
> >>>>
> >>>> bad.mon.out_50000000_64_TIMESTAMP
> >>>> bad.mon.out_50000000_non64_TIMESTAMP
> >>>> good.mon.out_50000000_64_TIMESTAMP
> >>>> good.mon.out_50000000_non64_TIMESTAMP
> >>>>
> >>>> where "64" denotes the ones done with that value in max_sectors_kb,
> >>>> and "not64" the ones without it (as far as I can tell, it has been
> >>>> always "120").
> >>>>
> >>>> So, we have 40 traces total. Each set of 10 trials is identified by
> >>>> a text file, which contains the output log of the test script (and the
> >>>> timestamps), also available in the download zipfile.
> >>>>
> >>>> Just to summarize here the times, they are respectively (number
> >>>> expressed  in seconds):
> >>>>
> >>>> BAD:
> >>>>   Logs: log_10trials_50MB_BAD_NonCanc_64.txt,
> >>>> log_10trials_50MB_BAD_NonCanc_non64.txt
> >>>>   64: 34, 34, 35, 39, 37, 32, 42, 44, 43, 40
> >>>>   not64: 61, 71, 59, 71, 62, 75, 62, 70, 62, 68
> >>>> GOOD:
> >>>>   Logs: log_10trials_50MB_GOOD_NonCanc_64.txt,
> >>>> log_10trials_50MB_GOOD_NonCanc_non64.txt
> >>>>   64: 34, 32, 35, 34, 35, 33, 34, 33, 33, 33
> >>>>   not64: 32, 30, 32, 31, 31, 30, 32, 30, 32, 31
> >>>
> >>> The improvement from using "64" with the bad kernel is quite large.  
> >>> That alone would be a big help for you.
> >>>
> >>> However, I did see what appears to be a very significant difference 
> >>> between the bad and good kernel traces.  It has to do with the order in 
> >>> which the blocks are accessed.
> >>>
> >>> Here is an extract from one of the bad traces.  I have erased all the 
> >>> information except for the columns containing the block numbers to be 
> >>> written:
> >>>
> >>> 00019628 00
> >>> 00019667 00
> >>> 00019628 80
> >>> 00019667 80
> >>> 00019629 00
> >>> 00019668 00
> >>> 00019629 80
> >>> 00019668 80
> >>>
> >>> Here is the equivalent portion from one of the good traces:
> >>>
> >>> 00019628 00
> >>> 00019628 80
> >>> 00019629 00
> >>> 00019629 80
> >>> 0001962a 00
> >>> 0001962a 80
> >>> 0001962b 00
> >>> 0001962b 80
> >>>
> >>> Notice that under the good kernel, the block numbers increase
> >>> monotonically in a single sequence.  But under the bad kernel, the
> >>> block numbers are not monotonic -- it looks like there are two separate
> >>> threads each with its own strictly increasing sequence.
> >>>
> >>> This is exactly the sort of difference one might expect to see from
> >>> the commit f664a3cc17b7 ("scsi: kill off the legacy IO path") you
> >>> identified as the cause of the problem.  With multiqueue I/O, it's not 
> >>> surprising to see multiple sequences of block numbers.
> >>>
> >>> Add it's not at all surprising that a consumer-grade USB storage device 
> >>> might do a much worse job of handling non-sequential writes than 
> >>> sequential ones.
> >>>
> >>> Which leads to a simple question for the SCSI or block-layer 
> >>> maintainers:  Is there a sysfs setting Andrea can tweak which will 
> >>> effectively restrict a particular disk device down to a single I/O
> >>> queue, forcing sequential access?
> >>
> >> The scheduling inefficiency you are seeing may be coming from the fact that the
> >> block layer does a direct issue of requests, bypassing the elevator, under some
> >> conditions. One of these is sync requests on a multiqueue device. We hit this
> >> problem on Zoned disks which can easily return an error for write requests
> >> without the elevator throttling writes per zones (zone write locking). This
> >> problem was discovered by Hans (on CC).
> >>
> >> I discussed this with Hannes yesterday and we think we have a fix, but we'll
> >> need to do a lot of testing as all block devices are potentially impacted by the
> >> change, including stacked drivers (DM). Performance regression is scary with any
> >> change in that area (see blk_mq_make_request() and use of
> >> blk_mq_try_issue_directly() vs blk_mq_sched_insert_request()).
> > 
> > Not sure this one is same with yours, for USB, mq-deadline is used at
> > default, and direct issue won't be possible. Direct issue is only used
> > in case of none or underlying queues of DM multipath.
> 
> For a multi-queue zoned disk, mq-deadline is also set, but we have observed
> unaligned write IO errors for sync writes because of mq-deadline being bypassed
> and as a result zones not being write-locked.
> 
> In blk_mq_make_request(), at the end, you have:
> 
> 	} else if ((q->nr_hw_queues > 1 && is_sync) || (!q->elevator &&
> 			!data.hctx->dispatch_busy)) {
> 		blk_mq_try_issue_directly(data.hctx, rq, &cookie);
> 	} else {
> 		blk_mq_sched_insert_request(rq, false, true, true);
> 	}
> 
> Which I read as "for a sync req on a multi-queue device, direct issue",
> regardless of the elevator being none or something else.

Yeah, looks elevator is bypassed in the above case, which seems a bug.
USB storage has only single queue.

> 
> The correct test should probably be:
> 
> 	} else if (!q->elevator &&
> 		   ((q->nr_hw_queues > 1 && is_sync) || 	
> 		     !data.hctx->dispatch_busy))) {
> 		blk_mq_try_issue_directly(data.hctx, rq, &cookie);
> 	} else {
> 		blk_mq_sched_insert_request(rq, false, true, true);
> 	}
> 
> That is, never bypass the elevator if one is set. Thoughts ?

IMO, elevator shouldn't be bypassed any time, looks it is bypassed
in the following branch too, but may not be reached for zone device.

blk_mq_make_request()
 ...
 } else if (plug && !blk_queue_nomerges(q)) {
	...
	if (same_queue_rq) {
                        data.hctx = same_queue_rq->mq_hctx;
                        trace_block_unplug(q, 1, true);
                        blk_mq_try_issue_directly(data.hctx, same_queue_rq,
                                        &cookie);
                }
 }


Thanks,
Ming

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-19  8:26               ` Damien Le Moal
  2019-09-19  8:55                 ` Ming Lei
@ 2019-09-19 14:01                 ` Alan Stern
  2019-09-19 14:14                   ` Damien Le Moal
  1 sibling, 1 reply; 104+ messages in thread
From: Alan Stern @ 2019-09-19 14:01 UTC (permalink / raw)
  To: Damien Le Moal
  Cc: Andrea Vai, Johannes Thumshirn, Jens Axboe, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg

On Thu, 19 Sep 2019, Damien Le Moal wrote:

> > This is exactly the sort of difference one might expect to see from
> > the commit f664a3cc17b7 ("scsi: kill off the legacy IO path") you
> > identified as the cause of the problem.  With multiqueue I/O, it's not 
> > surprising to see multiple sequences of block numbers.
> > 
> > Add it's not at all surprising that a consumer-grade USB storage device 
> > might do a much worse job of handling non-sequential writes than 
> > sequential ones.
> > 
> > Which leads to a simple question for the SCSI or block-layer 
> > maintainers:  Is there a sysfs setting Andrea can tweak which will 
> > effectively restrict a particular disk device down to a single I/O
> > queue, forcing sequential access?
> 
> The scheduling inefficiency you are seeing may be coming from the fact that the
> block layer does a direct issue of requests, bypassing the elevator, under some
> conditions. One of these is sync requests on a multiqueue device. We hit this
> problem on Zoned disks which can easily return an error for write requests
> without the elevator throttling writes per zones (zone write locking). This
> problem was discovered by Hans (on CC).

Is there any way for Andrea to check whether this is the underlying
cause?

> I discussed this with Hannes yesterday and we think we have a fix, but we'll
> need to do a lot of testing as all block devices are potentially impacted by the
> change, including stacked drivers (DM). Performance regression is scary with any
> change in that area (see blk_mq_make_request() and use of
> blk_mq_try_issue_directly() vs blk_mq_sched_insert_request()).

No doubt Andrea will be happy to test your fix when it's ready.

Alan Stern


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-19 14:01                 ` Alan Stern
@ 2019-09-19 14:14                   ` Damien Le Moal
  2019-09-20  7:03                     ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Damien Le Moal @ 2019-09-19 14:14 UTC (permalink / raw)
  To: Alan Stern
  Cc: Andrea Vai, Johannes Thumshirn, Jens Axboe, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg

On 2019/09/19 16:01, Alan Stern wrote:
> On Thu, 19 Sep 2019, Damien Le Moal wrote:
> 
>>> This is exactly the sort of difference one might expect to see from
>>> the commit f664a3cc17b7 ("scsi: kill off the legacy IO path") you
>>> identified as the cause of the problem.  With multiqueue I/O, it's not 
>>> surprising to see multiple sequences of block numbers.
>>>
>>> Add it's not at all surprising that a consumer-grade USB storage device 
>>> might do a much worse job of handling non-sequential writes than 
>>> sequential ones.
>>>
>>> Which leads to a simple question for the SCSI or block-layer 
>>> maintainers:  Is there a sysfs setting Andrea can tweak which will 
>>> effectively restrict a particular disk device down to a single I/O
>>> queue, forcing sequential access?
>>
>> The scheduling inefficiency you are seeing may be coming from the fact that the
>> block layer does a direct issue of requests, bypassing the elevator, under some
>> conditions. One of these is sync requests on a multiqueue device. We hit this
>> problem on Zoned disks which can easily return an error for write requests
>> without the elevator throttling writes per zones (zone write locking). This
>> problem was discovered by Hans (on CC).
> 
> Is there any way for Andrea to check whether this is the underlying
> cause?>
>> I discussed this with Hannes yesterday and we think we have a fix, but we'll
>> need to do a lot of testing as all block devices are potentially impacted by the
>> change, including stacked drivers (DM). Performance regression is scary with any
>> change in that area (see blk_mq_make_request() and use of
>> blk_mq_try_issue_directly() vs blk_mq_sched_insert_request()).
> 
> No doubt Andrea will be happy to test your fix when it's ready.

Hannes posted an RFC series:

https://www.spinics.net/lists/linux-scsi/msg133848.html

Andrea can try it. But If the USB device is not multi-queue, this fix will
probably have no effect.

Andrea,

What is the device in question ? Is it a USB external HDD ? SSD ? Flash key ?

Best regards.


-- 
Damien Le Moal
Western Digital Research

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-19  7:33               ` Andrea Vai
@ 2019-09-19 17:54                 ` Alan Stern
  2019-09-20  7:25                   ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Alan Stern @ 2019-09-19 17:54 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Johannes Thumshirn, Jens Axboe, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH

On Thu, 19 Sep 2019, Andrea Vai wrote:

> BTW, another question: Alan refers to the slow media as a "consumer-
> grade USB storage device". What could I do to identify and buy a "good
> media"? Are there any features to look for?

In general, USB flash drives should not be expected to work as well as 
an actual disk drive connected over USB.

Alan Stern


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-19 14:14                   ` Damien Le Moal
@ 2019-09-20  7:03                     ` Andrea Vai
  2019-09-25 19:30                       ` Alan Stern
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-09-20  7:03 UTC (permalink / raw)
  To: Damien Le Moal, Alan Stern
  Cc: Johannes Thumshirn, Jens Axboe, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg

Il giorno gio, 19/09/2019 alle 14.14 +0000, Damien Le Moal ha scritto:
> On 2019/09/19 16:01, Alan Stern wrote:
> [...]
> > No doubt Andrea will be happy to test your fix when it's ready.

Yes, of course.

> 
> Hannes posted an RFC series:
> 
> https://www.spinics.net/lists/linux-scsi/msg133848.html
> 
> Andrea can try it.

Ok, but I would need some instructions please, because I am not able
to understand how to "try it". Sorry for that.

> Andrea,
> 
> What is the device in question ? Is it a USB external HDD ? SSD ?
> Flash key ?

It is a USB flash key (a.k.a. pendrive, flash drive, etc.):

ID 0951:1666 Kingston Technology DataTraveler 100 G3/G4/SE9 G2

Thanks, and bye
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-19 17:54                 ` Alan Stern
@ 2019-09-20  7:25                   ` Andrea Vai
  2019-09-20  7:44                     ` Greg KH
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-09-20  7:25 UTC (permalink / raw)
  To: Alan Stern
  Cc: Johannes Thumshirn, Jens Axboe, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH

Il giorno gio, 19/09/2019 alle 13.54 -0400, Alan Stern ha scritto:
> 
> In general, USB flash drives should not be expected to work as well
> as 
> an actual disk drive connected over USB.

Ok, so I think I'll buy some different hardware. Would an SSD drive
(connected over USB) behave like a flash drive or like an "actual disk
drive" from this point of view?

Many thanks, and bye
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-20  7:25                   ` Andrea Vai
@ 2019-09-20  7:44                     ` Greg KH
  0 siblings, 0 replies; 104+ messages in thread
From: Greg KH @ 2019-09-20  7:44 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Alan Stern, Johannes Thumshirn, Jens Axboe, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Ming Lei, Omar Sandoval, Martin K. Petersen

On Fri, Sep 20, 2019 at 09:25:17AM +0200, Andrea Vai wrote:
> Il giorno gio, 19/09/2019 alle 13.54 -0400, Alan Stern ha scritto:
> > 
> > In general, USB flash drives should not be expected to work as well
> > as 
> > an actual disk drive connected over USB.
> 
> Ok, so I think I'll buy some different hardware. Would an SSD drive
> (connected over USB) behave like a flash drive or like an "actual disk
> drive" from this point of view?

It all depends on the drive.  Some are a lot better than others, and
it's almost impossible to tell until you buy the thing and try it out :(

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-20  7:03                     ` Andrea Vai
@ 2019-09-25 19:30                       ` Alan Stern
  2019-09-25 19:36                         ` Jens Axboe
  0 siblings, 1 reply; 104+ messages in thread
From: Alan Stern @ 2019-09-25 19:30 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Damien Le Moal, Johannes Thumshirn, Jens Axboe, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg

[-- Attachment #1: Type: TEXT/PLAIN, Size: 920 bytes --]

On Fri, 20 Sep 2019, Andrea Vai wrote:

> Il giorno gio, 19/09/2019 alle 14.14 +0000, Damien Le Moal ha scritto:
> > On 2019/09/19 16:01, Alan Stern wrote:
> > [...]
> > > No doubt Andrea will be happy to test your fix when it's ready.
> 
> Yes, of course.
> 
> > 
> > Hannes posted an RFC series:
> > 
> > https://www.spinics.net/lists/linux-scsi/msg133848.html
> > 
> > Andrea can try it.
> 
> Ok, but I would need some instructions please, because I am not able
> to understand how to "try it". Sorry for that.

I have attached the two patches to this email.  You should start with a 
recent kernel source tree and apply the patches by doing:

	git apply patch1 patch2

or something similar.  Then build a kernel from the new source code and 
test it.

Ultimately, if nobody can find a way to restore the sequential I/O 
behavior we had prior to commit f664a3cc17b7, that commit may have to 
be reverted.

Alan Stern

[-- Attachment #2: Type: TEXT/PLAIN, Size: 980 bytes --]

From: Hannes Reinecke <hare@suse.com>

When blk_mq_request_issue_directly() returns BLK_STS_RESOURCE we
need to requeue the I/O, but adding it to the global request list
will mess up with the passed-in request list. So re-add the request
to the original list and leave it to the caller to handle situations
where the list wasn't completely emptied.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 block/blk-mq.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index b038ec680e84..44ff3c1442a4 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1899,8 +1899,7 @@ void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx,
 		if (ret != BLK_STS_OK) {
 			if (ret == BLK_STS_RESOURCE ||
 					ret == BLK_STS_DEV_RESOURCE) {
-				blk_mq_request_bypass_insert(rq,
-							list_empty(list));
+				list_add(list, &rq->queuelist);
 				break;
 			}
 			blk_mq_end_request(rq, ret);
-- 
2.16.4

[-- Attachment #3: Type: TEXT/PLAIN, Size: 1721 bytes --]

From: Hannes Reinecke <hare@suse.com>

A scheduler might be attached even for devices exposing more than
one hardware queue, so the check for the number of hardware queue
is pointless and should be removed.

Signed-off-by: Hannes Reinecke <hare@suse.com>
---
 block/blk-mq.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 44ff3c1442a4..faab542e4836 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1931,7 +1931,6 @@ static void blk_add_rq_to_plug(struct blk_plug *plug, struct request *rq)
 
 static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)
 {
-	const int is_sync = op_is_sync(bio->bi_opf);
 	const int is_flush_fua = op_is_flush(bio->bi_opf);
 	struct blk_mq_alloc_data data = { .flags = 0};
 	struct request *rq;
@@ -1977,7 +1976,7 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)
 		/* bypass scheduler for flush rq */
 		blk_insert_flush(rq);
 		blk_mq_run_hw_queue(data.hctx, true);
-	} else if (plug && (q->nr_hw_queues == 1 || q->mq_ops->commit_rqs)) {
+	} else if (plug && q->mq_ops->commit_rqs) {
 		/*
 		 * Use plugging if we have a ->commit_rqs() hook as well, as
 		 * we know the driver uses bd->last in a smart fashion.
@@ -2020,9 +2019,6 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)
 			blk_mq_try_issue_directly(data.hctx, same_queue_rq,
 					&cookie);
 		}
-	} else if ((q->nr_hw_queues > 1 && is_sync) || (!q->elevator &&
-			!data.hctx->dispatch_busy)) {
-		blk_mq_try_issue_directly(data.hctx, rq, &cookie);
 	} else {
 		blk_mq_sched_insert_request(rq, false, true, true);
 	}
-- 
2.16.4

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-25 19:30                       ` Alan Stern
@ 2019-09-25 19:36                         ` Jens Axboe
  2019-09-27 15:47                           ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Jens Axboe @ 2019-09-25 19:36 UTC (permalink / raw)
  To: Alan Stern, Andrea Vai
  Cc: Damien Le Moal, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg

On 9/25/19 9:30 PM, Alan Stern wrote:
> On Fri, 20 Sep 2019, Andrea Vai wrote:
> 
>> Il giorno gio, 19/09/2019 alle 14.14 +0000, Damien Le Moal ha scritto:
>>> On 2019/09/19 16:01, Alan Stern wrote:
>>> [...]
>>>> No doubt Andrea will be happy to test your fix when it's ready.
>>
>> Yes, of course.
>>
>>>
>>> Hannes posted an RFC series:
>>>
>>> https://www.spinics.net/lists/linux-scsi/msg133848.html
>>>
>>> Andrea can try it.
>>
>> Ok, but I would need some instructions please, because I am not able
>> to understand how to "try it". Sorry for that.
> 
> I have attached the two patches to this email.  You should start with a
> recent kernel source tree and apply the patches by doing:
> 
> 	git apply patch1 patch2
> 
> or something similar.  Then build a kernel from the new source code and
> test it.
> 
> Ultimately, if nobody can find a way to restore the sequential I/O
> behavior we had prior to commit f664a3cc17b7, that commit may have to
> be reverted.

Don't use patch1, it's buggy. patch2 should be enough to test the theory.

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-25 19:36                         ` Jens Axboe
@ 2019-09-27 15:47                           ` Andrea Vai
  2019-11-04 16:00                             ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-09-27 15:47 UTC (permalink / raw)
  To: Jens Axboe, Alan Stern
  Cc: Damien Le Moal, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg

Il giorno mer, 25/09/2019 alle 21.36 +0200, Jens Axboe ha scritto:
> On 9/25/19 9:30 PM, Alan Stern wrote:
> [...]
> > 
> > I have attached the two patches to this email.  You should start
> with a
> > recent kernel source tree and apply the patches by doing:
> > 
> > 	git apply patch1 patch2
> > 
> > or something similar.  Then build a kernel from the new source
> code and
> > test it.
> > 
> > Ultimately, if nobody can find a way to restore the sequential I/O
> > behavior we had prior to commit f664a3cc17b7, that commit may have
> to
> > be reverted.
> 
> Don't use patch1, it's buggy. patch2 should be enough to test the
> theory.

Sorry, but if I cd into the "linux" directory and run the command

# git apply -v patch2

the result is that the patch cannot be applied correctly:

------------------------------------------------------------------------------
Controllo della patch block/blk-mq.c in corso...
error: durante la ricerca per:
?
static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)?
{?
	const int is_sync = op_is_sync(bio->bi_opf);?
	const int is_flush_fua = op_is_flush(bio->bi_opf);?
	struct blk_mq_alloc_data data = { .flags = 0};?
	struct request *rq;?

error: patch non riuscita: block/blk-mq.c:1931
error: block/blk-mq.c: la patch non si applica correttamente
------------------------------------------------------------------------------

The "linux" directory is the one generated by a fresh git clone:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

What am I doing wrong?

Thanks, and bye
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-09-27 15:47                           ` Andrea Vai
@ 2019-11-04 16:00                             ` Andrea Vai
  2019-11-04 18:20                               ` Alan Stern
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-11-04 16:00 UTC (permalink / raw)
  To: Jens Axboe, Alan Stern
  Cc: Damien Le Moal, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg

[-- Attachment #1: Type: text/plain, Size: 2368 bytes --]

Il giorno ven, 27/09/2019 alle 17.47 +0200, Andrea Vai ha scritto:
> Il giorno mer, 25/09/2019 alle 21.36 +0200, Jens Axboe ha scritto:
> > On 9/25/19 9:30 PM, Alan Stern wrote:
> > [...]
> > > 
> > > I have attached the two patches to this email.  You should start
> > with a
> > > recent kernel source tree and apply the patches by doing:
> > > 
> > > 	git apply patch1 patch2
> > > 
> > > or something similar.  Then build a kernel from the new source
> > code and
> > > test it.
> > > 
> > > Ultimately, if nobody can find a way to restore the sequential
> I/O
> > > behavior we had prior to commit f664a3cc17b7, that commit may
> have
> > to
> > > be reverted.
> > 
> > Don't use patch1, it's buggy. patch2 should be enough to test the
> > theory.

As I didn't have any answer, I am quoting my last reply here:

> 
> Sorry, but if I cd into the "linux" directory and run the command
> 
> # git apply -v patch2
> 
> the result is that the patch cannot be applied correctly:
> 
> --------------------------------------------------------------------
> ----------
> Controllo della patch block/blk-mq.c in corso...
> error: durante la ricerca per:
> ?
> static blk_qc_t blk_mq_make_request(struct request_queue *q, struct
> bio *bio)?
> {?
> 	const int is_sync = op_is_sync(bio->bi_opf);?
> 	const int is_flush_fua = op_is_flush(bio->bi_opf);?
> 	struct blk_mq_alloc_data data = { .flags = 0};?
> 	struct request *rq;?
> 
> error: patch non riuscita: block/blk-mq.c:1931
> error: block/blk-mq.c: la patch non si applica correttamente
> --------------------------------------------------------------------
> ----------
> 
> The "linux" directory is the one generated by a fresh git clone:
> 
> git clone
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> 
> What am I doing wrong?
> 

Meanwhile, Alan tried to help me and gave me another patch (attached),
which doesn't work too, but gives a different error: "The git diff
header does not contain information about the file once removed 1
initial component of the path (row 14)" (actually, this is my
translation from the original message in Italian: "error:
l'intestazione git diff non riporta le informazioni sul file una volta
rimosso 1 componente iniziale del percorso (riga 14)")

I tested the two patches after a fresh git clone today, a few minutes
ago.

What can I do?

Thank you,
Bye
Andrea

[-- Attachment #2: patch2_alan --]
[-- Type: message/rfc822, Size: 296 bytes --]

From: Hannes Reinecke <hare@suse.com>
Subject: No Subject
Date: Mon, 04 Nov 2019 16:58:21 +0100
Message-ID: <fe072bc69e13435573d824133c3981f8841cf2c7.camel@suse.com>



^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-04 16:00                             ` Andrea Vai
@ 2019-11-04 18:20                               ` Alan Stern
  2019-11-05 11:48                                 ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Alan Stern @ 2019-11-04 18:20 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Jens Axboe, Damien Le Moal, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1088 bytes --]

On Mon, 4 Nov 2019, Andrea Vai wrote:

> > The "linux" directory is the one generated by a fresh git clone:
> > 
> > git clone
> > git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> > 
> > What am I doing wrong?
> > 
> 
> Meanwhile, Alan tried to help me and gave me another patch (attached),
> which doesn't work too, but gives a different error: "The git diff
> header does not contain information about the file once removed 1
> initial component of the path (row 14)" (actually, this is my
> translation from the original message in Italian: "error:
> l'intestazione git diff non riporta le informazioni sul file una volta
> rimosso 1 componente iniziale del percorso (riga 14)")
> 
> I tested the two patches after a fresh git clone today, a few minutes
> ago.
> 
> What can I do?

You should be able to do something like this:

	cd linux
	patch -p1 </path/to/patch2

and that should work with no errors.  You don't need to use git to 
apply a patch.

In case that patch2 file was mangled somewhere along the way, I have 
attached a copy to this message.

Alan Stern

[-- Attachment #2: Type: TEXT/PLAIN, Size: 1777 bytes --]

From: Hannes Reinecke <hare@suse.com>\r
\r
A scheduler might be attached even for devices exposing more than\r
one hardware queue, so the check for the number of hardware queue\r
is pointless and should be removed.\r
\r
Signed-off-by: Hannes Reinecke <hare@suse.com>\r
---\r
 block/blk-mq.c |    7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)
\r
diff --git a/block/blk-mq.c b/block/blk-mq.c\r
index 44ff3c1442a4..faab542e4836 100644\r
Index: usb-devel/block/blk-mq.c
===================================================================
--- usb-devel.orig/block/blk-mq.c
+++ usb-devel/block/blk-mq.c
@@ -1946,7 +1946,6 @@ static void blk_add_rq_to_plug(struct bl
 
 static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)
 {
-	const int is_sync = op_is_sync(bio->bi_opf);
 	const int is_flush_fua = op_is_flush(bio->bi_opf);
 	struct blk_mq_alloc_data data = { .flags = 0};
 	struct request *rq;
@@ -1992,8 +1991,7 @@ static blk_qc_t blk_mq_make_request(stru
 		/* bypass scheduler for flush rq */
 		blk_insert_flush(rq);
 		blk_mq_run_hw_queue(data.hctx, true);
-	} else if (plug && (q->nr_hw_queues == 1 || q->mq_ops->commit_rqs ||
-				!blk_queue_nonrot(q))) {
+	} else if (plug && (q->mq_ops->commit_rqs || !blk_queue_nonrot(q))) {
 		/*
 		 * Use plugging if we have a ->commit_rqs() hook as well, as
 		 * we know the driver uses bd->last in a smart fashion.
@@ -2041,9 +2039,6 @@ static blk_qc_t blk_mq_make_request(stru
 			blk_mq_try_issue_directly(data.hctx, same_queue_rq,
 					&cookie);
 		}
-	} else if ((q->nr_hw_queues > 1 && is_sync) ||
-			!data.hctx->dispatch_busy) {
-		blk_mq_try_issue_directly(data.hctx, rq, &cookie);
 	} else {
 		blk_mq_sched_insert_request(rq, false, true, true);
 	}

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-04 18:20                               ` Alan Stern
@ 2019-11-05 11:48                                 ` Andrea Vai
  2019-11-05 18:31                                   ` Alan Stern
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-11-05 11:48 UTC (permalink / raw)
  To: Alan Stern
  Cc: Jens Axboe, Damien Le Moal, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg

Il giorno lun, 04/11/2019 alle 13.20 -0500, Alan Stern ha scritto:
> On Mon, 4 Nov 2019, Andrea Vai wrote:
> 
> > > The "linux" directory is the one generated by a fresh git clone:
> > > 
> > > git clone
> > > git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> > > 
> > > What am I doing wrong?
> > > 
> > 
> > Meanwhile, Alan tried to help me and gave me another patch
> (attached),
> > which doesn't work too, but gives a different error: "The git diff
> > header does not contain information about the file once removed 1
> > initial component of the path (row 14)" (actually, this is my
> > translation from the original message in Italian: "error:
> > l'intestazione git diff non riporta le informazioni sul file una
> volta
> > rimosso 1 componente iniziale del percorso (riga 14)")
> > 
> > I tested the two patches after a fresh git clone today, a few
> minutes
> > ago.
> > 
> > What can I do?
> 
> You should be able to do something like this:
> 
>         cd linux
>         patch -p1 </path/to/patch2
> 
> and that should work with no errors.  You don't need to use git to 
> apply a patch.
> 
> In case that patch2 file was mangled somewhere along the way, I
> have 
> attached a copy to this message.

Ok, so the "patch" command worked, the kernel compiled and ran, but
the test still failed (273, 108, 104, 260, 177, 236, 179, 1123, 289,
873 seconds to copy a 500MB file, vs. ~30 seconds with the "good"
kernel).

Let me know what else could I do,

Thanks, and bye
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-05 11:48                                 ` Andrea Vai
@ 2019-11-05 18:31                                   ` Alan Stern
  2019-11-05 23:29                                     ` Jens Axboe
  0 siblings, 1 reply; 104+ messages in thread
From: Alan Stern @ 2019-11-05 18:31 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Jens Axboe, Damien Le Moal, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

On Tue, 5 Nov 2019, Andrea Vai wrote:

> Il giorno lun, 04/11/2019 alle 13.20 -0500, Alan Stern ha scritto:

> > You should be able to do something like this:
> > 
> >         cd linux
> >         patch -p1 </path/to/patch2
> > 
> > and that should work with no errors.  You don't need to use git to 
> > apply a patch.
> > 
> > In case that patch2 file was mangled somewhere along the way, I
> > have 
> > attached a copy to this message.
> 
> Ok, so the "patch" command worked, the kernel compiled and ran, but
> the test still failed (273, 108, 104, 260, 177, 236, 179, 1123, 289,
> 873 seconds to copy a 500MB file, vs. ~30 seconds with the "good"
> kernel).
> 
> Let me know what else could I do,

I'm out of suggestions.  If anyone else knows how to make a kernel with 
no legacy queuing support -- only multiqueue -- issue I/O requests 
sequentially, please speak up.

In the absence of any responses, after a week or so I will submit a
patch to revert the f664a3cc17b7 ("scsi: kill off the legacy IO path")  
commit.

Alan Stern


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-05 18:31                                   ` Alan Stern
@ 2019-11-05 23:29                                     ` Jens Axboe
  2019-11-06 16:03                                       ` Alan Stern
  0 siblings, 1 reply; 104+ messages in thread
From: Jens Axboe @ 2019-11-05 23:29 UTC (permalink / raw)
  To: Alan Stern, Andrea Vai
  Cc: Damien Le Moal, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

On 11/5/19 11:31 AM, Alan Stern wrote:
> On Tue, 5 Nov 2019, Andrea Vai wrote:
> 
>> Il giorno lun, 04/11/2019 alle 13.20 -0500, Alan Stern ha scritto:
> 
>>> You should be able to do something like this:
>>>
>>>          cd linux
>>>          patch -p1 </path/to/patch2
>>>
>>> and that should work with no errors.  You don't need to use git to
>>> apply a patch.
>>>
>>> In case that patch2 file was mangled somewhere along the way, I
>>> have
>>> attached a copy to this message.
>>
>> Ok, so the "patch" command worked, the kernel compiled and ran, but
>> the test still failed (273, 108, 104, 260, 177, 236, 179, 1123, 289,
>> 873 seconds to copy a 500MB file, vs. ~30 seconds with the "good"
>> kernel).
>>
>> Let me know what else could I do,
> 
> I'm out of suggestions.  If anyone else knows how to make a kernel with
> no legacy queuing support -- only multiqueue -- issue I/O requests
> sequentially, please speak up.

Do we know for a fact that the device needs strictly serialized requests
to not stall? And writes in particular? I won't comment on how broken
that is, just trying to establish this as the problem that's making this
particular device be slow?

I've lost track of this thread, but has mq-deadline been tried as the
IO scheduler? We do have support for strictly serialized (writes)
since that's required for zoned device, wouldn't be hard at all to make
this cover a blacklisted device like this one.

> In the absence of any responses, after a week or so I will submit a
> patch to revert the f664a3cc17b7 ("scsi: kill off the legacy IO path")
> commit.

That's not going to be feasible.

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-05 23:29                                     ` Jens Axboe
@ 2019-11-06 16:03                                       ` Alan Stern
  2019-11-06 22:13                                         ` Damien Le Moal
  0 siblings, 1 reply; 104+ messages in thread
From: Alan Stern @ 2019-11-06 16:03 UTC (permalink / raw)
  To: Jens Axboe
  Cc: Andrea Vai, Damien Le Moal, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

On Tue, 5 Nov 2019, Jens Axboe wrote:

> On 11/5/19 11:31 AM, Alan Stern wrote:
> > On Tue, 5 Nov 2019, Andrea Vai wrote:
> > 
> >> Il giorno lun, 04/11/2019 alle 13.20 -0500, Alan Stern ha scritto:
> > 
> >>> You should be able to do something like this:
> >>>
> >>>          cd linux
> >>>          patch -p1 </path/to/patch2
> >>>
> >>> and that should work with no errors.  You don't need to use git to
> >>> apply a patch.
> >>>
> >>> In case that patch2 file was mangled somewhere along the way, I
> >>> have
> >>> attached a copy to this message.
> >>
> >> Ok, so the "patch" command worked, the kernel compiled and ran, but
> >> the test still failed (273, 108, 104, 260, 177, 236, 179, 1123, 289,
> >> 873 seconds to copy a 500MB file, vs. ~30 seconds with the "good"
> >> kernel).
> >>
> >> Let me know what else could I do,
> > 
> > I'm out of suggestions.  If anyone else knows how to make a kernel with
> > no legacy queuing support -- only multiqueue -- issue I/O requests
> > sequentially, please speak up.
> 
> Do we know for a fact that the device needs strictly serialized requests
> to not stall?

Not exactly, but that is far and away the most likely explanation for
the device's behavior.  We tried making a bunch of changes, some of
which helped a little bit, but all of them left a very large
performance gap.  I/O monitoring showed that the only noticeable
difference in the kernel-device interaction caused by the $SUBJECT
commit was the non-sequential access pattern.

> And writes in particular?

Andrea has tested only the write behavior.  Possibly reading will be
affected too, but my guess is that it won't be.

> I won't comment on how broken
> that is, just trying to establish this as the problem that's making this
> particular device be slow?

It seems reasonable that the access pattern could make a significant
difference.  The device's behavior suggests that it buffers incoming
data and pauses from time to time to write the accumulated data into
non-volatile storage.  If its algorithm for allocating, erasing, and
writing data blocks is optimized for the sequential case, you can
easily imagine that non-sequential accesses would cause it to pause
more often and for longer times -- which is exactly what we observed.
These extra pauses are what resulted in the overall performance 
decrease.

So far we have had no way to perform a direct test.  That is, we don't
know of any setting that would change a single kernel between
sequential and non-sequential access.  If you can suggest a simple way
to force a kernel without the $SUBJECT commit to do non-sequential
writes, I'm sure Andrea will be happy to try it out and see if it
causes a slowdown.

> I've lost track of this thread, but has mq-deadline been tried as the
> IO scheduler? We do have support for strictly serialized (writes)
> since that's required for zoned device, wouldn't be hard at all to make
> this cover a blacklisted device like this one.

Please spell out the exact procedure in detail so that Andrea can try 
it.  He's not a kernel hacker, and I know very little about the block 
layer.

Alan Stern


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-06 16:03                                       ` Alan Stern
@ 2019-11-06 22:13                                         ` Damien Le Moal
  2019-11-07  7:04                                           ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Damien Le Moal @ 2019-11-06 22:13 UTC (permalink / raw)
  To: Alan Stern, Jens Axboe
  Cc: Andrea Vai, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

On 2019/11/07 1:04, Alan Stern wrote:
> On Tue, 5 Nov 2019, Jens Axboe wrote:
> 
>> On 11/5/19 11:31 AM, Alan Stern wrote:
>>> On Tue, 5 Nov 2019, Andrea Vai wrote:
>>>
>>>> Il giorno lun, 04/11/2019 alle 13.20 -0500, Alan Stern ha scritto:
>>>
>>>>> You should be able to do something like this:
>>>>>
>>>>>          cd linux
>>>>>          patch -p1 </path/to/patch2
>>>>>
>>>>> and that should work with no errors.  You don't need to use git to
>>>>> apply a patch.
>>>>>
>>>>> In case that patch2 file was mangled somewhere along the way, I
>>>>> have
>>>>> attached a copy to this message.
>>>>
>>>> Ok, so the "patch" command worked, the kernel compiled and ran, but
>>>> the test still failed (273, 108, 104, 260, 177, 236, 179, 1123, 289,
>>>> 873 seconds to copy a 500MB file, vs. ~30 seconds with the "good"
>>>> kernel).
>>>>
>>>> Let me know what else could I do,
>>>
>>> I'm out of suggestions.  If anyone else knows how to make a kernel with
>>> no legacy queuing support -- only multiqueue -- issue I/O requests
>>> sequentially, please speak up.
>>
>> Do we know for a fact that the device needs strictly serialized requests
>> to not stall?
> 
> Not exactly, but that is far and away the most likely explanation for
> the device's behavior.  We tried making a bunch of changes, some of
> which helped a little bit, but all of them left a very large
> performance gap.  I/O monitoring showed that the only noticeable
> difference in the kernel-device interaction caused by the $SUBJECT
> commit was the non-sequential access pattern.
> 
>> And writes in particular?
> 
> Andrea has tested only the write behavior.  Possibly reading will be
> affected too, but my guess is that it won't be.
> 
>> I won't comment on how broken
>> that is, just trying to establish this as the problem that's making this
>> particular device be slow?
> 
> It seems reasonable that the access pattern could make a significant
> difference.  The device's behavior suggests that it buffers incoming
> data and pauses from time to time to write the accumulated data into
> non-volatile storage.  If its algorithm for allocating, erasing, and
> writing data blocks is optimized for the sequential case, you can
> easily imagine that non-sequential accesses would cause it to pause
> more often and for longer times -- which is exactly what we observed.
> These extra pauses are what resulted in the overall performance 
> decrease.
> 
> So far we have had no way to perform a direct test.  That is, we don't
> know of any setting that would change a single kernel between
> sequential and non-sequential access.  If you can suggest a simple way
> to force a kernel without the $SUBJECT commit to do non-sequential
> writes, I'm sure Andrea will be happy to try it out and see if it
> causes a slowdown.
> 
>> I've lost track of this thread, but has mq-deadline been tried as the
>> IO scheduler? We do have support for strictly serialized (writes)
>> since that's required for zoned device, wouldn't be hard at all to make
>> this cover a blacklisted device like this one.
> 
> Please spell out the exact procedure in detail so that Andrea can try 
> it.  He's not a kernel hacker, and I know very little about the block 
> layer.

Please simply try your write tests after doing this:

echo mq-deadline > /sys/block/<name of your USB disk>/queue/scheduler

And confirm that mq-deadline is selected with:

cat /sys/block/<name of your USB disk>/queue/scheduler
[mq-deadline] kyber bfq none


> 
> Alan Stern
> 
> 


-- 
Damien Le Moal
Western Digital Research

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-06 22:13                                         ` Damien Le Moal
@ 2019-11-07  7:04                                           ` Andrea Vai
  2019-11-07  7:54                                             ` Damien Le Moal
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-11-07  7:04 UTC (permalink / raw)
  To: Damien Le Moal, Alan Stern, Jens Axboe
  Cc: Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha scritto:
> 
> 
> Please simply try your write tests after doing this:
> 
> echo mq-deadline > /sys/block/<name of your USB
> disk>/queue/scheduler
> 
> And confirm that mq-deadline is selected with:
> 
> cat /sys/block/<name of your USB disk>/queue/scheduler
> [mq-deadline] kyber bfq none

ok, which kernel should I test with this: the fresh git cloned, or the
one just patched with Alan's patch, or doesn't matter which one?

Thanks, and bye,
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-07  7:04                                           ` Andrea Vai
@ 2019-11-07  7:54                                             ` Damien Le Moal
  2019-11-07 18:59                                               ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Damien Le Moal @ 2019-11-07  7:54 UTC (permalink / raw)
  To: Andrea Vai, Alan Stern, Jens Axboe
  Cc: Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

On 2019/11/07 16:04, Andrea Vai wrote:
> Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha scritto:
>>
>>
>> Please simply try your write tests after doing this:
>>
>> echo mq-deadline > /sys/block/<name of your USB
>> disk>/queue/scheduler
>>
>> And confirm that mq-deadline is selected with:
>>
>> cat /sys/block/<name of your USB disk>/queue/scheduler
>> [mq-deadline] kyber bfq none
> 
> ok, which kernel should I test with this: the fresh git cloned, or the
> one just patched with Alan's patch, or doesn't matter which one?

Probably all of them to see if there are any differences.

> 
> Thanks, and bye,
> Andrea
> 
> 


-- 
Damien Le Moal
Western Digital Research

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-07  7:54                                             ` Damien Le Moal
@ 2019-11-07 18:59                                               ` Andrea Vai
  2019-11-08  8:42                                                 ` Damien Le Moal
  2019-11-09 22:28                                                 ` Ming Lei
  0 siblings, 2 replies; 104+ messages in thread
From: Andrea Vai @ 2019-11-07 18:59 UTC (permalink / raw)
  To: Damien Le Moal
  Cc: Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

[Sorry for the duplicate message, it didn't reach the lists due to
html formatting]
Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
<Damien.LeMoal@wdc.com> ha scritto:
>
> On 2019/11/07 16:04, Andrea Vai wrote:
> > Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha scritto:
> >>
> >>
> >> Please simply try your write tests after doing this:
> >>
> >> echo mq-deadline > /sys/block/<name of your USB
> >> disk>/queue/scheduler
> >>
> >> And confirm that mq-deadline is selected with:
> >>
> >> cat /sys/block/<name of your USB disk>/queue/scheduler
> >> [mq-deadline] kyber bfq none
> >
> > ok, which kernel should I test with this: the fresh git cloned, or the
> > one just patched with Alan's patch, or doesn't matter which one?
>
> Probably all of them to see if there are any differences.

with both kernels, the output of
cat /sys/block/sdh/queue/schedule

already contains [mq-deadline]: is it correct to assume that the echo
command and the subsequent testing is useless? What to do now?

Thanks, and bye
Andrea

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-07 18:59                                               ` Andrea Vai
@ 2019-11-08  8:42                                                 ` Damien Le Moal
  2019-11-08 14:33                                                   ` Jens Axboe
  2019-11-09 10:09                                                   ` Ming Lei
  2019-11-09 22:28                                                 ` Ming Lei
  1 sibling, 2 replies; 104+ messages in thread
From: Damien Le Moal @ 2019-11-08  8:42 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

On 2019/11/08 4:00, Andrea Vai wrote:
> [Sorry for the duplicate message, it didn't reach the lists due to
> html formatting]
> Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
> <Damien.LeMoal@wdc.com> ha scritto:
>>
>> On 2019/11/07 16:04, Andrea Vai wrote:
>>> Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha scritto:
>>>>
>>>>
>>>> Please simply try your write tests after doing this:
>>>>
>>>> echo mq-deadline > /sys/block/<name of your USB
>>>> disk>/queue/scheduler
>>>>
>>>> And confirm that mq-deadline is selected with:
>>>>
>>>> cat /sys/block/<name of your USB disk>/queue/scheduler
>>>> [mq-deadline] kyber bfq none
>>>
>>> ok, which kernel should I test with this: the fresh git cloned, or the
>>> one just patched with Alan's patch, or doesn't matter which one?
>>
>> Probably all of them to see if there are any differences.
> 
> with both kernels, the output of
> cat /sys/block/sdh/queue/schedule
> 
> already contains [mq-deadline]: is it correct to assume that the echo
> command and the subsequent testing is useless? What to do now?

Probably, yes. Have you obtained a blktrace of the workload during these
tests ? Any significant difference in the IO pattern (IO size and
randomness) and IO timing (any device idle time where the device has no
command to process) ? Asking because the problem may be above the block
layer, with the file system for instance.

> 
> Thanks, and bye
> Andrea
> 


-- 
Damien Le Moal
Western Digital Research

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-08  8:42                                                 ` Damien Le Moal
@ 2019-11-08 14:33                                                   ` Jens Axboe
  2019-11-11 10:46                                                     ` Andrea Vai
  2019-11-09 10:09                                                   ` Ming Lei
  1 sibling, 1 reply; 104+ messages in thread
From: Jens Axboe @ 2019-11-08 14:33 UTC (permalink / raw)
  To: Damien Le Moal, Andrea Vai
  Cc: Alan Stern, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

On 11/8/19 1:42 AM, Damien Le Moal wrote:
> On 2019/11/08 4:00, Andrea Vai wrote:
>> [Sorry for the duplicate message, it didn't reach the lists due to
>> html formatting]
>> Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
>> <Damien.LeMoal@wdc.com> ha scritto:
>>>
>>> On 2019/11/07 16:04, Andrea Vai wrote:
>>>> Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha scritto:
>>>>>
>>>>>
>>>>> Please simply try your write tests after doing this:
>>>>>
>>>>> echo mq-deadline > /sys/block/<name of your USB
>>>>> disk>/queue/scheduler
>>>>>
>>>>> And confirm that mq-deadline is selected with:
>>>>>
>>>>> cat /sys/block/<name of your USB disk>/queue/scheduler
>>>>> [mq-deadline] kyber bfq none
>>>>
>>>> ok, which kernel should I test with this: the fresh git cloned, or the
>>>> one just patched with Alan's patch, or doesn't matter which one?
>>>
>>> Probably all of them to see if there are any differences.
>>
>> with both kernels, the output of
>> cat /sys/block/sdh/queue/schedule
>>
>> already contains [mq-deadline]: is it correct to assume that the echo
>> command and the subsequent testing is useless? What to do now?
> 
> Probably, yes. Have you obtained a blktrace of the workload during these
> tests ? Any significant difference in the IO pattern (IO size and
> randomness) and IO timing (any device idle time where the device has no
> command to process) ? Asking because the problem may be above the block
> layer, with the file system for instance.

blktrace would indeed be super useful, especially if you can do that
with a kernel that's fast for you, and one with the current kernel
where it's slow.

Given that your device is sdh, you simply do:

# blktrace /dev/sdh

and then run the test, then ctrl-c the blktrace. Then do:

# blkparse sdh > output

and save that output file. Do both runs, and bzip2 them up. The shorter
the run you can reproduce with the better, to cut down on the size of
the traces.

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-08  8:42                                                 ` Damien Le Moal
  2019-11-08 14:33                                                   ` Jens Axboe
@ 2019-11-09 10:09                                                   ` Ming Lei
  1 sibling, 0 replies; 104+ messages in thread
From: Ming Lei @ 2019-11-09 10:09 UTC (permalink / raw)
  To: Damien Le Moal
  Cc: Andrea Vai, Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

On Fri, Nov 08, 2019 at 08:42:53AM +0000, Damien Le Moal wrote:
> On 2019/11/08 4:00, Andrea Vai wrote:
> > [Sorry for the duplicate message, it didn't reach the lists due to
> > html formatting]
> > Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
> > <Damien.LeMoal@wdc.com> ha scritto:
> >>
> >> On 2019/11/07 16:04, Andrea Vai wrote:
> >>> Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha scritto:
> >>>>
> >>>>
> >>>> Please simply try your write tests after doing this:
> >>>>
> >>>> echo mq-deadline > /sys/block/<name of your USB
> >>>> disk>/queue/scheduler
> >>>>
> >>>> And confirm that mq-deadline is selected with:
> >>>>
> >>>> cat /sys/block/<name of your USB disk>/queue/scheduler
> >>>> [mq-deadline] kyber bfq none
> >>>
> >>> ok, which kernel should I test with this: the fresh git cloned, or the
> >>> one just patched with Alan's patch, or doesn't matter which one?
> >>
> >> Probably all of them to see if there are any differences.
> > 
> > with both kernels, the output of
> > cat /sys/block/sdh/queue/schedule
> > 
> > already contains [mq-deadline]: is it correct to assume that the echo
> > command and the subsequent testing is useless? What to do now?
> 
> Probably, yes. Have you obtained a blktrace of the workload during these
> tests ? Any significant difference in the IO pattern (IO size and
> randomness) and IO timing (any device idle time where the device has no
> command to process) ? Asking because the problem may be above the block
> layer, with the file system for instance.

You may get the IO pattern via the previous trace 

https://lore.kernel.org/linux-usb/20190710024439.GA2621@ming.t460p/

IMO, if it is related write order, one possibility could be that
the queue lock is killed in .make_request_fn().


Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-07 18:59                                               ` Andrea Vai
  2019-11-08  8:42                                                 ` Damien Le Moal
@ 2019-11-09 22:28                                                 ` Ming Lei
  2019-11-11 10:50                                                   ` Andrea Vai
  2019-11-22 19:16                                                   ` Andrea Vai
  1 sibling, 2 replies; 104+ messages in thread
From: Ming Lei @ 2019-11-09 22:28 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

On Thu, Nov 07, 2019 at 07:59:44PM +0100, Andrea Vai wrote:
> [Sorry for the duplicate message, it didn't reach the lists due to
> html formatting]
> Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
> <Damien.LeMoal@wdc.com> ha scritto:
> >
> > On 2019/11/07 16:04, Andrea Vai wrote:
> > > Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha scritto:
> > >>
> > >>
> > >> Please simply try your write tests after doing this:
> > >>
> > >> echo mq-deadline > /sys/block/<name of your USB
> > >> disk>/queue/scheduler
> > >>
> > >> And confirm that mq-deadline is selected with:
> > >>
> > >> cat /sys/block/<name of your USB disk>/queue/scheduler
> > >> [mq-deadline] kyber bfq none
> > >
> > > ok, which kernel should I test with this: the fresh git cloned, or the
> > > one just patched with Alan's patch, or doesn't matter which one?
> >
> > Probably all of them to see if there are any differences.
> 
> with both kernels, the output of
> cat /sys/block/sdh/queue/schedule
> 
> already contains [mq-deadline]: is it correct to assume that the echo
> command and the subsequent testing is useless? What to do now?

Another thing we could try is to use 'none' via the following command:

 echo none > /sys/block/sdh/queue/scheduler  #suppose 'sdh' points to the usb storage disk

Because USB storage HBA is single hw queue, which depth is 1. This way
should change to dispatch IO in the order of bio submission.

Andrea, could you switch io scheduler to none and update us if difference
can be made?

Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-08 14:33                                                   ` Jens Axboe
@ 2019-11-11 10:46                                                     ` Andrea Vai
  0 siblings, 0 replies; 104+ messages in thread
From: Andrea Vai @ 2019-11-11 10:46 UTC (permalink / raw)
  To: Jens Axboe, Damien Le Moal
  Cc: Alan Stern, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

Il giorno ven, 08/11/2019 alle 07.33 -0700, Jens Axboe ha scritto:
> On 11/8/19 1:42 AM, Damien Le Moal wrote:
> > On 2019/11/08 4:00, Andrea Vai wrote:
> >> [Sorry for the duplicate message, it didn't reach the lists due
> to
> >> html formatting]
> >> Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
> >> <Damien.LeMoal@wdc.com> ha scritto:
> >>>
> >>> On 2019/11/07 16:04, Andrea Vai wrote:
> >>>> Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha
> scritto:
> >>>>>
> >>>>>
> >>>>> Please simply try your write tests after doing this:
> >>>>>
> >>>>> echo mq-deadline > /sys/block/<name of your USB
> >>>>> disk>/queue/scheduler
> >>>>>
> >>>>> And confirm that mq-deadline is selected with:
> >>>>>
> >>>>> cat /sys/block/<name of your USB disk>/queue/scheduler
> >>>>> [mq-deadline] kyber bfq none
> >>>>
> >>>> ok, which kernel should I test with this: the fresh git cloned,
> or the
> >>>> one just patched with Alan's patch, or doesn't matter which
> one?
> >>>
> >>> Probably all of them to see if there are any differences.
> >>
> >> with both kernels, the output of
> >> cat /sys/block/sdh/queue/schedule
> >>
> >> already contains [mq-deadline]: is it correct to assume that the
> echo
> >> command and the subsequent testing is useless? What to do now?
> > 
> > Probably, yes. Have you obtained a blktrace of the workload during
> these
> > tests ? Any significant difference in the IO pattern (IO size and
> > randomness) and IO timing (any device idle time where the device
> has no
> > command to process) ? Asking because the problem may be above the
> block
> > layer, with the file system for instance.
> 
> blktrace would indeed be super useful, especially if you can do that
> with a kernel that's fast for you, and one with the current kernel
> where it's slow.
> 
> Given that your device is sdh, you simply do:
> 
> # blktrace /dev/sdh
> 
> and then run the test, then ctrl-c the blktrace. Then do:
> 
> # blkparse sdh > output
> 
> and save that output file. Do both runs, and bzip2 them up. The
> shorter
> the run you can reproduce with the better, to cut down on the size
> of
> the traces.

Sorry, the next message from Ming...

-----
You may get the IO pattern via the previous trace 
https://lore.kernel.org/linux-usb/20190710024439.GA2621@ming.t460p/

IMO, if it is related write order, one possibility could be that
the queue lock is killed in .make_request_fn().
-----

...made me wonder if I should really do the blkparse trace test, or
not. So please confirm if it's needed (testing is quite time-consuming 
, so I'd like to do it if it's needed).

Thanks, and bye,
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-09 22:28                                                 ` Ming Lei
@ 2019-11-11 10:50                                                   ` Andrea Vai
  2019-11-11 11:05                                                     ` Ming Lei
  2019-11-22 19:16                                                   ` Andrea Vai
  1 sibling, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-11-11 10:50 UTC (permalink / raw)
  To: Ming Lei
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

Il giorno dom, 10/11/2019 alle 06.28 +0800, Ming Lei ha scritto:
> On Thu, Nov 07, 2019 at 07:59:44PM +0100, Andrea Vai wrote:
> > [Sorry for the duplicate message, it didn't reach the lists due to
> > html formatting]
> > Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
> > <Damien.LeMoal@wdc.com> ha scritto:
> > >
> > > On 2019/11/07 16:04, Andrea Vai wrote:
> > > > Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha
> scritto:
> > > >>
> > > >>
> > > >> Please simply try your write tests after doing this:
> > > >>
> > > >> echo mq-deadline > /sys/block/<name of your USB
> > > >> disk>/queue/scheduler
> > > >>
> > > >> And confirm that mq-deadline is selected with:
> > > >>
> > > >> cat /sys/block/<name of your USB disk>/queue/scheduler
> > > >> [mq-deadline] kyber bfq none
> > > >
> > > > ok, which kernel should I test with this: the fresh git
> cloned, or the
> > > > one just patched with Alan's patch, or doesn't matter which
> one?
> > >
> > > Probably all of them to see if there are any differences.
> > 
> > with both kernels, the output of
> > cat /sys/block/sdh/queue/schedule
> > 
> > already contains [mq-deadline]: is it correct to assume that the
> echo
> > command and the subsequent testing is useless? What to do now?
> 
> Another thing we could try is to use 'none' via the following
> command:
> 
>  echo none > /sys/block/sdh/queue/scheduler  #suppose 'sdh' points
> to the usb storage disk
> 
> Because USB storage HBA is single hw queue, which depth is 1. This
> way
> should change to dispatch IO in the order of bio submission.
> 
> Andrea, could you switch io scheduler to none and update us if
> difference
> can be made?

Of course I would to it, but I see that with the "good" kernel the
output of "cat /sys/block/sdf/queue/scheduler" (yes, now it's sdf) is

noop deadline [cfq]

, i.e. it doesn't show "none". Does it matter? (sorry if it's a silly
question)

Thanks, and bye
Andrea



^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-11 10:50                                                   ` Andrea Vai
@ 2019-11-11 11:05                                                     ` Ming Lei
  2019-11-11 11:13                                                       ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-11-11 11:05 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

On Mon, Nov 11, 2019 at 11:50:49AM +0100, Andrea Vai wrote:
> Il giorno dom, 10/11/2019 alle 06.28 +0800, Ming Lei ha scritto:
> > On Thu, Nov 07, 2019 at 07:59:44PM +0100, Andrea Vai wrote:
> > > [Sorry for the duplicate message, it didn't reach the lists due to
> > > html formatting]
> > > Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
> > > <Damien.LeMoal@wdc.com> ha scritto:
> > > >
> > > > On 2019/11/07 16:04, Andrea Vai wrote:
> > > > > Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha
> > scritto:
> > > > >>
> > > > >>
> > > > >> Please simply try your write tests after doing this:
> > > > >>
> > > > >> echo mq-deadline > /sys/block/<name of your USB
> > > > >> disk>/queue/scheduler
> > > > >>
> > > > >> And confirm that mq-deadline is selected with:
> > > > >>
> > > > >> cat /sys/block/<name of your USB disk>/queue/scheduler
> > > > >> [mq-deadline] kyber bfq none
> > > > >
> > > > > ok, which kernel should I test with this: the fresh git
> > cloned, or the
> > > > > one just patched with Alan's patch, or doesn't matter which
> > one?
> > > >
> > > > Probably all of them to see if there are any differences.
> > > 
> > > with both kernels, the output of
> > > cat /sys/block/sdh/queue/schedule
> > > 
> > > already contains [mq-deadline]: is it correct to assume that the
> > echo
> > > command and the subsequent testing is useless? What to do now?
> > 
> > Another thing we could try is to use 'none' via the following
> > command:
> > 
> >  echo none > /sys/block/sdh/queue/scheduler  #suppose 'sdh' points
> > to the usb storage disk
> > 
> > Because USB storage HBA is single hw queue, which depth is 1. This
> > way
> > should change to dispatch IO in the order of bio submission.
> > 
> > Andrea, could you switch io scheduler to none and update us if
> > difference
> > can be made?
> 
> Of course I would to it, but I see that with the "good" kernel the
> output of "cat /sys/block/sdf/queue/scheduler" (yes, now it's sdf) is
> 
> noop deadline [cfq]

Not sure if cfq makes a difference, and I guess you may get same result
with noop or deadline. However, if you only see good write performance with
cfq, you may try 'bfq' and see if it works as cfq.

> 
> , i.e. it doesn't show "none". Does it matter? (sorry if it's a silly
> question)

We are talking about new kernel in which there can't be 'noop deadline [cfq]'
any more. And you should see the following output from '/sys/block/sdf/queue/scheduler'
in the new kernel:

	[mq-deadline] kyber bfq none


thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-11 11:05                                                     ` Ming Lei
@ 2019-11-11 11:13                                                       ` Andrea Vai
  0 siblings, 0 replies; 104+ messages in thread
From: Andrea Vai @ 2019-11-11 11:13 UTC (permalink / raw)
  To: Ming Lei
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

Il giorno lun, 11/11/2019 alle 19.05 +0800, Ming Lei ha scritto:
> On Mon, Nov 11, 2019 at 11:50:49AM +0100, Andrea Vai wrote:
> > Il giorno dom, 10/11/2019 alle 06.28 +0800, Ming Lei ha scritto:
> > > On Thu, Nov 07, 2019 at 07:59:44PM +0100, Andrea Vai wrote:
> > > > [Sorry for the duplicate message, it didn't reach the lists
> due to
> > > > html formatting]
> > > > Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
> > > > <Damien.LeMoal@wdc.com> ha scritto:
> > > > >
> > > > > On 2019/11/07 16:04, Andrea Vai wrote:
> > > > > > Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal
> ha
> > > scritto:
> > > > > >>
> > > > > >>
> > > > > >> Please simply try your write tests after doing this:
> > > > > >>
> > > > > >> echo mq-deadline > /sys/block/<name of your USB
> > > > > >> disk>/queue/scheduler
> > > > > >>
> > > > > >> And confirm that mq-deadline is selected with:
> > > > > >>
> > > > > >> cat /sys/block/<name of your USB disk>/queue/scheduler
> > > > > >> [mq-deadline] kyber bfq none
> > > > > >
> > > > > > ok, which kernel should I test with this: the fresh git
> > > cloned, or the
> > > > > > one just patched with Alan's patch, or doesn't matter
> which
> > > one?
> > > > >
> > > > > Probably all of them to see if there are any differences.
> > > > 
> > > > with both kernels, the output of
> > > > cat /sys/block/sdh/queue/schedule
> > > > 
> > > > already contains [mq-deadline]: is it correct to assume that
> the
> > > echo
> > > > command and the subsequent testing is useless? What to do now?
> > > 
> > > Another thing we could try is to use 'none' via the following
> > > command:
> > > 
> > >  echo none > /sys/block/sdh/queue/scheduler  #suppose 'sdh'
> points
> > > to the usb storage disk
> > > 
> > > Because USB storage HBA is single hw queue, which depth is 1.
> This
> > > way
> > > should change to dispatch IO in the order of bio submission.
> > > 
> > > Andrea, could you switch io scheduler to none and update us if
> > > difference
> > > can be made?
> > 
> > Of course I would to it, but I see that with the "good" kernel the
> > output of "cat /sys/block/sdf/queue/scheduler" (yes, now it's sdf)
> is
> > 
> > noop deadline [cfq]
> 
> Not sure if cfq makes a difference, and I guess you may get same
> result
> with noop or deadline. However, if you only see good write
> performance with
> cfq, you may try 'bfq' and see if it works as cfq.
> 
> > 
> > , i.e. it doesn't show "none". Does it matter? (sorry if it's a
> silly
> > question)
> 
> We are talking about new kernel in which there can't be 'noop
> deadline [cfq]'
> any more. And you should see the following output from
> '/sys/block/sdf/queue/scheduler'
> in the new kernel:
> 
> 	[mq-deadline] kyber bfq none
> 
> 

ok sorry I misunderstood, assumed you wanted me to compare the "none"
setting in the old kernel with the same setting in the new kernel. Now
it's clear to me that you want me to compare the different scheduler
settings in the new kernel.

Thanks, and bye
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-09 22:28                                                 ` Ming Lei
  2019-11-11 10:50                                                   ` Andrea Vai
@ 2019-11-22 19:16                                                   ` Andrea Vai
  2019-11-23  7:28                                                     ` Ming Lei
  1 sibling, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-11-22 19:16 UTC (permalink / raw)
  To: Ming Lei
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

Il giorno dom, 10/11/2019 alle 06.28 +0800, Ming Lei ha scritto:
> Another thing we could try is to use 'none' via the following
> command:
> 
>  echo none > /sys/block/sdh/queue/scheduler  #suppose 'sdh' points
> to the usb storage disk
> 
> Because USB storage HBA is single hw queue, which depth is 1. This
> way
> should change to dispatch IO in the order of bio submission.
> 
> Andrea, could you switch io scheduler to none and update us if
> difference
> can be made?

Using the new kernel, there is indeed a difference because the time to
copy a file is 1800 seconds with [mq-deadline], and 340 seconds with
[none]. But that is still far away from the old kernel, which performs
the copy of the same file in 76 seconds.

Side notes:

- The numbers above are average values calculated on 100 trials for
each  different situation. As previously noticed on this thread, with
the new kernel the times are also very different among the different
trials in the same situation. With the old kernel the standard
deviation on the times in a set of 100 trials is much smaller (to give
some mean/sigma values: m=1800->s=530; m=340->s=131; m=76->s=13; ).

- The size of the transferred file has been 1GB in these trials.
Smaller files don't always give appreciable differences, but if you
want I can also provide those data. Of course, I can also provide the
raw data of each set of trials.

Thanks,
and bye,

Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-22 19:16                                                   ` Andrea Vai
@ 2019-11-23  7:28                                                     ` Ming Lei
  2019-11-23 15:44                                                       ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-11-23  7:28 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

On Fri, Nov 22, 2019 at 08:16:30PM +0100, Andrea Vai wrote:
> Il giorno dom, 10/11/2019 alle 06.28 +0800, Ming Lei ha scritto:
> > Another thing we could try is to use 'none' via the following
> > command:
> > 
> >  echo none > /sys/block/sdh/queue/scheduler  #suppose 'sdh' points
> > to the usb storage disk
> > 
> > Because USB storage HBA is single hw queue, which depth is 1. This
> > way
> > should change to dispatch IO in the order of bio submission.
> > 
> > Andrea, could you switch io scheduler to none and update us if
> > difference
> > can be made?
> 
> Using the new kernel, there is indeed a difference because the time to
> copy a file is 1800 seconds with [mq-deadline], and 340 seconds with
> [none]. But that is still far away from the old kernel, which performs
> the copy of the same file in 76 seconds.

Please post the log of 'lsusb -v', and I will try to make a patch for
addressing the issue.


thanks, 
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-23  7:28                                                     ` Ming Lei
@ 2019-11-23 15:44                                                       ` Andrea Vai
  2019-11-25  3:54                                                         ` Ming Lei
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-11-23 15:44 UTC (permalink / raw)
  To: Ming Lei
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

[-- Attachment #1: Type: text/plain, Size: 201 bytes --]

Il giorno sab, 23/11/2019 alle 15.28 +0800, Ming Lei ha scritto:
> 
> Please post the log of 'lsusb -v', and I will try to make a patch
> for
> addressing the issue.

attached,

Thanks, and bye
Andrea

[-- Attachment #2: lsusb.txt --]
[-- Type: text/plain, Size: 41247 bytes --]

can't get debug descriptor: Resource temporarily unavailable

Bus 002 Device 002: ID 8087:8000 Intel Corp. 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         1 Single TT
  bMaxPacketSize0        64
  idVendor           0x8087 Intel Corp.
  idProduct          0x8000 
  bcdDevice            0.04
  iManufacturer           0 
  iProduct                0 
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0019
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0001  1x 1 bytes
        bInterval              12
Hub Descriptor:
  bLength               9
  bDescriptorType      41
  nNbrPorts             6
  wHubCharacteristic 0x0009
    Per-port power switching
    Per-port overcurrent protection
    TT think time 8 FS bits
  bPwrOn2PwrGood        0 * 2 milli seconds
  bHubContrCurrent      0 milli Ampere
  DeviceRemovable    0x00
  PortPwrCtrlMask    0xff
 Hub Port Status:
   Port 1: 0000.0100 power
   Port 2: 0000.0100 power
   Port 3: 0000.0100 power
   Port 4: 0000.0100 power
   Port 5: 0000.0100 power
   Port 6: 0000.0100 power
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         0 Full speed (or root) hub
  bMaxPacketSize0        64
  bNumConfigurations      1
Device Status:     0x0001
  Self Powered

Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         0 Full speed (or root) hub
  bMaxPacketSize0        64
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0002 2.0 root hub
  bcdDevice            5.03
  iManufacturer           3 Linux 5.3.8-200.fc30.x86_64 ehci_hcd
  iProduct                2 EHCI Host Controller
  iSerial                 1 0000:00:1d.0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0019
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval              12
Hub Descriptor:
  bLength               9
  bDescriptorType      41
 can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
 nNbrPorts             2
  wHubCharacteristic 0x000a
    No power switching (usb 1.0)
    Per-port overcurrent protection
  bPwrOn2PwrGood       10 * 2 milli seconds
  bHubContrCurrent      0 milli Ampere
  DeviceRemovable    0x02
  PortPwrCtrlMask    0xff
 Hub Port Status:
   Port 1: 0000.0507 highspeed power suspend enable connect
   Port 2: 0000.0100 power
Device Status:     0x0001
  Self Powered

Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         0 Full speed (or root) hub
  bMaxPacketSize0        64
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0002 2.0 root hub
  bcdDevice            5.03
  iManufacturer           3 Linux 5.3.8-200.fc30.x86_64 ehci_hcd
  iProduct                2 EHCI Host Controller
  iSerial                 1 0000:05:00.2
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0019
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval              12
Hub Descriptor:
  bLength               9
  bDescriptorType      41
  nNbrPorts             4
  wHubCharacteristic 0x000a
    No power switching (usb 1.0)
    Per-port overcurrent protection
  bPwrOn2PwrGood       10 * 2 milli seconds
  bHubContrCurrent      0 milli Ampere
  DeviceRemovable    0x00
  PortPwrCtrlMask    0xff
 Hub Port Status:
   Port 1: 0000.0100 power
   Port 2: 0000.0100 power
   Port 3: 0000.0100 power
   Port 4: 0000.0100 power
Device Status:     0x0001
  Self Powered

Bus 005 Device 002: ID 04a9:2206 Canon, Inc. CanoScan N650U/N656U
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.00
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         8
  idVendor           0x04a9 Canon, Inc.
  idProduct          0x2206 CanoScan N650U/N656U
  bcdDevice            1.00
  iManufacturer          64 Canon
  iProduct               77 CanoScan
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0027
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              500mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           3
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      0 
      bInterfaceProtocol    255 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0001  1x 1 bytes
        bInterval              16
      Endpoint Descriptorcan't get debug descriptor: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x03  EP 3 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
Device Status:     0x0000
  (Bus Powered)

Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         0 Full speed (or root) hub
  bMaxPacketSize0        64
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0001 1.1 root hub
  bcdDevice            5.03
  iManufacturer           3 Linux 5.3.8-200.fc30.x86_64 uhci_hcd
  iProduct                2 UHCI Host Controller
  iSerial                 1 0000:05:00.1
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0019
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0002  1x 2 bytes
        bInterval             255
Hub Descriptor:
  bLength               9
  bDescriptorType      41
  nNbrPorts             2
  wHubCharacteristic 0x000a
    No power switching (usb 1.0)
    Per-port overcurrent protection
  bPwrOn2PwrGood        1 * 2 milli seconds
  bHubContrCurrent      0 milli Ampere
  DeviceRemovable    0x00
  PortPwrCtrlMask    0xff
 Hub Port Status:
   Port 1: 0000.0100 power
   Port 2: 0000.0103 power enable connect
Device Status:     0x0001
  Self Powered

Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         0 Full speed (or root) hub
  bMaxPacketSize0        64
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0001 1.1 root hub
  bcdDevice            5.03
  iManufacturer           3 Linux 5.3.8-200.fc30.x86_64 uhci_hcd
  iProduct                2 UHCI Host Controller
  iSerial                 1 0000:05:00.0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0019
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iIntercan't get debug descriptor: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
face              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0002  1x 2 bytes
        bInterval             255
Hub Descriptor:
  bLength               9
  bDescriptorType      41
  nNbrPorts             2
  wHubCharacteristic 0x000a
    No power switching (usb 1.0)
    Per-port overcurrent protection
  bPwrOn2PwrGood        1 * 2 milli seconds
  bHubContrCurrent      0 milli Ampere
  DeviceRemovable    0x00
  PortPwrCtrlMask    0xff
 Hub Port Status:
   Port 1: 0000.0100 power
   Port 2: 0000.0100 power
Device Status:     0x0001
  Self Powered

Bus 001 Device 002: ID 8087:8008 Intel Corp. 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         1 Single TT
  bMaxPacketSize0        64
  idVendor           0x8087 Intel Corp.
  idProduct          0x8008 
  bcdDevice            0.04
  iManufacturer           0 
  iProduct                0 
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0019
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0001  1x 1 bytes
        bInterval              12
Hub Descriptor:
  bLength               9
  bDescriptorType      41
  nNbrPorts             6
  wHubCharacteristic 0x0009
    Per-port power switching
    Per-port overcurrent protection
    TT think time 8 FS bits
  bPwrOn2PwrGood        0 * 2 milli seconds
  bHubContrCurrent      0 milli Ampere
  DeviceRemovable    0x00
  PortPwrCtrlMask    0xff
 Hub Port Status:
   Port 1: 0000.0100 power
   Port 2: 0000.0100 power
   Port 3: 0000.0100 power
   Port 4: 0000.0100 power
   Port 5: 0000.0100 power
   Port 6: 0000.0100 power
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         0 Full speed (or root) hub
  bMaxPacketSize0        64
  bNumConfigurations      1
Device Status:     0x0001
  Self Powered

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         0 Full speed (or root) hub
  bMaxPacketSize0        64
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0002 2.0 root hub
  bcdDevice            5.03
  iManufacturer           3 Linux 5.3.8-200.fc30.x86_64 ehci_hcd
  iProduct                2 EHCI Host Controller
  iSerial                 1 0000:00:1a.0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0019
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval              12
Hub Descriptor:
  bLength               9
  bDescriptorType      41
  nNbrPorts             2
  wHubCharacteristic 0x000a
    No power switching (usb 1.0)
    Per-port overcurrent protection
  bPwrOn2PwrGood       10 * 2 milli seconds
  bHubContrCurrent      0 milli Ampere
  DeviceRemovable    0x02
  PortPwrCtrlMask    0xff
 Hub Port Status:
   Port 1: 0000.0507 highspeed power suspend enable connect
   Port 2: 0000.0100 power
Device Status:     0x0001
  Self Powered

Bus 007 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         3 
  bMaxPacketSize0         9
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0003 3.0 root hub
  bcdDevice            5.03
  iManufacturer           3 Linux 5.3.8-200.fc30.x86_64 xhci-hcd
  iProduct                2 xHCI Host Controller
  iSerial                 1 0000:00:14.0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x001f
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval              12
        bMaxBurst               0
Hub Descriptor:
  bLength              12
  bDescriptorType      42
  nNbrPorts             6
  wHubCharacteristic 0x000a
    No power switching (usb 1.0)
    Per-port overcurrent protection
  bPwrOn2PwrGood       10 * 2 milli seconds
  bHubContrCurrent      0 milli Ampere
  bHubDecLat          0.0 micro seconds
  wHubDelay             0 nano seconds
  DeviceRemovable    0x00
 Hub Port Status:
   Port 1: 0000.02a0 5Gbps power Rx.Detect
   Port 2: 0000.02a0 5Gbps power Rx.Detect
   Port 3: 0000.02a0 5Gbps power Rx.Detect
   Port 4: 0000.02a0 5Gbps power Rx.Detect
   Port 5: 0000.02a0 5Gbps power Rx.Detect
   Port 6: 0000.02a0 5Gbps power Rx.Detect
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength       0x000f
  bNumDeviceCaps          1
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x02
      Latency Tolerance Messages (LTM) Supported
    wSpeedsSupported   0x0008
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   3
      Lowest fully-functional device speed is SuperSpeed (5Gbps)
    bU1DevExitLat          10 mican't get debug descriptor: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
cannot read device status, Resource temporarily unavailable (11)
can't get debug descriptor: Resource temporarily unavailable
cro seconds
    bU2DevExitLat         512 micro seconds
Device Status:     0x0001
  Self Powered

Bus 006 Device 007: ID 0aec:3050 Neodio Technologies Corp. ND3050 8-in-1 Card Reader
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        16
  idVendor           0x0aec Neodio Technologies Corp.
  idProduct          0x3050 ND3050 8-in-1 Card Reader
  bcdDevice            1.00
  iManufacturer           1 (error)
  iProduct                2 (error)
  iSerial                 3 (error)
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0020
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk-Only
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0

Bus 006 Device 005: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         8
  idVendor           0x051d American Power Conversion
  idProduct          0x0002 Uninterruptible Power Supply
  bcdDevice            1.06
  iManufacturer           3 American Power Conversion
  iProduct                1 Back-UPS XS 700U   FW:924.Z3 .I USB FW:Z3 
  iSerial                 2 3B1828X60578  
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0022
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower               24mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 
      iInterface              0 
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.10
          bCountryCode           33 US
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength    1029
         Report Descriptors: 
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0006  1x 6 bytes
        bInterval              10
Device Status:     0xcan't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
cannot read device status, Resource temporarily unavailable (11)
0000
  (Bus Powered)

Bus 006 Device 003: ID 04e8:330e Samsung Electronics Co., Ltd ML-2950 Series
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x04e8 Samsung Electronics Co., Ltd
  idProduct          0x330e 
  bcdDevice            1.00
  iManufacturer           1 (error)
  iProduct                2 (error)
  iSerial                 3 (error)
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0020
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xc0
      Self Powered
    MaxPower                2mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         7 Printer
      bInterfaceSubClass      1 Printer
      bInterfaceProtocol      2 Bidirectional
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval              10
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval              10

Bus 006 Device 004: ID 046d:c52f Logitech, Inc. Unifying Receiver
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         8
  idVendor           0x046d Logitech, Inc.
  idProduct          0xc52f Unifying Receiver
  bcdDevice           22.00
  iManufacturer           1 Logitech
  iProduct                2 USB Receiver
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x003b
    bNumInterfaces          2
    bConfigurationValue     1
    iConfiguration          4 RQR22.00_B0005
    bmAttributes         0xa0
      (Bus Powered)
      Remote Wakeup
    MaxPower               98mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      1 Boot Interface Subclass
      bInterfaceProtocol      2 Mouse
      iInterface              0 
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.11
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      67
         Report Descriptors: 
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval               2
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints  can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
         1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 
      iInterface              0 
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.11
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      79
         Report Descriptors: 
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0014  1x 20 bytes
        bInterval               2
Device Status:     0x0000
  (Bus Powered)

Bus 006 Device 008: ID 0951:1666 Kingston Technology DataTraveler 100 G3/G4/SE9 G2
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.10
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x0951 Kingston Technology
  idProduct          0x1666 DataTraveler 100 G3/G4/SE9 G2
  bcdDevice            0.01
  iManufacturer           1 Kingston
  iProduct                2 DataTraveler 3.0
  iSerial                 3 60A44C4139D4FF70899506DC
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0020
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              500mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk-Only
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval             255
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval             255
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength       0x0016
  bNumDeviceCaps          2
  USB 2.0 Extension Device Capability:
    bLength                 7
    bDescriptorType        16
    bDevCapabilityType      2
    bmAttributes   0x00000006
      BESL Link Power Management (LPM) Supported
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x00
    wSpeedsSupported   0x000e
      Device can operate at Full Speed (12Mbps)
      Device can operate at High Speed (480Mbps)
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   2
      Lowest fully-functional device speed is High Speed (480Mbps)
    bU1DevExitLat          10 micro seconds
    bU2DevExitLat        2047 micro seconds
Device Status:     0x0000
  (Bus Powered)

Bus 006 Device 006: ID 045b:0209 Hitachi, Ltd 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         1 Single TT
  bMaxPcan't get debug descriptor: Resource temporarily unavailable
acketSize0        64
  idVendor           0x045b Hitachi, Ltd
  idProduct          0x0209 
  bcdDevice            1.00
  iManufacturer           0 
  iProduct                0 
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0019
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0001  1x 1 bytes
        bInterval              12
Hub Descriptor:
  bLength               9
  bDescriptorType      41
  nNbrPorts             4
  wHubCharacteristic 0x0029
    Per-port power switching
    Per-port overcurrent protection
    TT think time 16 FS bits
  bPwrOn2PwrGood       50 * 2 milli seconds
  bHubContrCurrent    100 milli Ampere
  DeviceRemovable    0x00
  PortPwrCtrlMask    0xff
 Hub Port Status:
   Port 1: 0000.0503 highspeed power enable connect
   Port 2: 0000.0100 power
   Port 3: 0000.0100 power
   Port 4: 0000.0100 power
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         0 Full speed (or root) hub
  bMaxPacketSize0        64
  bNumConfigurations      1
Device Status:     0x0001
  Self Powered

Bus 006 Device 002: ID 045b:0209 Hitachi, Ltd 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         1 Single TT
  bMaxPacketSize0        64
  idVendor           0x045b Hitachi, Ltd
  idProduct          0x0209 
  bcdDevice            1.00
  iManufacturer           0 
  iProduct                0 
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0019
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0001  1x 1 bytes
        bInterval              12
Hub Descriptor:
  bLength               9
  bDescriptorType      41
  nNbrPorts             4
  wHubCharacteristic 0x0029
    Per-port power switching
    Per-port overcurrent protection
    TT think time 16 FS bits
  bPwrOn2PwrGood       50 * 2 milli seconds
  bHubContrCurrent    100 milli Ampere
  DeviceRemovable    0x00
  PortPwrCtrlMask    0xff
 Hub Port Status:
   Port 1: 0000.0503 highspeed power enable connect
   Port 2: 0000.0100 power
   Port 3: 0000.010can't get debug descriptor: Resource temporarily unavailable
can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
0 power
   Port 4: 0000.0103 power enable connect
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         0 Full speed (or root) hub
  bMaxPacketSize0        64
  bNumConfigurations      1
Device Status:     0x0001
  Self Powered

Bus 006 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         1 Single TT
  bMaxPacketSize0        64
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0002 2.0 root hub
  bcdDevice            5.03
  iManufacturer           3 Linux 5.3.8-200.fc30.x86_64 xhci-hcd
  iProduct                2 xHCI Host Controller
  iSerial                 1 0000:00:14.0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0019
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval              12
Hub Descriptor:
  bLength              11
  bDescriptorType      41
  nNbrPorts            12
  wHubCharacteristic 0x000a
    No power switching (usb 1.0)
    Per-port overcurrent protection
    TT think time 8 FS bits
  bPwrOn2PwrGood       10 * 2 milli seconds
  bHubContrCurrent      0 milli Ampere
  DeviceRemovable    0x00 0x00
  PortPwrCtrlMask    0xff 0xff
 Hub Port Status:
   Port 1: 0000.0100 power
   Port 2: 0000.0100 power
   Port 3: 0000.0100 power
   Port 4: 0000.0503 highspeed power enable connect
   Port 5: 0000.0503 highspeed power enable connect
   Port 6: 0000.0303 lowspeed power enable connect
   Port 7: 0000.0100 power
   Port 8: 0000.0103 power enable connect
   Port 9: 0000.0100 power
   Port 10: 0000.0100 power
   Port 11: 0000.0100 power
   Port 12: 0000.0100 power
Device Status:     0x0001
  Self Powered

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-23 15:44                                                       ` Andrea Vai
@ 2019-11-25  3:54                                                         ` Ming Lei
  2019-11-25 10:11                                                           ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-11-25  3:54 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

[-- Attachment #1: Type: text/plain, Size: 400 bytes --]

On Sat, Nov 23, 2019 at 04:44:55PM +0100, Andrea Vai wrote:
> Il giorno sab, 23/11/2019 alle 15.28 +0800, Ming Lei ha scritto:
> > 
> > Please post the log of 'lsusb -v', and I will try to make a patch
> > for
> > addressing the issue.
> 
> attached,

Please apply the attached patch, and re-build & install & reboot kernel.

This time, please don't switch io scheduler.

Thanks,
Ming

[-- Attachment #2: usb.patch --]
[-- Type: text/plain, Size: 3616 bytes --]

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 5c9adcaa27ac..eecb46020bfb 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1436,7 +1436,13 @@ static void __blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async,
 	if (unlikely(blk_mq_hctx_stopped(hctx)))
 		return;
 
-	if (!async && !(hctx->flags & BLK_MQ_F_BLOCKING)) {
+	/*
+	 * Some single-queue devices may need to dispatch IO in order
+	 * which was guaranteed for the legacy queue via the big queue
+	 * lock. Now we reply on single hctx->run_work for that.
+	 */
+	if (!async && !(hctx->flags & (BLK_MQ_F_BLOCKING |
+					BLK_MQ_F_STRICT_DISPATCH_ORDER))) {
 		int cpu = get_cpu();
 		if (cpumask_test_cpu(cpu, hctx->cpumask)) {
 			__blk_mq_run_hw_queue(hctx);
@@ -3042,6 +3048,10 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
 	if (!set->ops->get_budget ^ !set->ops->put_budget)
 		return -EINVAL;
 
+	if (set->queue_depth > 1 && (set->flags &
+				BLK_MQ_F_STRICT_DISPATCH_ORDER))
+		return -EINVAL;
+
 	if (set->queue_depth > BLK_MQ_MAX_DEPTH) {
 		pr_info("blk-mq: reduced tag depth to %u\n",
 			BLK_MQ_MAX_DEPTH);
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index d3d237a09a78..563188844143 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -1939,6 +1939,9 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost)
 	shost->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
 	shost->tag_set.flags |=
 		BLK_ALLOC_POLICY_TO_MQ_FLAG(shost->hostt->tag_alloc_policy);
+	if (shost->hostt->strict_dispatch_order)
+		shost->tag_set.flags |= BLK_MQ_F_STRICT_DISPATCH_ORDER;
+
 	shost->tag_set.driver_data = shost;
 
 	return blk_mq_alloc_tag_set(&shost->tag_set);
diff --git a/drivers/usb/storage/scsiglue.c b/drivers/usb/storage/scsiglue.c
index 6737fab94959..77795edad8e8 100644
--- a/drivers/usb/storage/scsiglue.c
+++ b/drivers/usb/storage/scsiglue.c
@@ -661,6 +661,18 @@ static const struct scsi_host_template usb_stor_host_template = {
 	/* we do our own delay after a device or bus reset */
 	.skip_settle_delay =		1,
 
+
+	/*
+	 * Some USB storage, such as Kingston Technology DataTraveler 100
+	 * G3/G4/SE9 G2(ID 0951:1666), requires IO dispatched in the
+	 * sequential order, otherwise IO performance may drop drastically.
+	 *
+	 * can_queue is always 1, so we set .strict_dispatch_order for
+	 * USB mass storage HBA. Another reason is that there can be such
+	 * kind of devices too.
+	 */
+	.strict_dispatch_order =	1,
+
 	/* sysfs device attributes */
 	.sdev_attrs =			sysfs_device_attr_list,
 
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index dc03e059fdff..844539690a27 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -388,6 +388,7 @@ struct blk_mq_ops {
 enum {
 	BLK_MQ_F_SHOULD_MERGE	= 1 << 0,
 	BLK_MQ_F_TAG_SHARED	= 1 << 1,
+	BLK_MQ_F_STRICT_DISPATCH_ORDER	= 1 << 2,
 	BLK_MQ_F_BLOCKING	= 1 << 5,
 	BLK_MQ_F_NO_SCHED	= 1 << 6,
 	BLK_MQ_F_ALLOC_POLICY_START_BIT = 8,
diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h
index d4452d0ea3c7..f932d6fa1a4c 100644
--- a/include/scsi/scsi_host.h
+++ b/include/scsi/scsi_host.h
@@ -442,6 +442,13 @@ struct scsi_host_template {
 	/* True if the low-level driver supports blk-mq only */
 	unsigned force_blk_mq:1;
 
+	/*
+	 * True if the low-level driver needs IO to be dispatched in
+	 * the order provided by legacy IO path. The flag is only
+	 * valid for single queue device.
+	 */
+	unsigned strict_dispatch_order:1;
+
 	/*
 	 * Countdown for host blocking with no commands outstanding.
 	 */

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-25  3:54                                                         ` Ming Lei
@ 2019-11-25 10:11                                                           ` Andrea Vai
  2019-11-25 10:29                                                             ` Ming Lei
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-11-25 10:11 UTC (permalink / raw)
  To: Ming Lei
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

Il giorno lun, 25/11/2019 alle 11.54 +0800, Ming Lei ha scritto:
> On Sat, Nov 23, 2019 at 04:44:55PM +0100, Andrea Vai wrote:
> > Il giorno sab, 23/11/2019 alle 15.28 +0800, Ming Lei ha scritto:
> > > 
> > > Please post the log of 'lsusb -v', and I will try to make a
> patch
> > > for
> > > addressing the issue.
> > 
> > attached,
> 
> Please apply the attached patch, and re-build & install & reboot
> kernel.
> 
> This time, please don't switch io scheduler.

# patch -p1 < usb.patch outputs:

(Stripping trailing CRs from patch; use --binary to disable.)
patching file block/blk-mq.c
Hunk #1 succeeded at 1465 (offset 29 lines).
Hunk #2 succeeded at 3061 (offset 13 lines).
(Stripping trailing CRs from patch; use --binary to disable.)
patching file drivers/scsi/scsi_lib.c
Hunk #1 succeeded at 1902 (offset -37 lines).
(Stripping trailing CRs from patch; use --binary to disable.)
patching file drivers/usb/storage/scsiglue.c
Hunk #1 succeeded at 651 (offset -10 lines).
(Stripping trailing CRs from patch; use --binary to disable.)
patching file include/linux/blk-mq.h
Hunk #1 succeeded at 226 (offset -162 lines).
(Stripping trailing CRs from patch; use --binary to disable.)
patching file include/scsi/scsi_host.h
patch unexpectedly ends in middle of line
patch unexpectedly ends in middle of line

Just to be sure I have to go on, is this correct? Sounds like an error
but I don't know if it is important.

Thanks,
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-25 10:11                                                           ` Andrea Vai
@ 2019-11-25 10:29                                                             ` Ming Lei
  2019-11-25 14:58                                                               ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-11-25 10:29 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

On Mon, Nov 25, 2019 at 11:11:00AM +0100, Andrea Vai wrote:
> Il giorno lun, 25/11/2019 alle 11.54 +0800, Ming Lei ha scritto:
> > On Sat, Nov 23, 2019 at 04:44:55PM +0100, Andrea Vai wrote:
> > > Il giorno sab, 23/11/2019 alle 15.28 +0800, Ming Lei ha scritto:
> > > > 
> > > > Please post the log of 'lsusb -v', and I will try to make a
> > patch
> > > > for
> > > > addressing the issue.
> > > 
> > > attached,
> > 
> > Please apply the attached patch, and re-build & install & reboot
> > kernel.
> > 
> > This time, please don't switch io scheduler.
> 
> # patch -p1 < usb.patch outputs:
> 
> (Stripping trailing CRs from patch; use --binary to disable.)
> patching file block/blk-mq.c
> Hunk #1 succeeded at 1465 (offset 29 lines).
> Hunk #2 succeeded at 3061 (offset 13 lines).
> (Stripping trailing CRs from patch; use --binary to disable.)
> patching file drivers/scsi/scsi_lib.c
> Hunk #1 succeeded at 1902 (offset -37 lines).
> (Stripping trailing CRs from patch; use --binary to disable.)
> patching file drivers/usb/storage/scsiglue.c
> Hunk #1 succeeded at 651 (offset -10 lines).
> (Stripping trailing CRs from patch; use --binary to disable.)
> patching file include/linux/blk-mq.h
> Hunk #1 succeeded at 226 (offset -162 lines).
> (Stripping trailing CRs from patch; use --binary to disable.)
> patching file include/scsi/scsi_host.h
> patch unexpectedly ends in middle of line
> patch unexpectedly ends in middle of line
> 
> Just to be sure I have to go on, is this correct? Sounds like an error
> but I don't know if it is important.

Looks there is small conflict, however it has been fixed by patch, so
it is correct, please go on your test.

Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-25 10:29                                                             ` Ming Lei
@ 2019-11-25 14:58                                                               ` Andrea Vai
  2019-11-25 15:15                                                                 ` Ming Lei
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-11-25 14:58 UTC (permalink / raw)
  To: Ming Lei
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

Il giorno lun, 25/11/2019 alle 18.29 +0800, Ming Lei ha scritto:
> On Mon, Nov 25, 2019 at 11:11:00AM +0100, Andrea Vai wrote:
> > Il giorno lun, 25/11/2019 alle 11.54 +0800, Ming Lei ha scritto:
> > > On Sat, Nov 23, 2019 at 04:44:55PM +0100, Andrea Vai wrote:
> > > > Il giorno sab, 23/11/2019 alle 15.28 +0800, Ming Lei ha
> scritto:
> > > > > 
> > > > > Please post the log of 'lsusb -v', and I will try to make a
> > > patch
> > > > > for
> > > > > addressing the issue.
> > > > 
> > > > attached,
> > > 
> > > Please apply the attached patch, and re-build & install & reboot
> > > kernel.
> > > 
> > > This time, please don't switch io scheduler.
> > 
> > # patch -p1 < usb.patch outputs:
> > 
> > (Stripping trailing CRs from patch; use --binary to disable.)
> > patching file block/blk-mq.c
> > Hunk #1 succeeded at 1465 (offset 29 lines).
> > Hunk #2 succeeded at 3061 (offset 13 lines).
> > (Stripping trailing CRs from patch; use --binary to disable.)
> > patching file drivers/scsi/scsi_lib.c
> > Hunk #1 succeeded at 1902 (offset -37 lines).
> > (Stripping trailing CRs from patch; use --binary to disable.)
> > patching file drivers/usb/storage/scsiglue.c
> > Hunk #1 succeeded at 651 (offset -10 lines).
> > (Stripping trailing CRs from patch; use --binary to disable.)
> > patching file include/linux/blk-mq.h
> > Hunk #1 succeeded at 226 (offset -162 lines).
> > (Stripping trailing CRs from patch; use --binary to disable.)
> > patching file include/scsi/scsi_host.h
> > patch unexpectedly ends in middle of line
> > patch unexpectedly ends in middle of line
> > 
> > Just to be sure I have to go on, is this correct? Sounds like an
> error
> > but I don't know if it is important.
> 
> Looks there is small conflict, however it has been fixed by patch,
> so
> it is correct, please go on your test.

Done, it still fails (2000 seconds or more to copy 1GB) :-(

cat /sys/block/sdf/queue/scheduler outputs:
[mq-deadline] none

What to try next?

Thanks,
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-25 14:58                                                               ` Andrea Vai
@ 2019-11-25 15:15                                                                 ` Ming Lei
  2019-11-25 18:51                                                                   ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-11-25 15:15 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> Il giorno lun, 25/11/2019 alle 18.29 +0800, Ming Lei ha scritto:
> > On Mon, Nov 25, 2019 at 11:11:00AM +0100, Andrea Vai wrote:
> > > Il giorno lun, 25/11/2019 alle 11.54 +0800, Ming Lei ha scritto:
> > > > On Sat, Nov 23, 2019 at 04:44:55PM +0100, Andrea Vai wrote:
> > > > > Il giorno sab, 23/11/2019 alle 15.28 +0800, Ming Lei ha
> > scritto:
> > > > > > 
> > > > > > Please post the log of 'lsusb -v', and I will try to make a
> > > > patch
> > > > > > for
> > > > > > addressing the issue.
> > > > > 
> > > > > attached,
> > > > 
> > > > Please apply the attached patch, and re-build & install & reboot
> > > > kernel.
> > > > 
> > > > This time, please don't switch io scheduler.
> > > 
> > > # patch -p1 < usb.patch outputs:
> > > 
> > > (Stripping trailing CRs from patch; use --binary to disable.)
> > > patching file block/blk-mq.c
> > > Hunk #1 succeeded at 1465 (offset 29 lines).
> > > Hunk #2 succeeded at 3061 (offset 13 lines).
> > > (Stripping trailing CRs from patch; use --binary to disable.)
> > > patching file drivers/scsi/scsi_lib.c
> > > Hunk #1 succeeded at 1902 (offset -37 lines).
> > > (Stripping trailing CRs from patch; use --binary to disable.)
> > > patching file drivers/usb/storage/scsiglue.c
> > > Hunk #1 succeeded at 651 (offset -10 lines).
> > > (Stripping trailing CRs from patch; use --binary to disable.)
> > > patching file include/linux/blk-mq.h
> > > Hunk #1 succeeded at 226 (offset -162 lines).
> > > (Stripping trailing CRs from patch; use --binary to disable.)
> > > patching file include/scsi/scsi_host.h
> > > patch unexpectedly ends in middle of line
> > > patch unexpectedly ends in middle of line
> > > 
> > > Just to be sure I have to go on, is this correct? Sounds like an
> > error
> > > but I don't know if it is important.
> > 
> > Looks there is small conflict, however it has been fixed by patch,
> > so
> > it is correct, please go on your test.
> 
> Done, it still fails (2000 seconds or more to copy 1GB) :-(
> 
> cat /sys/block/sdf/queue/scheduler outputs:
> [mq-deadline] none
> 
> What to try next?

1) cat /sys/kernel/debug/block/$DISK/hctx0/flags

note: replace $DISK with disk name of your usb drive, such as, if it is
/dev/sdb, pass $DISK as sdb.

2) echo 128 > /sys/block/$DISK/queue/nr_requests and run your copy 1GB
test again.


Thanks, 
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-25 15:15                                                                 ` Ming Lei
@ 2019-11-25 18:51                                                                   ` Andrea Vai
  2019-11-26  2:32                                                                     ` Ming Lei
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-11-25 18:51 UTC (permalink / raw)
  To: Ming Lei
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

Il giorno lun, 25/11/2019 alle 23.15 +0800, Ming Lei ha scritto:
> On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> 
> [...]
> 
> > What to try next?
> 
> 1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
result:

alloc_policy=FIFO SHOULD_MERGE|2

> 
> 
> 2) echo 128 > /sys/block/$DISK/queue/nr_requests and run your copy
> 1GB
> test again.

done, and still fails. What to try next?

Thanks,
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-25 18:51                                                                   ` Andrea Vai
@ 2019-11-26  2:32                                                                     ` Ming Lei
  2019-11-26  7:46                                                                       ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-11-26  2:32 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

On Mon, Nov 25, 2019 at 07:51:33PM +0100, Andrea Vai wrote:
> Il giorno lun, 25/11/2019 alle 23.15 +0800, Ming Lei ha scritto:
> > On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> > 
> > [...]
> > 
> > > What to try next?
> > 
> > 1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
> result:
> 
> alloc_policy=FIFO SHOULD_MERGE|2
> 
> > 
> > 
> > 2) echo 128 > /sys/block/$DISK/queue/nr_requests and run your copy
> > 1GB
> > test again.
> 
> done, and still fails. What to try next?

I just run 256M cp test to one USB storage device on patched kernel,
and WRITE data IO is really in ascending order. The filesystem is ext4,
and mount without '-o sync'. From previous discussion, looks that is
exactly your test setting. The order can be observed via the following script:

#!/bin/sh
MAJ=$1
MIN=$2
MAJ=$(( $MAJ << 20 ))
DEV=$(( $MAJ | $MIN ))
/usr/share/bcc/tools/trace -t -C \
  't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args->rwbs, args->sector, args->nr_sector'

$MAJ & $MIN can be retrieved via lsblk for your USB storage disk.

So I think we need to check if the patch is applied correctly first.

If your kernel tree is managed via git, please post 'git diff'.
Otherwise, share us your kernel version, and I will send you one
backported patch on the kernel version.

Meantime, you can collect IO order log via the above script as you did last
time, then send us the log.

Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-26  2:32                                                                     ` Ming Lei
@ 2019-11-26  7:46                                                                       ` Andrea Vai
  2019-11-26  9:15                                                                         ` Ming Lei
  2019-11-27  0:21                                                                         ` Finn Thain
  0 siblings, 2 replies; 104+ messages in thread
From: Andrea Vai @ 2019-11-26  7:46 UTC (permalink / raw)
  To: Ming Lei
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

[-- Attachment #1: Type: text/plain, Size: 2944 bytes --]

Il giorno mar, 26/11/2019 alle 10.32 +0800, Ming Lei ha scritto:
> On Mon, Nov 25, 2019 at 07:51:33PM +0100, Andrea Vai wrote:
> > Il giorno lun, 25/11/2019 alle 23.15 +0800, Ming Lei ha scritto:
> > > On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> > > 
> > > [...]
> > > 
> > > > What to try next?
> > > 
> > > 1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
> > result:
> > 
> > alloc_policy=FIFO SHOULD_MERGE|2
> > 
> > > 
> > > 
> > > 2) echo 128 > /sys/block/$DISK/queue/nr_requests and run your
> copy
> > > 1GB
> > > test again.
> > 
> > done, and still fails. What to try next?
> 
> I just run 256M cp test

I would like to point out that 256MB is a filesize that usually don't
trigger the issue (don't know if it matters, sorry).

Another info I would provide is about another strange behavior I
noticed: yesterday I ran the test two times (as usual with 1GB
filesize) and took 2370s, 1786s, and a third test was going on when I
stopped it. Then I started another set of 100 trials and let them run
tonight, and the first 10 trials were around 1000s, then gradually
decreased to ~300s, and finally settled around 200s with some trials
below 70-80s. This to say, times are extremely variable and for the
first time I noticed a sort of "performance increase" with time.

>  to one USB storage device on patched kernel,
> and WRITE data IO is really in ascending order. The filesystem is
> ext4,
> and mount without '-o sync'. From previous discussion, looks that is
> exactly your test setting. The order can be observed via the
> following script:
> 
> #!/bin/sh
> MAJ=$1
> MIN=$2
> MAJ=$(( $MAJ << 20 ))
> DEV=$(( $MAJ | $MIN ))
> /usr/share/bcc/tools/trace -t -C \
>   't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args-
> >rwbs, args->sector, args->nr_sector'
> 
> $MAJ & $MIN can be retrieved via lsblk for your USB storage disk.
> 
> So I think we need to check if the patch is applied correctly first.
> 
> If your kernel tree is managed via git,
yes it is,

>  please post 'git diff'.
attached. Is it correctly patched? thanks.


> Otherwise, share us your kernel version,
btw, is 5.4.0+

>  and I will send you one
> backported patch on the kernel version.
> 
> Meantime, you can collect IO order log via the above script as you
> did last
> time, then send us the log.

ok, will try; is it just required to run it for a short period of time
(say, some seconds) during the copy, or should I run it before the
beginning (or before the mount?), and terminate it after the end of
the copy? (Please note that in the latter case a large amount of time
(and data, I suppose) would be involved, because, as said, to be sure
the problem triggers I have to use a large file... but we can try to
better understand and tune this. If it can help, you can get an ods
file with the complete statistic at [1] (look at the "prove_nov19"
sheet)).

Thanks,
Andrea

[1]: http://fisica.unipv.it/transfer/kernelstats.zip

[-- Attachment #2: git_diff.txt --]
[-- Type: text/plain, Size: 6424 bytes --]

# git diff
diff --git a/block/blk-mq.c b/block/blk-mq.c
index ec791156e9cc..92d60a5e1d15 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1465,7 +1465,13 @@ static void __blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async,
        if (unlikely(blk_mq_hctx_stopped(hctx)))
                return;
 
-       if (!async && !(hctx->flags & BLK_MQ_F_BLOCKING)) {
+       /*
+        * Some single-queue devices may need to dispatch IO in order
+        * which was guaranteed for the legacy queue via the big queue
+        * lock. Now we reply on single hctx->run_work for that.
+        */
+       if (!async && !(hctx->flags & (BLK_MQ_F_BLOCKING |
+                                       BLK_MQ_F_STRICT_DISPATCH_ORDER))) {
                int cpu = get_cpu();
                if (cpumask_test_cpu(cpu, hctx->cpumask)) {
                        __blk_mq_run_hw_queue(hctx);
@@ -3055,6 +3061,10 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
        if (!set->ops->get_budget ^ !set->ops->put_budget)
                return -EINVAL;
 
+       if (set->queue_depth > 1 && (set->flags &
+                               BLK_MQ_F_STRICT_DISPATCH_ORDER))
+               return -EINVAL;
+
        if (set->queue_depth > BLK_MQ_MAX_DEPTH) {
                pr_info("blk-mq: reduced tag depth to %u\n",
                        BLK_MQ_MAX_DEPTH);
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 91c007d26c1e..f013630275c9 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -1902,6 +1902,9 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost)
        shost->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
        shost->tag_set.flags |=
                BLK_ALLOC_POLICY_TO_MQ_FLAG(shost->hostt->tag_alloc_policy);
+       if (shost->hostt->strict_dispatch_order)
+               shost->tag_set.flags |= BLK_MQ_F_STRICT_DISPATCH_ORDER;
+
        shost->tag_set.driver_data = shost;
 
        return blk_mq_alloc_tag_set(&shost->tag_set);
diff --git a/drivers/usb/storage/scsiglue.c b/drivers/usb/storage/scsiglue.c
index 54a3c8195c96..df1674d7f0fc 100644
--- a/drivers/usb/storage/scsiglue.c
+++ b/drivers/usb/storage/scsiglue.c
@@ -651,6 +651,18 @@ static const struct scsi_host_template usb_stor_host_template = {
        /* we do our own delay after a device or bus reset */
        .skip_settle_delay =            1,
diff --git a/block/blk-mq.c b/block/blk-mq.c
index ec791156e9cc..92d60a5e1d15 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1465,7 +1465,13 @@ static void __blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async,
        if (unlikely(blk_mq_hctx_stopped(hctx)))
                return;
 
-       if (!async && !(hctx->flags & BLK_MQ_F_BLOCKING)) {
+       /*
+        * Some single-queue devices may need to dispatch IO in order
+        * which was guaranteed for the legacy queue via the big queue
+        * lock. Now we reply on single hctx->run_work for that.
+        */
+       if (!async && !(hctx->flags & (BLK_MQ_F_BLOCKING |
+                                       BLK_MQ_F_STRICT_DISPATCH_ORDER))) {
                int cpu = get_cpu();
                if (cpumask_test_cpu(cpu, hctx->cpumask)) {
                        __blk_mq_run_hw_queue(hctx);
@@ -3055,6 +3061,10 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
        if (!set->ops->get_budget ^ !set->ops->put_budget)
                return -EINVAL;
 
+       if (set->queue_depth > 1 && (set->flags &
+                               BLK_MQ_F_STRICT_DISPATCH_ORDER))
+               return -EINVAL;
+
        if (set->queue_depth > BLK_MQ_MAX_DEPTH) {
                pr_info("blk-mq: reduced tag depth to %u\n",
                        BLK_MQ_MAX_DEPTH);
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 91c007d26c1e..f013630275c9 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -1902,6 +1902,9 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost)
        shost->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
        shost->tag_set.flags |=
                BLK_ALLOC_POLICY_TO_MQ_FLAG(shost->hostt->tag_alloc_policy);
+       if (shost->hostt->strict_dispatch_order)
+               shost->tag_set.flags |= BLK_MQ_F_STRICT_DISPATCH_ORDER;
+
        shost->tag_set.driver_data = shost;
 
        return blk_mq_alloc_tag_set(&shost->tag_set);
diff --git a/drivers/usb/storage/scsiglue.c b/drivers/usb/storage/scsiglue.c
index 54a3c8195c96..df1674d7f0fc 100644
--- a/drivers/usb/storage/scsiglue.c
+++ b/drivers/usb/storage/scsiglue.c
@@ -651,6 +651,18 @@ static const struct scsi_host_template usb_stor_host_template = {
        /* we do our own delay after a device or bus reset */
        .skip_settle_delay =            1,
 
+
+       /*
+        * Some USB storage, such as Kingston Technology DataTraveler 100
+        * G3/G4/SE9 G2(ID 0951:1666), requires IO dispatched in the
+        * sequential order, otherwise IO performance may drop drastically.
+        *
+        * can_queue is always 1, so we set .strict_dispatch_order for
+        * USB mass storage HBA. Another reason is that there can be such
+        * kind of devices too.
+        */
+       .strict_dispatch_order =        1,
+
        /* sysfs device attributes */
        .sdev_attrs =                   sysfs_device_attr_list,
 
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index 0bf056de5cc3..89b1c28da36a 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -226,6 +226,7 @@ struct blk_mq_ops {
 enum {
        BLK_MQ_F_SHOULD_MERGE   = 1 << 0,
        BLK_MQ_F_TAG_SHARED     = 1 << 1,
+       BLK_MQ_F_STRICT_DISPATCH_ORDER  = 1 << 2,
        BLK_MQ_F_BLOCKING       = 1 << 5,
        BLK_MQ_F_NO_SCHED       = 1 << 6,
        BLK_MQ_F_ALLOC_POLICY_START_BIT = 8,
diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h
index 31e0d6ca1eba..dbcbc9ef6993 100644
--- a/include/scsi/scsi_host.h
+++ b/include/scsi/scsi_host.h
@@ -442,6 +442,13 @@ struct scsi_host_template {
        /* True if the low-level driver supports blk-mq only */
        unsigned force_blk_mq:1;
 
+       /*
+        * True if the low-level driver needs IO to be dispatched in
+        * the order provided by legacy IO path. The flag is only
+        * valid for single queue device.
+        */
+       unsigned strict_dispatch_order:1;
+
        /*
         * Countdown for host blocking with no commands outstanding.
         */


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-26  7:46                                                                       ` Andrea Vai
@ 2019-11-26  9:15                                                                         ` Ming Lei
  2019-11-26 10:24                                                                           ` Ming Lei
  2019-11-26 11:14                                                                           ` Andrea Vai
  2019-11-27  0:21                                                                         ` Finn Thain
  1 sibling, 2 replies; 104+ messages in thread
From: Ming Lei @ 2019-11-26  9:15 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

On Tue, Nov 26, 2019 at 08:46:07AM +0100, Andrea Vai wrote:
> Il giorno mar, 26/11/2019 alle 10.32 +0800, Ming Lei ha scritto:
> > On Mon, Nov 25, 2019 at 07:51:33PM +0100, Andrea Vai wrote:
> > > Il giorno lun, 25/11/2019 alle 23.15 +0800, Ming Lei ha scritto:
> > > > On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> > > > 
> > > > [...]
> > > > 
> > > > > What to try next?
> > > > 
> > > > 1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
> > > result:
> > > 
> > > alloc_policy=FIFO SHOULD_MERGE|2
> > > 
> > > > 
> > > > 
> > > > 2) echo 128 > /sys/block/$DISK/queue/nr_requests and run your
> > copy
> > > > 1GB
> > > > test again.
> > > 
> > > done, and still fails. What to try next?
> > 
> > I just run 256M cp test
> 
> I would like to point out that 256MB is a filesize that usually don't
> trigger the issue (don't know if it matters, sorry).

OK.

I tested 256M because IO timeout is often triggered in case of
qemu-ehci, and it is a long-term issue. When setting up the disk
via xhci-qemu, the max request size is increased to 1MB from 120KB,
and IO pattern changed too. When the disk is connected via uhci-qemu,
the transfer is too slow(1MB/s) because max endpoint size is too small.

However, I just waited 16min and collected all the 1GB IO log by
connecting disk over uhci-qemu, but the sector of each data IO
is still in order.

> 
> Another info I would provide is about another strange behavior I
> noticed: yesterday I ran the test two times (as usual with 1GB
> filesize) and took 2370s, 1786s, and a third test was going on when I
> stopped it. Then I started another set of 100 trials and let them run
> tonight, and the first 10 trials were around 1000s, then gradually
> decreased to ~300s, and finally settled around 200s with some trials
> below 70-80s. This to say, times are extremely variable and for the
> first time I noticed a sort of "performance increase" with time.

The 'cp' test is buffered IO, can you reproduce it every time by
running copy just after fresh mount on the USB disk?

> 
> >  to one USB storage device on patched kernel,
> > and WRITE data IO is really in ascending order. The filesystem is
> > ext4,
> > and mount without '-o sync'. From previous discussion, looks that is
> > exactly your test setting. The order can be observed via the
> > following script:
> > 
> > #!/bin/sh
> > MAJ=$1
> > MIN=$2
> > MAJ=$(( $MAJ << 20 ))
> > DEV=$(( $MAJ | $MIN ))
> > /usr/share/bcc/tools/trace -t -C \
> >   't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args-
> > >rwbs, args->sector, args->nr_sector'
> > 
> > $MAJ & $MIN can be retrieved via lsblk for your USB storage disk.
> > 
> > So I think we need to check if the patch is applied correctly first.
> > 
> > If your kernel tree is managed via git,
> yes it is,
> 
> >  please post 'git diff'.
> attached. Is it correctly patched? thanks.

Yeah, it should be correct except for the change on __blk_mq_delay_run_hw_queue()
is duplicated.

> 
> 
> > Otherwise, share us your kernel version,
> btw, is 5.4.0+
> 
> >  and I will send you one
> > backported patch on the kernel version.
> > 
> > Meantime, you can collect IO order log via the above script as you
> > did last
> > time, then send us the log.
> 
> ok, will try; is it just required to run it for a short period of time
> (say, some seconds) during the copy, or should I run it before the
> beginning (or before the mount?), and terminate it after the end of
> the copy? (Please note that in the latter case a large amount of time
> (and data, I suppose) would be involved, because, as said, to be sure
> the problem triggers I have to use a large file... but we can try to
> better understand and tune this. If it can help, you can get an ods
> file with the complete statistic at [1] (look at the "prove_nov19"
> sheet)).

The data won't be very big, each line covers 120KB, and ~10K line
is enough for cover 1GB transfer. Then ~300KB compressed file should
hold all the trace.


Thanks, 
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-26  9:15                                                                         ` Ming Lei
@ 2019-11-26 10:24                                                                           ` Ming Lei
  2019-11-26 11:14                                                                           ` Andrea Vai
  1 sibling, 0 replies; 104+ messages in thread
From: Ming Lei @ 2019-11-26 10:24 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

On Tue, Nov 26, 2019 at 05:15:33PM +0800, Ming Lei wrote:
> On Tue, Nov 26, 2019 at 08:46:07AM +0100, Andrea Vai wrote:
> > Il giorno mar, 26/11/2019 alle 10.32 +0800, Ming Lei ha scritto:
> > > On Mon, Nov 25, 2019 at 07:51:33PM +0100, Andrea Vai wrote:
> > > > Il giorno lun, 25/11/2019 alle 23.15 +0800, Ming Lei ha scritto:
> > > > > On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> > > > > 
> > > > > [...]
> > > > > 
> > > > > > What to try next?
> > > > > 
> > > > > 1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
> > > > result:
> > > > 
> > > > alloc_policy=FIFO SHOULD_MERGE|2
> > > > 
> > > > > 
> > > > > 
> > > > > 2) echo 128 > /sys/block/$DISK/queue/nr_requests and run your
> > > copy
> > > > > 1GB
> > > > > test again.
> > > > 
> > > > done, and still fails. What to try next?
> > > 
> > > I just run 256M cp test
> > 
> > I would like to point out that 256MB is a filesize that usually don't
> > trigger the issue (don't know if it matters, sorry).
> 
> OK.
> 
> I tested 256M because IO timeout is often triggered in case of
> qemu-ehci, and it is a long-term issue. When setting up the disk
> via xhci-qemu, the max request size is increased to 1MB from 120KB,
> and IO pattern changed too. When the disk is connected via uhci-qemu,
> the transfer is too slow(1MB/s) because max endpoint size is too small.
> 
> However, I just waited 16min and collected all the 1GB IO log by
> connecting disk over uhci-qemu, but the sector of each data IO
> is still in order.
> 
> > 
> > Another info I would provide is about another strange behavior I
> > noticed: yesterday I ran the test two times (as usual with 1GB
> > filesize) and took 2370s, 1786s, and a third test was going on when I
> > stopped it. Then I started another set of 100 trials and let them run
> > tonight, and the first 10 trials were around 1000s, then gradually
> > decreased to ~300s, and finally settled around 200s with some trials
> > below 70-80s. This to say, times are extremely variable and for the
> > first time I noticed a sort of "performance increase" with time.
> 
> The 'cp' test is buffered IO, can you reproduce it every time by
> running copy just after fresh mount on the USB disk?
> 
> > 
> > >  to one USB storage device on patched kernel,
> > > and WRITE data IO is really in ascending order. The filesystem is
> > > ext4,
> > > and mount without '-o sync'. From previous discussion, looks that is
> > > exactly your test setting. The order can be observed via the
> > > following script:
> > > 
> > > #!/bin/sh
> > > MAJ=$1
> > > MIN=$2
> > > MAJ=$(( $MAJ << 20 ))
> > > DEV=$(( $MAJ | $MIN ))
> > > /usr/share/bcc/tools/trace -t -C \
> > >   't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args-
> > > >rwbs, args->sector, args->nr_sector'
> > > 
> > > $MAJ & $MIN can be retrieved via lsblk for your USB storage disk.
> > > 
> > > So I think we need to check if the patch is applied correctly first.
> > > 
> > > If your kernel tree is managed via git,
> > yes it is,
> > 
> > >  please post 'git diff'.
> > attached. Is it correctly patched? thanks.
> 
> Yeah, it should be correct except for the change on __blk_mq_delay_run_hw_queue()
> is duplicated.
> 
> > 
> > 
> > > Otherwise, share us your kernel version,
> > btw, is 5.4.0+
> > 
> > >  and I will send you one
> > > backported patch on the kernel version.
> > > 
> > > Meantime, you can collect IO order log via the above script as you
> > > did last
> > > time, then send us the log.
> > 
> > ok, will try; is it just required to run it for a short period of time
> > (say, some seconds) during the copy, or should I run it before the
> > beginning (or before the mount?), and terminate it after the end of
> > the copy? (Please note that in the latter case a large amount of time
> > (and data, I suppose) would be involved, because, as said, to be sure
> > the problem triggers I have to use a large file... but we can try to
> > better understand and tune this. If it can help, you can get an ods
> > file with the complete statistic at [1] (look at the "prove_nov19"
> > sheet)).
> 
> The data won't be very big, each line covers 120KB, and ~10K line
> is enough for cover 1GB transfer. Then ~300KB compressed file should
> hold all the trace.

Also use the following trace script this time:

#!/bin/sh

MAJ=$1
MIN=$2
MAJ=$(( $MAJ << 20 ))
DEV=$(( $MAJ | $MIN ))

/usr/share/bcc/tools/trace -t -C \
    't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args->rwbs, args->sector, args->nr_sector' \
    't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d", args->rwbs, args->sector, args->nr_sector'


Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-26  9:15                                                                         ` Ming Lei
  2019-11-26 10:24                                                                           ` Ming Lei
@ 2019-11-26 11:14                                                                           ` Andrea Vai
  2019-11-27  2:05                                                                             ` Ming Lei
  1 sibling, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-11-26 11:14 UTC (permalink / raw)
  To: Ming Lei
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

[-- Attachment #1: Type: text/plain, Size: 3585 bytes --]

Il giorno mar, 26/11/2019 alle 17.15 +0800, Ming Lei ha scritto:
> On Tue, Nov 26, 2019 at 08:46:07AM +0100, Andrea Vai wrote:
> > Il giorno mar, 26/11/2019 alle 10.32 +0800, Ming Lei ha scritto:
> > > On Mon, Nov 25, 2019 at 07:51:33PM +0100, Andrea Vai wrote:
> > > > Il giorno lun, 25/11/2019 alle 23.15 +0800, Ming Lei ha
> scritto:
> > > > > On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> > > > > 
> > > > > [...]
> > > > > 
> > > > > > What to try next?
> > > > > 
> > > > > 1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
> > > > result:
> > > > 
> > > > alloc_policy=FIFO SHOULD_MERGE|2
> > > > 
> > > > > 
> > > > > 
> > > > > 2) echo 128 > /sys/block/$DISK/queue/nr_requests and run
> your
> > > copy
> > > > > 1GB
> > > > > test again.
> > > > 
> > > > done, and still fails. What to try next?
> > > 
> > > I just run 256M cp test
> > 
> > I would like to point out that 256MB is a filesize that usually
> don't
> > trigger the issue (don't know if it matters, sorry).
> 
> OK.
> 
> I tested 256M because IO timeout is often triggered in case of
> qemu-ehci, and it is a long-term issue. When setting up the disk
> via xhci-qemu, the max request size is increased to 1MB from 120KB,
> and IO pattern changed too. When the disk is connected via uhci-
> qemu,
> the transfer is too slow(1MB/s) because max endpoint size is too
> small.
> 
> However, I just waited 16min and collected all the 1GB IO log by
> connecting disk over uhci-qemu, but the sector of each data IO
> is still in order.
> 
> > 
> > Another info I would provide is about another strange behavior I
> > noticed: yesterday I ran the test two times (as usual with 1GB
> > filesize) and took 2370s, 1786s, and a third test was going on
> when I
> > stopped it. Then I started another set of 100 trials and let them
> run
> > tonight, and the first 10 trials were around 1000s, then gradually
> > decreased to ~300s, and finally settled around 200s with some
> trials
> > below 70-80s. This to say, times are extremely variable and for
> the
> > first time I noticed a sort of "performance increase" with time.
> 
> The 'cp' test is buffered IO, can you reproduce it every time by
> running copy just after fresh mount on the USB disk?

yes, every time my test script (attached) mounts, copy, unmount (but I
don't unplug and replug the pendrive each time). Is this enough?

> 
> > 
> > >  to one USB storage device on patched kernel,
> > > and WRITE data IO is really in ascending order. The filesystem
> is
> > > ext4,
> > > and mount without '-o sync'. From previous discussion, looks
> that is
> > > exactly your test setting. The order can be observed via the
> > > following script:
> > > 
> > > #!/bin/sh
> > > MAJ=$1
> > > MIN=$2
> > > MAJ=$(( $MAJ << 20 ))
> > > DEV=$(( $MAJ | $MIN ))
> > > /usr/share/bcc/tools/trace -t -C \
> > >   't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d",
> args-
> > > >rwbs, args->sector, args->nr_sector'
> > > 
> > > $MAJ & $MIN can be retrieved via lsblk for your USB storage
> disk.

ok, so I try:

# lsblk /dev/sdf
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdf      8:80   1 28,8G  0 disk 
└─sdf1   8:81   1 28,8G  0 part 

so I ran your script (the second one, which you sent me in the next
email message) with:

./test_ming 8 80

but it fails to run (terminal output is in attached errors.txt).
What am I doing wrong?

It's still not clear to me if I need to start the trace script and
then the test, or the opposite (or doesn't matter). The above errors
are in the former case (I didn't even start the test, actually)

Thanks,
Andrea

[-- Attachment #2: errors.txt --]
[-- Type: text/plain, Size: 60490 bytes --]

In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:26:
In file included from /lib/modules/5.4.0+/build/include/linux/irqflags.h:16:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/irqflags.h:9:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/nospec-branch.h:314:
/lib/modules/5.4.0+/build/arch/x86/include/asm/segment.h:254:2: error: expected '(' after 'asm'
        alternative_io ("lsl %[seg],%[p]",
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:240:2: note: expanded from macro 'alternative_io'
        asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature)   \
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:27:
In file included from /lib/modules/5.4.0+/build/include/linux/preempt.h:78:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/preempt.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/thread_info.h:38:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/thread_info.h:12:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/page.h:12:
/lib/modules/5.4.0+/build/arch/x86/include/asm/page_64.h:49:2: error: expected '(' after 'asm'
        alternative_call_2(clear_page_orig,
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:256:2: note: expanded from macro 'alternative_call_2'
        asm_inline volatile (ALTERNATIVE_2("call %P[old]", "call %P[new1]", feature1,\
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:27:
In file included from /lib/modules/5.4.0+/build/include/linux/preempt.h:78:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/preempt.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/thread_info.h:38:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/thread_info.h:53:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/cpufeature.h:5:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/processor.h:24:
/lib/modules/5.4.0+/build/arch/x86/include/asm/special_insns.h:205:2: error: expected '(' after 'asm'
        alternative_io(".byte " __stringify(NOP_DS_PREFIX) "; clflush %P0",
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:240:2: note: expanded from macro 'alternative_io'
        asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature)   \
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:27:
In file included from /lib/modules/5.4.0+/build/include/linux/preempt.h:78:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/preempt.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/thread_info.h:38:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/thread_info.h:53:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/cpufeature.h:5:
/lib/modules/5.4.0+/build/arch/x86/include/asm/processor.h:795:2: error: expected '(' after 'asm'
        alternative_input(BASE_PREFETCH, "prefetchnta %P1",
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:221:2: note: expanded from macro 'alternative_input'
        asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature)   \
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:27:
In file included from /lib/modules/5.4.0+/build/include/linux/preempt.h:78:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/preempt.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/thread_info.h:38:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/thread_info.h:53:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/cpufeature.h:5:
/lib/modules/5.4.0+/build/arch/x86/include/asm/processor.h:807:2: error: expected '(' after 'asm'
        alternative_input(BASE_PREFETCH, "prefetchw %P1",
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:221:2: note: expanded from macro 'alternative_input'
        asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature)   \
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:27:
In file included from /lib/modules/5.4.0+/build/include/linux/preempt.h:78:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/preempt.h:7:
/lib/modules/5.4.0+/build/include/linux/thread_info.h:134:2: error: expected '(' after 'asm'
        WARN(1, "Buffer overflow detected (%d < %lu)!\n", size, count);
        ^
/lib/modules/5.4.0+/build/include/asm-generic/bug.h:124:3: note: expanded from macro 'WARN'
                __WARN_printf(TAINT_WARN, format);                      \
                ^
/lib/modules/5.4.0+/build/include/asm-generic/bug.h:93:3: note: expanded from macro '__WARN_printf'
                __WARN_FLAGS(BUGFLAG_NO_CUT_HERE | BUGFLAG_TAINT(taint));\
                ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:79:2: note: expanded from macro '__WARN_FLAGS'
        _BUG_FLAGS(ASM_UD2, BUGFLAG_WARNING|(flags));           \
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:35:2: note: expanded from macro '_BUG_FLAGS'
        asm_inline volatile("1:\t" ins "\n"                             \
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
/lib/modules/5.4.0+/build/include/linux/rcupdate.h:893:2: error: expected '(' after 'asm'
        WARN_ON_ONCE(func != (rcu_callback_t)~0L);
        ^
/lib/modules/5.4.0+/build/include/asm-generic/bug.h:98:3: note: expanded from macro 'WARN_ON_ONCE'
                __WARN_FLAGS(BUGFLAG_ONCE |                     \
                ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:79:2: note: expanded from macro '__WARN_FLAGS'
        _BUG_FLAGS(ASM_UD2, BUGFLAG_WARNING|(flags));           \
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:35:2: note: expanded from macro '_BUG_FLAGS'
        asm_inline volatile("1:\t" ins "\n"                             \
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:15:
In file included from /lib/modules/5.4.0+/build/include/linux/sem.h:5:
In file included from /lib/modules/5.4.0+/build/include/uapi/linux/sem.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/ipc.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/rhashtable-types.h:15:
In file included from /lib/modules/5.4.0+/build/include/linux/workqueue.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/timer.h:6:
/lib/modules/5.4.0+/build/include/linux/ktime.h:171:2: error: expected '(' after 'asm'
        WARN_ON(div < 0);
        ^
/lib/modules/5.4.0+/build/include/asm-generic/bug.h:115:3: note: expanded from macro 'WARN_ON'
                __WARN();                                               \
                ^
/lib/modules/5.4.0+/build/include/asm-generic/bug.h:90:19: note: expanded from macro '__WARN'
#define __WARN()                __WARN_FLAGS(BUGFLAG_TAINT(TAINT_WARN))
                                ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:79:2: note: expanded from macro '__WARN_FLAGS'
        _BUG_FLAGS(ASM_UD2, BUGFLAG_WARNING|(flags));           \
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:35:2: note: expanded from macro '_BUG_FLAGS'
        asm_inline volatile("1:\t" ins "\n"                             \
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:20:
In file included from /lib/modules/5.4.0+/build/include/linux/hrtimer.h:19:
In file included from /lib/modules/5.4.0+/build/include/linux/percpu.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/smp.h:68:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/smp.h:13:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/apic.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/fixmap.h:190:
In file included from /lib/modules/5.4.0+/build/include/asm-generic/fixmap.h:19:
In file included from /lib/modules/5.4.0+/build/include/linux/mm_types.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/uprobes.h:49:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uprobes.h:13:
In file included from /lib/modules/5.4.0+/build/include/linux/notifier.h:16:
/lib/modules/5.4.0+/build/include/linux/srcu.h:179:2: error: expected '(' after 'asm'
        WARN_ON_ONCE(idx & ~0x1);
        ^
/lib/modules/5.4.0+/build/include/asm-generic/bug.h:98:3: note: expanded from macro 'WARN_ON_ONCE'
                __WARN_FLAGS(BUGFLAG_ONCE |                     \
                ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:79:2: note: expanded from macro '__WARN_FLAGS'
        _BUG_FLAGS(ASM_UD2, BUGFLAG_WARNING|(flags));           \
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:35:2: note: expanded from macro '_BUG_FLAGS'
        asm_inline volatile("1:\t" ins "\n"                             \
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:20:
In file included from /lib/modules/5.4.0+/build/include/linux/hrtimer.h:19:
In file included from /lib/modules/5.4.0+/build/include/linux/percpu.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/smp.h:68:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/smp.h:13:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/apic.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/fixmap.h:190:
/lib/modules/5.4.0+/build/include/asm-generic/fixmap.h:38:2: error: expected '(' after 'asm'
        BUG_ON(vaddr >= FIXADDR_TOP || vaddr < FIXADDR_START);
        ^
/lib/modules/5.4.0+/build/include/asm-generic/bug.h:62:57: note: expanded from macro 'BUG_ON'
#define BUG_ON(condition) do { if (unlikely(condition)) BUG(); } while (0)
                                                        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:73:2: note: expanded from macro 'BUG'
        _BUG_FLAGS(ASM_UD2, 0);                                 \
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:35:2: note: expanded from macro '_BUG_FLAGS'
        asm_inline volatile("1:\t" ins "\n"                             \
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:20:
In file included from /lib/modules/5.4.0+/build/include/linux/hrtimer.h:19:
In file included from /lib/modules/5.4.0+/build/include/linux/percpu.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/smp.h:68:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/smp.h:13:
/lib/modules/5.4.0+/build/arch/x86/include/asm/apic.h:107:2: error: expected '(' after 'asm'
        alternative_io("movl %0, %P1", "xchgl %0, %P1", X86_BUG_11AP,
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:240:2: note: expanded from macro 'alternative_io'
        asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature)   \
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:87:11: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
                return (set->sig[3] | set->sig[2] |
                        ^        ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:87:25: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
                return (set->sig[3] | set->sig[2] |
                                      ^        ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:88:4: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
                        set->sig[1] | set->sig[0]) == 0;
                        ^        ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:90:11: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
                return (set->sig[1] | set->sig[0]) == 0;
                        ^        ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:103:11: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
                return  (set1->sig[3] == set2->sig[3]) &&
                         ^         ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:103:27: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
                return  (set1->sig[3] == set2->sig[3]) &&
                                         ^         ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:104:5: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
                        (set1->sig[2] == set2->sig[2]) &&
                         ^         ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:104:21: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
                        (set1->sig[2] == set2->sig[2]) &&
                                         ^         ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:105:5: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
                        (set1->sig[1] == set2->sig[1]) &&
                         ^         ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:105:21: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
                        (set1->sig[1] == set2->sig[1]) &&
                                         ^         ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:108:11: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
                return  (set1->sig[1] == set2->sig[1]) &&
                         ^         ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:108:27: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
                return  (set1->sig[1] == set2->sig[1]) &&
                                         ^         ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:128:8: note: expanded from macro '_SIG_SET_BINOP'
                a3 = a->sig[3]; a2 = a->sig[2];                         \
                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:128:24: note: expanded from macro '_SIG_SET_BINOP'
                a3 = a->sig[3]; a2 = a->sig[2];                         \
                                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:129:8: note: expanded from macro '_SIG_SET_BINOP'
                b3 = b->sig[3]; b2 = b->sig[2];                         \
                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:129:24: note: expanded from macro '_SIG_SET_BINOP'
                b3 = b->sig[3]; b2 = b->sig[2];                         \
                                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:130:3: note: expanded from macro '_SIG_SET_BINOP'
                r->sig[3] = op(a3, b3);                                 \
                ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:131:3: note: expanded from macro '_SIG_SET_BINOP'
                r->sig[2] = op(a2, b2);                                 \
                ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:134:8: note: expanded from macro '_SIG_SET_BINOP'
                a1 = a->sig[1]; b1 = b->sig[1];                         \
                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:134:24: note: expanded from macro '_SIG_SET_BINOP'
                a1 = a->sig[1]; b1 = b->sig[1];                         \
                                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:135:3: note: expanded from macro '_SIG_SET_BINOP'
                r->sig[1] = op(a1, b1);                                 \
                ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:128:8: note: expanded from macro '_SIG_SET_BINOP'
                a3 = a->sig[3]; a2 = a->sig[2];                         \
                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:128:24: note: expanded from macro '_SIG_SET_BINOP'
                a3 = a->sig[3]; a2 = a->sig[2];                         \
                                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:129:8: note: expanded from macro '_SIG_SET_BINOP'
                b3 = b->sig[3]; b2 = b->sig[2];                         \
                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:129:24: note: expanded from macro '_SIG_SET_BINOP'
                b3 = b->sig[3]; b2 = b->sig[2];                         \
                                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:130:3: note: expanded from macro '_SIG_SET_BINOP'
                r->sig[3] = op(a3, b3);                                 \
                ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:131:3: note: expanded from macro '_SIG_SET_BINOP'
                r->sig[2] = op(a2, b2);                                 \
                ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:134:8: note: expanded from macro '_SIG_SET_BINOP'
                a1 = a->sig[1]; b1 = b->sig[1];                         \
                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:134:24: note: expanded from macro '_SIG_SET_BINOP'
                a1 = a->sig[1]; b1 = b->sig[1];                         \
                                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:135:3: note: expanded from macro '_SIG_SET_BINOP'
                r->sig[1] = op(a1, b1);                                 \
                ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:128:8: note: expanded from macro '_SIG_SET_BINOP'
                a3 = a->sig[3]; a2 = a->sig[2];                         \
                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:128:24: note: expanded from macro '_SIG_SET_BINOP'
                a3 = a->sig[3]; a2 = a->sig[2];                         \
                                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:129:8: note: expanded from macro '_SIG_SET_BINOP'
                b3 = b->sig[3]; b2 = b->sig[2];                         \
                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:129:24: note: expanded from macro '_SIG_SET_BINOP'
                b3 = b->sig[3]; b2 = b->sig[2];                         \
                                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:130:3: note: expanded from macro '_SIG_SET_BINOP'
                r->sig[3] = op(a3, b3);                                 \
                ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:131:3: note: expanded from macro '_SIG_SET_BINOP'
                r->sig[2] = op(a2, b2);                                 \
                ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:134:8: note: expanded from macro '_SIG_SET_BINOP'
                a1 = a->sig[1]; b1 = b->sig[1];                         \
                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:134:24: note: expanded from macro '_SIG_SET_BINOP'
                a1 = a->sig[1]; b1 = b->sig[1];                         \
                                     ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:135:3: note: expanded from macro '_SIG_SET_BINOP'
                r->sig[1] = op(a1, b1);                                 \
                ^      ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:177:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_OP(signotset, _sig_not)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:164:27: note: expanded from macro '_SIG_SET_OP'
        case 4: set->sig[3] = op(set->sig[3]);                          \
                                 ^        ~
/lib/modules/5.4.0+/build/include/linux/signal.h:176:24: note: expanded from macro '_sig_not'
#define _sig_not(x)     (~(x))
                           ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:177:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_OP(signotset, _sig_not)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:164:10: note: expanded from macro '_SIG_SET_OP'
        case 4: set->sig[3] = op(set->sig[3]);                          \
                ^        ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:177:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_OP(signotset, _sig_not)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:165:20: note: expanded from macro '_SIG_SET_OP'
                set->sig[2] = op(set->sig[2]);                          \
                                 ^        ~
/lib/modules/5.4.0+/build/include/linux/signal.h:176:24: note: expanded from macro '_sig_not'
#define _sig_not(x)     (~(x))
                           ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:177:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_OP(signotset, _sig_not)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:165:3: note: expanded from macro '_SIG_SET_OP'
                set->sig[2] = op(set->sig[2]);                          \
                ^        ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:177:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_OP(signotset, _sig_not)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:167:27: note: expanded from macro '_SIG_SET_OP'
        case 2: set->sig[1] = op(set->sig[1]);                          \
                                 ^        ~
/lib/modules/5.4.0+/build/include/linux/signal.h:176:24: note: expanded from macro '_sig_not'
#define _sig_not(x)     (~(x))
                           ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:177:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_OP(signotset, _sig_not)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:167:10: note: expanded from macro '_SIG_SET_OP'
        case 2: set->sig[1] = op(set->sig[1]);                          \
                ^        ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:188:10: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
        case 2: set->sig[1] = 0;
                ^        ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:201:10: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
        case 2: set->sig[1] = -1;
                ^        ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:232:10: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
        case 2: set->sig[1] = 0;
                ^        ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:244:10: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
        case 2: set->sig[1] = -1;
                ^        ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
        unsigned long sig[_NSIG_WORDS];
        ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:12:
/lib/modules/5.4.0+/build/arch/x86/include/asm/smap.h:47:2: error: expected '(' after 'asm'
        alternative("", __ASM_CLAC, X86_FEATURE_SMAP);
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:204:2: note: expanded from macro 'alternative'
        asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory")
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:12:
/lib/modules/5.4.0+/build/arch/x86/include/asm/smap.h:53:2: error: expected '(' after 'asm'
        alternative("", __ASM_STAC, X86_FEATURE_SMAP);
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:204:2: note: expanded from macro 'alternative'
        asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory")
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:694:
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess_64.h:37:2: error: expected '(' after 'asm'
        alternative_call_2(copy_user_generic_unrolled,
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:256:2: note: expanded from macro 'alternative_call_2'
        asm_inline volatile (ALTERNATIVE_2("call %P[old]", "call %P[new1]", feature1,\
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:694:
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess_64.h:74:3: error: expected '(' after 'asm'
                __uaccess_begin_nospec();
                ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:125:2: note: expanded from macro '__uaccess_begin_nospec'
        barrier_nospec();               \
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/barrier.h:52:26: note: expanded from macro 'barrier_nospec'
#define barrier_nospec() alternative("", "lfence", X86_FEATURE_LFENCE_RDTSC)
                         ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:204:2: note: expanded from macro 'alternative'
        asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory")
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:694:
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess_64.h:80:3: error: expected '(' after 'asm'
                __uaccess_begin_nospec();
                ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:125:2: note: expanded from macro '__uaccess_begin_nospec'
        barrier_nospec();               \
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/barrier.h:52:26: note: expanded from macro 'barrier_nospec'
#define barrier_nospec() alternative("", "lfence", X86_FEATURE_LFENCE_RDTSC)
                         ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:204:2: note: expanded from macro 'alternative'
        asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory")
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:694:
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess_64.h:86:3: error: expected '(' after 'asm'
                __uaccess_begin_nospec();
                ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:125:2: note: expanded from macro '__uaccess_begin_nospec'
        barrier_nospec();               \
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/barrier.h:52:26: note: expanded from macro 'barrier_nospec'
#define barrier_nospec() alternative("", "lfence", X86_FEATURE_LFENCE_RDTSC)
                         ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:204:2: note: expanded from macro 'alternative'
        asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory")
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:694:
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess_64.h:92:3: error: expected '(' after 'asm'
                __uaccess_begin_nospec();
                ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:125:2: note: expanded from macro '__uaccess_begin_nospec'
        barrier_nospec();               \
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/barrier.h:52:26: note: expanded from macro 'barrier_nospec'
#define barrier_nospec() alternative("", "lfence", X86_FEATURE_LFENCE_RDTSC)
                         ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:204:2: note: expanded from macro 'alternative'
        asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory")
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:694:
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess_64.h:98:3: error: expected '(' after 'asm'
                __uaccess_begin_nospec();
                ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:125:2: note: expanded from macro '__uaccess_begin_nospec'
        barrier_nospec();               \
        ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/barrier.h:52:26: note: expanded from macro 'barrier_nospec'
#define barrier_nospec() alternative("", "lfence", X86_FEATURE_LFENCE_RDTSC)
                         ^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:204:2: note: expanded from macro 'alternative'
        asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory")
        ^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
                       ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
49 warnings and 20 errors generated.
Failed to compile BPF text

[-- Attachment #3: test --]
[-- Type: application/x-shellscript, Size: 1080 bytes --]

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-26  7:46                                                                       ` Andrea Vai
  2019-11-26  9:15                                                                         ` Ming Lei
@ 2019-11-27  0:21                                                                         ` Finn Thain
  2019-11-27  8:14                                                                           ` AW: " Schmid, Carsten
  2019-11-28 17:10                                                                           ` Andrea Vai
  1 sibling, 2 replies; 104+ messages in thread
From: Finn Thain @ 2019-11-27  0:21 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Ming Lei, Damien Le Moal, Alan Stern, Jens Axboe,
	Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

On Tue, 26 Nov 2019, Andrea Vai wrote:

> Then I started another set of 100 trials and let them run tonight, and 
> the first 10 trials were around 1000s, then gradually decreased to 
> ~300s, and finally settled around 200s with some trials below 70-80s. 
> This to say, times are extremely variable and for the first time I 
> noticed a sort of "performance increase" with time.
> 

The sheer volume of testing (probably some terabytes by now) would 
exercise the wear leveling algorithm in the FTL.

This in itself seems unlikely to improve performance significantly. But if 
the flash memory came from a bad batch, perhaps it would have that effect.

To find out, someone may need to source another (genuine) Kingston 
DataTraveller device.

-- 

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-26 11:14                                                                           ` Andrea Vai
@ 2019-11-27  2:05                                                                             ` Ming Lei
  2019-11-27  9:39                                                                               ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-11-27  2:05 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

On Tue, Nov 26, 2019 at 12:14:19PM +0100, Andrea Vai wrote:
> Il giorno mar, 26/11/2019 alle 17.15 +0800, Ming Lei ha scritto:
> > On Tue, Nov 26, 2019 at 08:46:07AM +0100, Andrea Vai wrote:
> > > Il giorno mar, 26/11/2019 alle 10.32 +0800, Ming Lei ha scritto:
> > > > On Mon, Nov 25, 2019 at 07:51:33PM +0100, Andrea Vai wrote:
> > > > > Il giorno lun, 25/11/2019 alle 23.15 +0800, Ming Lei ha
> > scritto:
> > > > > > On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> > > > > > 
> > > > > > [...]
> > > > > > 
> > > > > > > What to try next?
> > > > > > 
> > > > > > 1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
> > > > > result:
> > > > > 
> > > > > alloc_policy=FIFO SHOULD_MERGE|2
> > > > > 
> > > > > > 
> > > > > > 
> > > > > > 2) echo 128 > /sys/block/$DISK/queue/nr_requests and run
> > your
> > > > copy
> > > > > > 1GB
> > > > > > test again.
> > > > > 
> > > > > done, and still fails. What to try next?
> > > > 
> > > > I just run 256M cp test
> > > 
> > > I would like to point out that 256MB is a filesize that usually
> > don't
> > > trigger the issue (don't know if it matters, sorry).
> > 
> > OK.
> > 
> > I tested 256M because IO timeout is often triggered in case of
> > qemu-ehci, and it is a long-term issue. When setting up the disk
> > via xhci-qemu, the max request size is increased to 1MB from 120KB,
> > and IO pattern changed too. When the disk is connected via uhci-
> > qemu,
> > the transfer is too slow(1MB/s) because max endpoint size is too
> > small.
> > 
> > However, I just waited 16min and collected all the 1GB IO log by
> > connecting disk over uhci-qemu, but the sector of each data IO
> > is still in order.
> > 
> > > 
> > > Another info I would provide is about another strange behavior I
> > > noticed: yesterday I ran the test two times (as usual with 1GB
> > > filesize) and took 2370s, 1786s, and a third test was going on
> > when I
> > > stopped it. Then I started another set of 100 trials and let them
> > run
> > > tonight, and the first 10 trials were around 1000s, then gradually
> > > decreased to ~300s, and finally settled around 200s with some
> > trials
> > > below 70-80s. This to say, times are extremely variable and for
> > the
> > > first time I noticed a sort of "performance increase" with time.
> > 
> > The 'cp' test is buffered IO, can you reproduce it every time by
> > running copy just after fresh mount on the USB disk?
> 
> yes, every time my test script (attached) mounts, copy, unmount (but I
> don't unplug and replug the pendrive each time). Is this enough?
> 
> > 
> > > 
> > > >  to one USB storage device on patched kernel,
> > > > and WRITE data IO is really in ascending order. The filesystem
> > is
> > > > ext4,
> > > > and mount without '-o sync'. From previous discussion, looks
> > that is
> > > > exactly your test setting. The order can be observed via the
> > > > following script:
> > > > 
> > > > #!/bin/sh
> > > > MAJ=$1
> > > > MIN=$2
> > > > MAJ=$(( $MAJ << 20 ))
> > > > DEV=$(( $MAJ | $MIN ))
> > > > /usr/share/bcc/tools/trace -t -C \
> > > >   't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d",
> > args-
> > > > >rwbs, args->sector, args->nr_sector'
> > > > 
> > > > $MAJ & $MIN can be retrieved via lsblk for your USB storage
> > disk.
> 
> ok, so I try:
> 
> # lsblk /dev/sdf
> NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
> sdf      8:80   1 28,8G  0 disk 
> └─sdf1   8:81   1 28,8G  0 part 
> 
> so I ran your script (the second one, which you sent me in the next
> email message) with:
> 
> ./test_ming 8 80
> 
> but it fails to run (terminal output is in attached errors.txt).
> What am I doing wrong?
> 
> It's still not clear to me if I need to start the trace script and
> then the test, or the opposite (or doesn't matter). The above errors
> are in the former case (I didn't even start the test, actually)
> 
> Thanks,
> Andrea

> In file included from /virtual/main.c:2:
> In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
> In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
> In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
> In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
> In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:26:
> In file included from /lib/modules/5.4.0+/build/include/linux/irqflags.h:16:
> In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/irqflags.h:9:
> In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/nospec-branch.h:314:
> /lib/modules/5.4.0+/build/arch/x86/include/asm/segment.h:254:2: error: expected '(' after 'asm'
>         alternative_io ("lsl %[seg],%[p]",
>         ^
> /lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:240:2: note: expanded from macro 'alternative_io'
>         asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature)   \
>         ^
> /lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
> #define asm_inline asm __inline
>                        ^
> In file included from /virtual/main.c:2:
> In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
> In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
> In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
> In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
> In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:27:
> In file included from /lib/modules/5.4.0+/build/include/linux/preempt.h:78:
> In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/preempt.h:7:
> In file included from /lib/modules/5.4.0+/build/include/linux/thread_info.h:38:
> In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/thread_info.h:12:
> In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/page.h:12:
> /lib/modules/5.4.0+/build/arch/x86/include/asm/page_64.h:49:2: error: expected '(' after 'asm'
>         alternative_call_2(clear_page_orig,
>         ^
> /lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:256:2: note: expanded from macro 'alternative_call_2'
>         asm_inline volatile (ALTERNATIVE_2("call %P[old]", "call %P[new1]", feature1,\
>         ^
> /lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
> #define asm_inline asm __inline


It can be workaround via the following change:

/lib/modules/5.4.0+/build/include/generated/autoconf.h:

//#define CONFIG_CC_HAS_ASM_INLINE 1


Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-27  0:21                                                                         ` Finn Thain
@ 2019-11-27  8:14                                                                           ` Schmid, Carsten
  2019-11-27 21:49                                                                             ` Finn Thain
  2019-11-28  7:46                                                                             ` Andrea Vai
  2019-11-28 17:10                                                                           ` Andrea Vai
  1 sibling, 2 replies; 104+ messages in thread
From: Schmid, Carsten @ 2019-11-27  8:14 UTC (permalink / raw)
  To: Finn Thain, Andrea Vai
  Cc: Ming Lei, Damien Le Moal, Alan Stern, Jens Axboe,
	Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

> 
> > Then I started another set of 100 trials and let them run tonight, and
> > the first 10 trials were around 1000s, then gradually decreased to
> > ~300s, and finally settled around 200s with some trials below 70-80s.
> > This to say, times are extremely variable and for the first time I
> > noticed a sort of "performance increase" with time.
> >
> 
> The sheer volume of testing (probably some terabytes by now) would
> exercise the wear leveling algorithm in the FTL.
> 
But with "old kernel" the copy operation still is "fast", as far as i understood.
If FTL (e.g. wear leveling) would slow down, we would see that also in
the old kernel, right?

Andrea, can you confirm that the same device used with the old fast
kernel is still fast today?

BR
Carsten

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-27  2:05                                                                             ` Ming Lei
@ 2019-11-27  9:39                                                                               ` Andrea Vai
  2019-11-27 13:08                                                                                 ` Ming Lei
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-11-27  9:39 UTC (permalink / raw)
  To: Ming Lei
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

[-- Attachment #1: Type: text/plain, Size: 503 bytes --]

Il giorno mer, 27/11/2019 alle 10.05 +0800, Ming Lei ha scritto:
> 
> 
> It can be workaround via the following change:
> 
> /lib/modules/5.4.0+/build/include/generated/autoconf.h:
> 
> //#define CONFIG_CC_HAS_ASM_INLINE 1

Thanks, it worked, trace attached. Produced by: start the trace script
(with the pendrive already plugged), wait some seconds, run the test
(1 trial, 1 GB), wait for the test to finish, stop the trace.

The copy took 2659 seconds, roughly as already seen before.

Thanks,
Andrea

[-- Attachment #2: log_ming.zip --]
[-- Type: application/zip, Size: 111658 bytes --]

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-27  9:39                                                                               ` Andrea Vai
@ 2019-11-27 13:08                                                                                 ` Ming Lei
  2019-11-27 15:01                                                                                   ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-11-27 13:08 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

On Wed, Nov 27, 2019 at 10:39:40AM +0100, Andrea Vai wrote:
> Il giorno mer, 27/11/2019 alle 10.05 +0800, Ming Lei ha scritto:
> > 
> > 
> > It can be workaround via the following change:
> > 
> > /lib/modules/5.4.0+/build/include/generated/autoconf.h:
> > 
> > //#define CONFIG_CC_HAS_ASM_INLINE 1
> 
> Thanks, it worked, trace attached. Produced by: start the trace script
> (with the pendrive already plugged), wait some seconds, run the test
> (1 trial, 1 GB), wait for the test to finish, stop the trace.
> 
> The copy took 2659 seconds, roughly as already seen before.

Thanks for collecting the log.

From the log, some of write IOs are out-of-order, such as, the 1st one
is 378880.

16.41240 2   266     266     kworker/2:1H    block_rq_issue   b'W' 370656 240
16.41961 3   485     485     kworker/3:1H    block_rq_issue   b'W' 378880 240
16.73729 2   266     266     kworker/2:1H    block_rq_issue   b'W' 370896 240
17.71161 2   266     266     kworker/2:1H    block_rq_issue   b'W' 379120 240
18.02344 2   266     266     kworker/2:1H    block_rq_issue   b'W' 371136 240
18.94314 3   485     485     kworker/3:1H    block_rq_issue   b'W' 379360 240
19.25624 2   266     266     kworker/2:1H    block_rq_issue   b'W' 371376 240

IO latency is increased a lot since the 1st out-of-order request(usb
storage HBA is single queue depth, one request can be issued only if 
the previous issued request is completed).

The reason is that there are two kind of tasks which inserts rq to device.
One is the 'cp' process, the other is kworker/u8:*.  The out-of-order
happens during the two task's interleaving.

Under such situation, I believe that the old legacy IO path may not
guarantee the order too. In blk_queue_bio(), after get_request()
allocates one request, the queue lock is released.  And request is
actually inserted & issued from blk_flush_plug_list() under the
branch of 'if (plug)'. If requests are from two tasks, then request
is inserted/issued from two plug list, and no order can be guaranteed.

In my test, except for several requests from the beginning, all other
requests are inserted via the kworker thread(guess it is writeback wq),
that is why I can't observe the issue in my test.

As Schmid suggested, you may run the same test on old kernel with
legacy io path, and see if the performance is still good.

Also, could you share the following info about your machine? So that
I can build my VM guest in this setting for reproducing your situation
(requests are inserted from two types of threads).

- lscpu
- free -h
- lsblk -d $USB_DISK
- exact commands for mount the disk, and running the copy operation

Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-27 13:08                                                                                 ` Ming Lei
@ 2019-11-27 15:01                                                                                   ` Andrea Vai
  0 siblings, 0 replies; 104+ messages in thread
From: Andrea Vai @ 2019-11-27 15:01 UTC (permalink / raw)
  To: Ming Lei
  Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list

[-- Attachment #1: Type: text/plain, Size: 3496 bytes --]

Il giorno mer, 27/11/2019 alle 21.08 +0800, Ming Lei ha scritto:
> On Wed, Nov 27, 2019 at 10:39:40AM +0100, Andrea Vai wrote:
> > Il giorno mer, 27/11/2019 alle 10.05 +0800, Ming Lei ha scritto:
> > > 
> > > 
> > > It can be workaround via the following change:
> > > 
> > > /lib/modules/5.4.0+/build/include/generated/autoconf.h:
> > > 
> > > //#define CONFIG_CC_HAS_ASM_INLINE 1
> > 
> > Thanks, it worked, trace attached. Produced by: start the trace
> script
> > (with the pendrive already plugged), wait some seconds, run the
> test
> > (1 trial, 1 GB), wait for the test to finish, stop the trace.
> > 
> > The copy took 2659 seconds, roughly as already seen before.
> 
> Thanks for collecting the log.
> 
> From the log, some of write IOs are out-of-order, such as, the 1st
> one
> is 378880.
> 
> 16.41240 2   266     266     kworker/2:1H    block_rq_issue   b'W'
> 370656 240
> 16.41961 3   485     485     kworker/3:1H    block_rq_issue   b'W'
> 378880 240
> 16.73729 2   266     266     kworker/2:1H    block_rq_issue   b'W'
> 370896 240
> 17.71161 2   266     266     kworker/2:1H    block_rq_issue   b'W'
> 379120 240
> 18.02344 2   266     266     kworker/2:1H    block_rq_issue   b'W'
> 371136 240
> 18.94314 3   485     485     kworker/3:1H    block_rq_issue   b'W'
> 379360 240
> 19.25624 2   266     266     kworker/2:1H    block_rq_issue   b'W'
> 371376 240
> 
> IO latency is increased a lot since the 1st out-of-order request(usb
> storage HBA is single queue depth, one request can be issued only
> if 
> the previous issued request is completed).
> 
> The reason is that there are two kind of tasks which inserts rq to
> device.
> One is the 'cp' process, the other is kworker/u8:*.  The out-of-
> order
> happens during the two task's interleaving.
> 
> Under such situation, I believe that the old legacy IO path may not
> guarantee the order too. In blk_queue_bio(), after get_request()
> allocates one request, the queue lock is released.  And request is
> actually inserted & issued from blk_flush_plug_list() under the
> branch of 'if (plug)'. If requests are from two tasks, then request
> is inserted/issued from two plug list, and no order can be
> guaranteed.
> 
> In my test, except for several requests from the beginning, all
> other
> requests are inserted via the kworker thread(guess it is writeback
> wq),
> that is why I can't observe the issue in my test.
> 
> As Schmid suggested, you may run the same test on old kernel with
> legacy io path, and see if the performance is still good.
> 
> Also, could you share the following info about your machine? So that
> I can build my VM guest in this setting for reproducing your
> situation
> (requests are inserted from two types of threads).
> 
> - lscpu
attached,

> - free -h
              total        used        free      shared  buff/cache   available
Mem:           23Gi       4,2Gi        11Gi       448Mi       7,0Gi        18Gi
Swap:         3,7Gi          0B       3,7Gi

> - lsblk -d $USB_DISK

NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdg    8:96   1 28,8G  0 disk 


> - exact commands for mount the disk, and running the copy operation

I attached the whole script to this thread, I attach it again to this
message and copy the relevant lines here:

  mount UUID=$uuid /mnt/pendrive 2>&1 |tee -a $logfile
  SECONDS=0
  cp $testfile /mnt/pendrive 2>&1 |tee -a $logfile
  umount /mnt/pendrive 2>&1 |tee -a $logfile

Meanwhile, I am going on with the further tests as suggested

Thanks,
Andrea

[-- Attachment #2: lscpu.txt --]
[-- Type: text/plain, Size: 1371 bytes --]

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       39 bits physical, 48 bits virtual
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               60
Model name:          Intel(R) Core(TM) i5-4430 CPU @ 3.00GHz
Stepping:            3
CPU MHz:             1674.727
CPU max MHz:         3200,0000
CPU min MHz:         800,0000
BogoMIPS:            5986.16
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            6144K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts md_clear flush_l1d

[-- Attachment #3: test --]
[-- Type: application/x-shellscript, Size: 1137 bytes --]

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-27  8:14                                                                           ` AW: " Schmid, Carsten
@ 2019-11-27 21:49                                                                             ` Finn Thain
  2019-11-28  7:46                                                                             ` Andrea Vai
  1 sibling, 0 replies; 104+ messages in thread
From: Finn Thain @ 2019-11-27 21:49 UTC (permalink / raw)
  To: Schmid, Carsten
  Cc: Andrea Vai, Ming Lei, Damien Le Moal, Alan Stern, Jens Axboe,
	Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

On Wed, 27 Nov 2019, Schmid, Carsten wrote:

> > 
> > The sheer volume of testing (probably some terabytes by now) would 
> > exercise the wear leveling algorithm in the FTL.
> > 
> But with "old kernel" the copy operation still is "fast", as far as i 
> understood. If FTL (e.g. wear leveling) would slow down, we would see 
> that also in the old kernel, right?
> 
> Andrea, can you confirm that the same device used with the old fast 
> kernel is still fast today?

You seem to be saying we should optimize the kernel for a pathological 
use-case merely because it used to be fast before the blk-mq conversion. 
That makes no sense to me. I suppose you have information that I don't.

I assume that your employer (and the other corporations involved in this) 
have plenty of regression test results from a variety of flash hardware to 
show that the regression is real and the device is not pathological.

I'm not privy to any of that information so I will shut up and leave you 
guys to it.

-- 

> > This in itself seems unlikely to improve performance significantly. 
> > But if the flash memory came from a bad batch, perhaps it would have 
> > that effect.
> > 
> > To find out, someone may need to source another (genuine) Kingston 
> > DataTraveller device.
> > 

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-27  8:14                                                                           ` AW: " Schmid, Carsten
  2019-11-27 21:49                                                                             ` Finn Thain
@ 2019-11-28  7:46                                                                             ` Andrea Vai
  2019-11-28  8:12                                                                               ` AW: " Schmid, Carsten
  2019-11-28  9:17                                                                               ` Ming Lei
  1 sibling, 2 replies; 104+ messages in thread
From: Andrea Vai @ 2019-11-28  7:46 UTC (permalink / raw)
  To: Schmid, Carsten, Finn Thain
  Cc: Ming Lei, Damien Le Moal, Alan Stern, Jens Axboe,
	Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

Il giorno mer, 27/11/2019 alle 08.14 +0000, Schmid, Carsten ha
scritto:
> > 
> > > Then I started another set of 100 trials and let them run
> tonight, and
> > > the first 10 trials were around 1000s, then gradually decreased
> to
> > > ~300s, and finally settled around 200s with some trials below
> 70-80s.
> > > This to say, times are extremely variable and for the first time
> I
> > > noticed a sort of "performance increase" with time.
> > >
> > 
> > The sheer volume of testing (probably some terabytes by now) would
> > exercise the wear leveling algorithm in the FTL.
> > 
> But with "old kernel" the copy operation still is "fast", as far as
> i understood.
> If FTL (e.g. wear leveling) would slow down, we would see that also
> in
> the old kernel, right?
> 
> Andrea, can you confirm that the same device used with the old fast
> kernel is still fast today?

Yes, it is still fast. Just ran a 100 trials test and got an average
of 70 seconds with standard deviation = 6 seconds, aligned with the
past values of the same kernel.

Thanks,
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* AW: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-28  7:46                                                                             ` Andrea Vai
@ 2019-11-28  8:12                                                                               ` Schmid, Carsten
  2019-11-28 11:40                                                                                 ` Andrea Vai
  2019-11-28 17:39                                                                                 ` Alan Stern
  2019-11-28  9:17                                                                               ` Ming Lei
  1 sibling, 2 replies; 104+ messages in thread
From: Schmid, Carsten @ 2019-11-28  8:12 UTC (permalink / raw)
  To: Andrea Vai, Finn Thain
  Cc: Ming Lei, Damien Le Moal, Alan Stern, Jens Axboe,
	Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

> > > The sheer volume of testing (probably some terabytes by now) would
> > > exercise the wear leveling algorithm in the FTL.
> > >
> > But with "old kernel" the copy operation still is "fast", as far as
> > i understood.
> > If FTL (e.g. wear leveling) would slow down, we would see that also
> > in
> > the old kernel, right?
> >
> > Andrea, can you confirm that the same device used with the old fast
> > kernel is still fast today?
> 
> Yes, it is still fast. Just ran a 100 trials test and got an average
> of 70 seconds with standard deviation = 6 seconds, aligned with the
> past values of the same kernel.
> 
> Thanks,
> Andrea
I have been involved in several benchmarkings of flash devices in the past.
So what we see here is definitely not a device issue regarding wear leveling.

I wanted to prevent all of you going into the wrong direction, that's why
i wanted Andrea to confirm that it's not a matter of the flash device.

There are so much items involved into benchmarking flash devices.
But Andrea's observations with factors of 10-30 times slow down
i have never seen before.

I assume the only thing that you change between the benchmarks
is the kernel (and the modules, of course), right, Andrea?
Then we can rule out cache settings which massively can impact
benchmarks.

The only thing that makes sense from my POV is:
- collect traces with the kernel before mentioned commit (fast)
- apply patch in doubt
- again collect traces (slow)
- compare the traces

Then we should be able to see the difference(s).
Unfortunately i'm not an expert on the SCSI and USB kernel stuff
involved here. Else i would try to understand what happens and
give you some hints.

BR
Carsten

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-28  7:46                                                                             ` Andrea Vai
  2019-11-28  8:12                                                                               ` AW: " Schmid, Carsten
@ 2019-11-28  9:17                                                                               ` Ming Lei
  2019-11-28 17:34                                                                                 ` Andrea Vai
  1 sibling, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-11-28  9:17 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

On Thu, Nov 28, 2019 at 08:46:57AM +0100, Andrea Vai wrote:
> Il giorno mer, 27/11/2019 alle 08.14 +0000, Schmid, Carsten ha
> scritto:
> > > 
> > > > Then I started another set of 100 trials and let them run
> > tonight, and
> > > > the first 10 trials were around 1000s, then gradually decreased
> > to
> > > > ~300s, and finally settled around 200s with some trials below
> > 70-80s.
> > > > This to say, times are extremely variable and for the first time
> > I
> > > > noticed a sort of "performance increase" with time.
> > > >
> > > 
> > > The sheer volume of testing (probably some terabytes by now) would
> > > exercise the wear leveling algorithm in the FTL.
> > > 
> > But with "old kernel" the copy operation still is "fast", as far as
> > i understood.
> > If FTL (e.g. wear leveling) would slow down, we would see that also
> > in
> > the old kernel, right?
> > 
> > Andrea, can you confirm that the same device used with the old fast
> > kernel is still fast today?
> 
> Yes, it is still fast. Just ran a 100 trials test and got an average
> of 70 seconds with standard deviation = 6 seconds, aligned with the
> past values of the same kernel.

Then can you collect trace on the old kernel via the previous script?

#!/bin/sh

MAJ=$1
MIN=$2
MAJ=$(( $MAJ << 20 ))
DEV=$(( $MAJ | $MIN ))

/usr/share/bcc/tools/trace -t -C \
    't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args->rwbs, args->sector, args->nr_sector' \
    't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d", args->rwbs, args->sector, args->nr_sector'

Both the two trace points and bcc should be available on the old kernel.

Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-28  8:12                                                                               ` AW: " Schmid, Carsten
@ 2019-11-28 11:40                                                                                 ` Andrea Vai
  2019-11-28 17:39                                                                                 ` Alan Stern
  1 sibling, 0 replies; 104+ messages in thread
From: Andrea Vai @ 2019-11-28 11:40 UTC (permalink / raw)
  To: Schmid, Carsten, Finn Thain
  Cc: Ming Lei, Damien Le Moal, Alan Stern, Jens Axboe,
	Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

Il giorno gio, 28/11/2019 alle 08.12 +0000, Schmid, Carsten ha
scritto:
> 
> [...]
> 
> I assume the only thing that you change between the benchmarks
> is the kernel (and the modules, of course), right, Andrea?
> 

It's my production machine, so apart from the changes involved in a
"normal use of a PC" I can say that there are no changes I am aware of
(apart from the kernel, and other changes you told me to do, such as
changing the IO scheduler, etc)... but please remember I am not an
expert, so feel free to ask me what other kind of changes I can tell
you about.

Thanks,
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-27  0:21                                                                         ` Finn Thain
  2019-11-27  8:14                                                                           ` AW: " Schmid, Carsten
@ 2019-11-28 17:10                                                                           ` Andrea Vai
  1 sibling, 0 replies; 104+ messages in thread
From: Andrea Vai @ 2019-11-28 17:10 UTC (permalink / raw)
  To: Finn Thain
  Cc: Ming Lei, Damien Le Moal, Alan Stern, Jens Axboe,
	Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

Il giorno mer, 27/11/2019 alle 11.21 +1100, Finn Thain ha scritto:
> On Tue, 26 Nov 2019, Andrea Vai wrote:
> 
> > Then I started another set of 100 trials and let them run tonight,
> and 
> > the first 10 trials were around 1000s, then gradually decreased
> to 
> > ~300s, and finally settled around 200s with some trials below 70-
> 80s. 
> > This to say, times are extremely variable and for the first time
> I 
> > noticed a sort of "performance increase" with time.
> > 
> 
> The sheer volume of testing (probably some terabytes by now) would 
> exercise the wear leveling algorithm in the FTL.
> 
> This in itself seems unlikely to improve performance significantly.
> But if 
> the flash memory came from a bad batch, perhaps it would have that
> effect.
> 
> To find out, someone may need to source another (genuine) Kingston 
> DataTraveller device.

I own another device (let's refer to it as "black odd"), identical to
the "slow" one (call it "black even"), and used it as well  to do the
tests, especially in the beginning of this story, because I suspected
the problem could be related to a faulty pen drive. At a certain time
I realized that the tests I performed didn't show any difference
between the two flash drives, so since that time I kept using just the
"black even". They were bought together, so of course both of them
probably belong to the same "maybe-bad batch".

But I have another Kingston DataTraveler ("White"), externally
slightly different from the other twos (it's white instead of black,
and labeled G4 instead of G3), though lsusb shows the same IDs:
0951:1666. It had been purchased some months after the other twos
(well, actually, it may be the result of an RMA exchange).

I have just ran one test on this White one, with the new (patched)
kernel, and it took an average of 200seconds (st.dev=46s), which is
not "good", but less "bad" than the real "bad" case of the "black"
ones (>1000 seconds).

I have also tried the "WHITE" one with the old fast kernel, and the
behavior is almost the same as with the new kernel, though a little
bit better (mean=173; st.dev.=11).

Feel free to let me know if I should do other tries,

thanks,
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-28  9:17                                                                               ` Ming Lei
@ 2019-11-28 17:34                                                                                 ` Andrea Vai
  2019-11-29  0:57                                                                                   ` Ming Lei
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-11-28 17:34 UTC (permalink / raw)
  To: Ming Lei
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

[-- Attachment #1: Type: text/plain, Size: 2094 bytes --]

Il giorno gio, 28/11/2019 alle 17.17 +0800, Ming Lei ha scritto:
> On Thu, Nov 28, 2019 at 08:46:57AM +0100, Andrea Vai wrote:
> > Il giorno mer, 27/11/2019 alle 08.14 +0000, Schmid, Carsten ha
> > scritto:
> > > > 
> > > > > Then I started another set of 100 trials and let them run
> > > tonight, and
> > > > > the first 10 trials were around 1000s, then gradually
> decreased
> > > to
> > > > > ~300s, and finally settled around 200s with some trials
> below
> > > 70-80s.
> > > > > This to say, times are extremely variable and for the first
> time
> > > I
> > > > > noticed a sort of "performance increase" with time.
> > > > >
> > > > 
> > > > The sheer volume of testing (probably some terabytes by now)
> would
> > > > exercise the wear leveling algorithm in the FTL.
> > > > 
> > > But with "old kernel" the copy operation still is "fast", as far
> as
> > > i understood.
> > > If FTL (e.g. wear leveling) would slow down, we would see that
> also
> > > in
> > > the old kernel, right?
> > > 
> > > Andrea, can you confirm that the same device used with the old
> fast
> > > kernel is still fast today?
> > 
> > Yes, it is still fast. Just ran a 100 trials test and got an
> average
> > of 70 seconds with standard deviation = 6 seconds, aligned with
> the
> > past values of the same kernel.
> 
> Then can you collect trace on the old kernel via the previous
> script?
> 
> #!/bin/sh
> 
> MAJ=$1
> MIN=$2
> MAJ=$(( $MAJ << 20 ))
> DEV=$(( $MAJ | $MIN ))
> 
> /usr/share/bcc/tools/trace -t -C \
>     't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args-
> >rwbs, args->sector, args->nr_sector' \
>     't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d", args-
> >rwbs, args->sector, args->nr_sector'
> 
> Both the two trace points and bcc should be available on the old
> kernel.
> 

Trace attached. Produced by: start the trace script
(with the pendrive already plugged), wait some seconds, run the test
(1 trial, 1 GB), wait for the test to finish, stop the trace.

The copy took 73 seconds, roughly as already seen before with the fast
old kernel.

Thanks,
Andrea

[-- Attachment #2: log_ming_20191128_182751.zip --]
[-- Type: application/zip, Size: 118068 bytes --]

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-28  8:12                                                                               ` AW: " Schmid, Carsten
  2019-11-28 11:40                                                                                 ` Andrea Vai
@ 2019-11-28 17:39                                                                                 ` Alan Stern
  1 sibling, 0 replies; 104+ messages in thread
From: Alan Stern @ 2019-11-28 17:39 UTC (permalink / raw)
  To: Schmid, Carsten
  Cc: Andrea Vai, Finn Thain, Ming Lei, Damien Le Moal, Jens Axboe,
	Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

On Thu, 28 Nov 2019, Schmid, Carsten wrote:

> I have been involved in several benchmarkings of flash devices in the past.
> So what we see here is definitely not a device issue regarding wear leveling.
> 
> I wanted to prevent all of you going into the wrong direction, that's why
> i wanted Andrea to confirm that it's not a matter of the flash device.
> 
> There are so much items involved into benchmarking flash devices.
> But Andrea's observations with factors of 10-30 times slow down
> i have never seen before.
> 
> I assume the only thing that you change between the benchmarks
> is the kernel (and the modules, of course), right, Andrea?
> Then we can rule out cache settings which massively can impact
> benchmarks.
> 
> The only thing that makes sense from my POV is:
> - collect traces with the kernel before mentioned commit (fast)
> - apply patch in doubt
> - again collect traces (slow)
> - compare the traces
> 
> Then we should be able to see the difference(s).

We have already done this.  I forget whether the traces are in the
email history available in the archives or whether they are stored 
somewhere else.

In any case, my analysis of the traces is in the archives.  It seemed 
very clear that the only difference which mattered was the ordering of 
the write commands (sequential vs. non-sequential).  This was obviously 
something which the commit in question would affect, and it also seemed 
likely to cause the device to slow down considerably.

Alan Stern

> Unfortunately i'm not an expert on the SCSI and USB kernel stuff
> involved here. Else i would try to understand what happens and
> give you some hints.
> 
> BR
> Carsten


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-28 17:34                                                                                 ` Andrea Vai
@ 2019-11-29  0:57                                                                                   ` Ming Lei
  2019-11-29  2:35                                                                                     ` Ming Lei
  2019-11-29 11:44                                                                                     ` AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Bernd Schubert
  0 siblings, 2 replies; 104+ messages in thread
From: Ming Lei @ 2019-11-29  0:57 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

On Thu, Nov 28, 2019 at 06:34:32PM +0100, Andrea Vai wrote:
> Il giorno gio, 28/11/2019 alle 17.17 +0800, Ming Lei ha scritto:
> > On Thu, Nov 28, 2019 at 08:46:57AM +0100, Andrea Vai wrote:
> > > Il giorno mer, 27/11/2019 alle 08.14 +0000, Schmid, Carsten ha
> > > scritto:
> > > > > 
> > > > > > Then I started another set of 100 trials and let them run
> > > > tonight, and
> > > > > > the first 10 trials were around 1000s, then gradually
> > decreased
> > > > to
> > > > > > ~300s, and finally settled around 200s with some trials
> > below
> > > > 70-80s.
> > > > > > This to say, times are extremely variable and for the first
> > time
> > > > I
> > > > > > noticed a sort of "performance increase" with time.
> > > > > >
> > > > > 
> > > > > The sheer volume of testing (probably some terabytes by now)
> > would
> > > > > exercise the wear leveling algorithm in the FTL.
> > > > > 
> > > > But with "old kernel" the copy operation still is "fast", as far
> > as
> > > > i understood.
> > > > If FTL (e.g. wear leveling) would slow down, we would see that
> > also
> > > > in
> > > > the old kernel, right?
> > > > 
> > > > Andrea, can you confirm that the same device used with the old
> > fast
> > > > kernel is still fast today?
> > > 
> > > Yes, it is still fast. Just ran a 100 trials test and got an
> > average
> > > of 70 seconds with standard deviation = 6 seconds, aligned with
> > the
> > > past values of the same kernel.
> > 
> > Then can you collect trace on the old kernel via the previous
> > script?
> > 
> > #!/bin/sh
> > 
> > MAJ=$1
> > MIN=$2
> > MAJ=$(( $MAJ << 20 ))
> > DEV=$(( $MAJ | $MIN ))
> > 
> > /usr/share/bcc/tools/trace -t -C \
> >     't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args-
> > >rwbs, args->sector, args->nr_sector' \
> >     't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d", args-
> > >rwbs, args->sector, args->nr_sector'
> > 
> > Both the two trace points and bcc should be available on the old
> > kernel.
> > 
> 
> Trace attached. Produced by: start the trace script
> (with the pendrive already plugged), wait some seconds, run the test
> (1 trial, 1 GB), wait for the test to finish, stop the trace.
> 
> The copy took 73 seconds, roughly as already seen before with the fast
> old kernel.

This trace shows a good write IO order because the writeback IOs are
queued to block layer serially from the 'cp' task and writeback wq.

However, writeback IO order is changed in current linus tree because
the IOs are queued to block layer concurrently from the 'cp' task
and writeback wq. It might be related with killing queue_congestion
by blk-mq.

The performance effect could be not only on this specific USB drive,
but also on all HDD., I guess.

However, I still can't reproduce it in my VM even though I built it
with similar setting of Andrea's test machine. Maybe the emulated disk
is too fast than Andrea's.

Andrea, can you collect the following log when running the test
on current new(bad) kernel?

	/usr/share/bcc/tools/stackcount  -K blk_mq_make_request

Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-29  0:57                                                                                   ` Ming Lei
@ 2019-11-29  2:35                                                                                     ` Ming Lei
  2019-11-29 14:41                                                                                       ` Andrea Vai
  2019-11-29 11:44                                                                                     ` AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Bernd Schubert
  1 sibling, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-11-29  2:35 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

On Fri, Nov 29, 2019 at 08:57:34AM +0800, Ming Lei wrote:
> On Thu, Nov 28, 2019 at 06:34:32PM +0100, Andrea Vai wrote:
> > Il giorno gio, 28/11/2019 alle 17.17 +0800, Ming Lei ha scritto:
> > > On Thu, Nov 28, 2019 at 08:46:57AM +0100, Andrea Vai wrote:
> > > > Il giorno mer, 27/11/2019 alle 08.14 +0000, Schmid, Carsten ha
> > > > scritto:
> > > > > > 
> > > > > > > Then I started another set of 100 trials and let them run
> > > > > tonight, and
> > > > > > > the first 10 trials were around 1000s, then gradually
> > > decreased
> > > > > to
> > > > > > > ~300s, and finally settled around 200s with some trials
> > > below
> > > > > 70-80s.
> > > > > > > This to say, times are extremely variable and for the first
> > > time
> > > > > I
> > > > > > > noticed a sort of "performance increase" with time.
> > > > > > >
> > > > > > 
> > > > > > The sheer volume of testing (probably some terabytes by now)
> > > would
> > > > > > exercise the wear leveling algorithm in the FTL.
> > > > > > 
> > > > > But with "old kernel" the copy operation still is "fast", as far
> > > as
> > > > > i understood.
> > > > > If FTL (e.g. wear leveling) would slow down, we would see that
> > > also
> > > > > in
> > > > > the old kernel, right?
> > > > > 
> > > > > Andrea, can you confirm that the same device used with the old
> > > fast
> > > > > kernel is still fast today?
> > > > 
> > > > Yes, it is still fast. Just ran a 100 trials test and got an
> > > average
> > > > of 70 seconds with standard deviation = 6 seconds, aligned with
> > > the
> > > > past values of the same kernel.
> > > 
> > > Then can you collect trace on the old kernel via the previous
> > > script?
> > > 
> > > #!/bin/sh
> > > 
> > > MAJ=$1
> > > MIN=$2
> > > MAJ=$(( $MAJ << 20 ))
> > > DEV=$(( $MAJ | $MIN ))
> > > 
> > > /usr/share/bcc/tools/trace -t -C \
> > >     't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args-
> > > >rwbs, args->sector, args->nr_sector' \
> > >     't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d", args-
> > > >rwbs, args->sector, args->nr_sector'
> > > 
> > > Both the two trace points and bcc should be available on the old
> > > kernel.
> > > 
> > 
> > Trace attached. Produced by: start the trace script
> > (with the pendrive already plugged), wait some seconds, run the test
> > (1 trial, 1 GB), wait for the test to finish, stop the trace.
> > 
> > The copy took 73 seconds, roughly as already seen before with the fast
> > old kernel.
> 
> This trace shows a good write IO order because the writeback IOs are
> queued to block layer serially from the 'cp' task and writeback wq.
> 
> However, writeback IO order is changed in current linus tree because
> the IOs are queued to block layer concurrently from the 'cp' task
> and writeback wq. It might be related with killing queue_congestion
> by blk-mq.
> 
> The performance effect could be not only on this specific USB drive,
> but also on all HDD., I guess.
> 
> However, I still can't reproduce it in my VM even though I built it
> with similar setting of Andrea's test machine. Maybe the emulated disk
> is too fast than Andrea's.
> 
> Andrea, can you collect the following log when running the test
> on current new(bad) kernel?
> 
> 	/usr/share/bcc/tools/stackcount  -K blk_mq_make_request

Instead, please run the following trace, given insert may be
called from other paths, such as flush plug:

	/usr/share/bcc/tools/stackcount -K t:block:block_rq_insert

If you are using python3, the following failure may be triggered:

	"cannot use a bytes pattern on a string-like object"

Then apply the following fix on /usr/lib/python3.7/site-packages/bcc/__init__.py

diff --git a/src/python/bcc/__init__.py b/src/python/bcc/__init__.py
index 6f114de8..bff5f282 100644
--- a/src/python/bcc/__init__.py
+++ b/src/python/bcc/__init__.py
@@ -769,7 +769,7 @@ class BPF(object):
                 evt_dir = os.path.join(cat_dir, event)
                 if os.path.isdir(evt_dir):
                     tp = ("%s:%s" % (category, event))
-                    if re.match(tp_re, tp):
+                    if re.match(tp_re.decode(), tp):
                         results.append(tp)
         return results

Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-29  0:57                                                                                   ` Ming Lei
  2019-11-29  2:35                                                                                     ` Ming Lei
@ 2019-11-29 11:44                                                                                     ` Bernd Schubert
  2019-12-02  7:01                                                                                       ` Andrea Vai
  1 sibling, 1 reply; 104+ messages in thread
From: Bernd Schubert @ 2019-11-29 11:44 UTC (permalink / raw)
  To: Ming Lei, Andrea Vai
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

>> Trace attached. Produced by: start the trace script
>> (with the pendrive already plugged), wait some seconds, run the test
>> (1 trial, 1 GB), wait for the test to finish, stop the trace.
>>
>> The copy took 73 seconds, roughly as already seen before with the fast
>> old kernel.
> 
> This trace shows a good write IO order because the writeback IOs are
> queued to block layer serially from the 'cp' task and writeback wq.
> 
> However, writeback IO order is changed in current linus tree because
> the IOs are queued to block layer concurrently from the 'cp' task
> and writeback wq. It might be related with killing queue_congestion
> by blk-mq.

What about using direct-io to ensure order is guaranteed? Pity that 'cp'
doesn't seem to have an option for it. But dd should do the trick.
Andrea, can you replace cp with a dd command (on the slow kernel)?

dd if=<path-to-src-file> of=<path-to-copy-on-flash-device> bs=1M
oflag=direct

 - Bernd

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-29  2:35                                                                                     ` Ming Lei
@ 2019-11-29 14:41                                                                                       ` Andrea Vai
  2019-12-03  2:23                                                                                         ` Ming Lei
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-11-29 14:41 UTC (permalink / raw)
  To: Ming Lei
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

[-- Attachment #1: Type: text/plain, Size: 740 bytes --]

Il giorno ven, 29/11/2019 alle 10.35 +0800, Ming Lei ha scritto:
> On Fri, Nov 29, 2019 at 08:57:34AM +0800, Ming Lei wrote:
> 
> > [...]
> 
> > Andrea, can you collect the following log when running the test
> > on current new(bad) kernel?
> > 
> > 	/usr/share/bcc/tools/stackcount  -K blk_mq_make_request
> 
> Instead, please run the following trace, given insert may be
> called from other paths, such as flush plug:
> 
> 	/usr/share/bcc/tools/stackcount -K t:block:block_rq_insert

Attached, for new (patched) bad kernel.

Produced by: start the trace script (with the pendrive already
plugged), wait some seconds, run the test (1 trial, 1 GB), wait for
the test to finish, stop the trace.

The copy took ~1700 seconds.

Thanks,
Andrea

[-- Attachment #2: log_ming_20191129_150609.zip --]
[-- Type: application/zip, Size: 2996 bytes --]

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-29 11:44                                                                                     ` AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Bernd Schubert
@ 2019-12-02  7:01                                                                                       ` Andrea Vai
  0 siblings, 0 replies; 104+ messages in thread
From: Andrea Vai @ 2019-12-02  7:01 UTC (permalink / raw)
  To: Bernd Schubert
  Cc: Ming Lei, Schmid, Carsten, Finn Thain, Damien Le Moal,
	Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list

On 29/11/19 12:44:53, Bernd Schubert wrote:
> >> Trace attached. Produced by: start the trace script
> >> (with the pendrive already plugged), wait some seconds, run the test
> >> (1 trial, 1 GB), wait for the test to finish, stop the trace.
> >>
> >> The copy took 73 seconds, roughly as already seen before with the fast
> >> old kernel.
> > 
> > This trace shows a good write IO order because the writeback IOs are
> > queued to block layer serially from the 'cp' task and writeback wq.
> > 
> > However, writeback IO order is changed in current linus tree because
> > the IOs are queued to block layer concurrently from the 'cp' task
> > and writeback wq. It might be related with killing queue_congestion
> > by blk-mq.
> 
> What about using direct-io to ensure order is guaranteed? Pity that 'cp'
> doesn't seem to have an option for it. But dd should do the trick.
> Andrea, can you replace cp with a dd command (on the slow kernel)?
> 
> dd if=<path-to-src-file> of=<path-to-copy-on-flash-device> bs=1M
> oflag=direct

On the "new bad patched" kernel, this command take 68 seconds to complete (mean on 100 trials, with a narrow standard deviation), so perfectly
aligned with the cp command on the old fast good kernel.

Thanks, and bye
Andrea

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-11-29 14:41                                                                                       ` Andrea Vai
@ 2019-12-03  2:23                                                                                         ` Ming Lei
  2019-12-10  7:35                                                                                           ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-12-03  2:23 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel,
	Theodore Ts'o

On Fri, Nov 29, 2019 at 03:41:01PM +0100, Andrea Vai wrote:
> Il giorno ven, 29/11/2019 alle 10.35 +0800, Ming Lei ha scritto:
> > On Fri, Nov 29, 2019 at 08:57:34AM +0800, Ming Lei wrote:
> > 
> > > [...]
> > 
> > > Andrea, can you collect the following log when running the test
> > > on current new(bad) kernel?
> > > 
> > > 	/usr/share/bcc/tools/stackcount  -K blk_mq_make_request
> > 
> > Instead, please run the following trace, given insert may be
> > called from other paths, such as flush plug:
> > 
> > 	/usr/share/bcc/tools/stackcount -K t:block:block_rq_insert
> 
> Attached, for new (patched) bad kernel.
> 
> Produced by: start the trace script (with the pendrive already
> plugged), wait some seconds, run the test (1 trial, 1 GB), wait for
> the test to finish, stop the trace.
> 
> The copy took ~1700 seconds.

See the two path[1][2] of inserting request, and path[1] is triggered
4358 times, and the path[2] is triggered 5763 times.

The path[2] is expected behaviour. Not sure path [1] is correct, given
ext4_release_file() is supposed to be called when this inode is
released. That means the file is closed 4358 times during 1GB file
copying to usb storage.

Cc filesystem list.


[1] insert requests when returning to user mode from syscall

  b'blk_mq_sched_request_inserted'
  b'blk_mq_sched_request_inserted'
  b'dd_insert_requests'
  b'blk_mq_sched_insert_requests'
  b'blk_mq_flush_plug_list'
  b'blk_flush_plug_list'
  b'io_schedule_prepare'
  b'io_schedule'
  b'rq_qos_wait'
  b'wbt_wait'
  b'__rq_qos_throttle'
  b'blk_mq_make_request'
  b'generic_make_request'
  b'submit_bio'
  b'ext4_io_submit'
  b'ext4_writepages'
  b'do_writepages'
  b'__filemap_fdatawrite_range'
  b'ext4_release_file'
  b'__fput'
  b'task_work_run'
  b'exit_to_usermode_loop'
  b'do_syscall_64'
  b'entry_SYSCALL_64_after_hwframe'
    4358

[2] insert requests from writeback wq context

  b'blk_mq_sched_request_inserted'
  b'blk_mq_sched_request_inserted'
  b'dd_insert_requests'
  b'blk_mq_sched_insert_requests'
  b'blk_mq_flush_plug_list'
  b'blk_flush_plug_list'
  b'io_schedule_prepare'
  b'io_schedule'
  b'rq_qos_wait'
  b'wbt_wait'
  b'__rq_qos_throttle'
  b'blk_mq_make_request'
  b'generic_make_request'
  b'submit_bio'
  b'ext4_io_submit'
  b'ext4_bio_write_page'
  b'mpage_submit_page'
  b'mpage_process_page_bufs'
  b'mpage_prepare_extent_to_map'
  b'ext4_writepages'
  b'do_writepages'
  b'__writeback_single_inode'
  b'writeback_sb_inodes'
  b'__writeback_inodes_wb'
  b'wb_writeback'
  b'wb_workfn'
  b'process_one_work'
  b'worker_thread'
  b'kthread'
  b'ret_from_fork'
    5763

Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-03  2:23                                                                                         ` Ming Lei
@ 2019-12-10  7:35                                                                                           ` Andrea Vai
  2019-12-10  8:05                                                                                             ` Ming Lei
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-12-10  7:35 UTC (permalink / raw)
  To: Ming Lei
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel,
	Theodore Ts'o

Il giorno mar, 03/12/2019 alle 10.23 +0800, Ming Lei ha scritto:
> On Fri, Nov 29, 2019 at 03:41:01PM +0100, Andrea Vai wrote:
> > Il giorno ven, 29/11/2019 alle 10.35 +0800, Ming Lei ha scritto:
> > > On Fri, Nov 29, 2019 at 08:57:34AM +0800, Ming Lei wrote:
> > > 
> > > > [...]
> > > 
> > > > Andrea, can you collect the following log when running the
> test
> > > > on current new(bad) kernel?
> > > > 
> > > > 	/usr/share/bcc/tools/stackcount  -K
> blk_mq_make_request
> > > 
> > > Instead, please run the following trace, given insert may be
> > > called from other paths, such as flush plug:
> > > 
> > > 	/usr/share/bcc/tools/stackcount -K t:block:block_rq_insert
> > 
> > Attached, for new (patched) bad kernel.
> > 
> > Produced by: start the trace script (with the pendrive already
> > plugged), wait some seconds, run the test (1 trial, 1 GB), wait
> for
> > the test to finish, stop the trace.
> > 
> > The copy took ~1700 seconds.
> 
> See the two path[1][2] of inserting request, and path[1] is
> triggered
> 4358 times, and the path[2] is triggered 5763 times.
> 
> The path[2] is expected behaviour. Not sure path [1] is correct,
> given
> ext4_release_file() is supposed to be called when this inode is
> released. That means the file is closed 4358 times during 1GB file
> copying to usb storage.
> 
> Cc filesystem list.
> 
> 
> [1] insert requests when returning to user mode from syscall
> 
>   b'blk_mq_sched_request_inserted'
>   b'blk_mq_sched_request_inserted'
>   b'dd_insert_requests'
>   b'blk_mq_sched_insert_requests'
>   b'blk_mq_flush_plug_list'
>   b'blk_flush_plug_list'
>   b'io_schedule_prepare'
>   b'io_schedule'
>   b'rq_qos_wait'
>   b'wbt_wait'
>   b'__rq_qos_throttle'
>   b'blk_mq_make_request'
>   b'generic_make_request'
>   b'submit_bio'
>   b'ext4_io_submit'
>   b'ext4_writepages'
>   b'do_writepages'
>   b'__filemap_fdatawrite_range'
>   b'ext4_release_file'
>   b'__fput'
>   b'task_work_run'
>   b'exit_to_usermode_loop'
>   b'do_syscall_64'
>   b'entry_SYSCALL_64_after_hwframe'
>     4358
> 
> [2] insert requests from writeback wq context
> 
>   b'blk_mq_sched_request_inserted'
>   b'blk_mq_sched_request_inserted'
>   b'dd_insert_requests'
>   b'blk_mq_sched_insert_requests'
>   b'blk_mq_flush_plug_list'
>   b'blk_flush_plug_list'
>   b'io_schedule_prepare'
>   b'io_schedule'
>   b'rq_qos_wait'
>   b'wbt_wait'
>   b'__rq_qos_throttle'
>   b'blk_mq_make_request'
>   b'generic_make_request'
>   b'submit_bio'
>   b'ext4_io_submit'
>   b'ext4_bio_write_page'
>   b'mpage_submit_page'
>   b'mpage_process_page_bufs'
>   b'mpage_prepare_extent_to_map'
>   b'ext4_writepages'
>   b'do_writepages'
>   b'__writeback_single_inode'
>   b'writeback_sb_inodes'
>   b'__writeback_inodes_wb'
>   b'wb_writeback'
>   b'wb_workfn'
>   b'process_one_work'
>   b'worker_thread'
>   b'kthread'
>   b'ret_from_fork'
>     5763
> 
> Thanks,
> Ming
> 

Is there any update on this? Sorry if I am making noise, but I would
like to help to improve the kernel (or fix it) if I can help.
Otherwise, please let me know how to consider this case,

Thanks, and bye
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-10  7:35                                                                                           ` Andrea Vai
@ 2019-12-10  8:05                                                                                             ` Ming Lei
  2019-12-11  2:41                                                                                               ` Theodore Y. Ts'o
  0 siblings, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-12-10  8:05 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel,
	Theodore Ts'o

On Tue, Dec 10, 2019 at 08:35:43AM +0100, Andrea Vai wrote:
> Il giorno mar, 03/12/2019 alle 10.23 +0800, Ming Lei ha scritto:
> > On Fri, Nov 29, 2019 at 03:41:01PM +0100, Andrea Vai wrote:
> > > Il giorno ven, 29/11/2019 alle 10.35 +0800, Ming Lei ha scritto:
> > > > On Fri, Nov 29, 2019 at 08:57:34AM +0800, Ming Lei wrote:
> > > > 
> > > > > [...]
> > > > 
> > > > > Andrea, can you collect the following log when running the
> > test
> > > > > on current new(bad) kernel?
> > > > > 
> > > > > 	/usr/share/bcc/tools/stackcount  -K
> > blk_mq_make_request
> > > > 
> > > > Instead, please run the following trace, given insert may be
> > > > called from other paths, such as flush plug:
> > > > 
> > > > 	/usr/share/bcc/tools/stackcount -K t:block:block_rq_insert
> > > 
> > > Attached, for new (patched) bad kernel.
> > > 
> > > Produced by: start the trace script (with the pendrive already
> > > plugged), wait some seconds, run the test (1 trial, 1 GB), wait
> > for
> > > the test to finish, stop the trace.
> > > 
> > > The copy took ~1700 seconds.
> > 
> > See the two path[1][2] of inserting request, and path[1] is
> > triggered
> > 4358 times, and the path[2] is triggered 5763 times.
> > 
> > The path[2] is expected behaviour. Not sure path [1] is correct,
> > given
> > ext4_release_file() is supposed to be called when this inode is
> > released. That means the file is closed 4358 times during 1GB file
> > copying to usb storage.
> > 
> > Cc filesystem list.
> > 
> > 
> > [1] insert requests when returning to user mode from syscall
> > 
> >   b'blk_mq_sched_request_inserted'
> >   b'blk_mq_sched_request_inserted'
> >   b'dd_insert_requests'
> >   b'blk_mq_sched_insert_requests'
> >   b'blk_mq_flush_plug_list'
> >   b'blk_flush_plug_list'
> >   b'io_schedule_prepare'
> >   b'io_schedule'
> >   b'rq_qos_wait'
> >   b'wbt_wait'
> >   b'__rq_qos_throttle'
> >   b'blk_mq_make_request'
> >   b'generic_make_request'
> >   b'submit_bio'
> >   b'ext4_io_submit'
> >   b'ext4_writepages'
> >   b'do_writepages'
> >   b'__filemap_fdatawrite_range'
> >   b'ext4_release_file'
> >   b'__fput'
> >   b'task_work_run'
> >   b'exit_to_usermode_loop'
> >   b'do_syscall_64'
> >   b'entry_SYSCALL_64_after_hwframe'
> >     4358
> > 
> > [2] insert requests from writeback wq context
> > 
> >   b'blk_mq_sched_request_inserted'
> >   b'blk_mq_sched_request_inserted'
> >   b'dd_insert_requests'
> >   b'blk_mq_sched_insert_requests'
> >   b'blk_mq_flush_plug_list'
> >   b'blk_flush_plug_list'
> >   b'io_schedule_prepare'
> >   b'io_schedule'
> >   b'rq_qos_wait'
> >   b'wbt_wait'
> >   b'__rq_qos_throttle'
> >   b'blk_mq_make_request'
> >   b'generic_make_request'
> >   b'submit_bio'
> >   b'ext4_io_submit'
> >   b'ext4_bio_write_page'
> >   b'mpage_submit_page'
> >   b'mpage_process_page_bufs'
> >   b'mpage_prepare_extent_to_map'
> >   b'ext4_writepages'
> >   b'do_writepages'
> >   b'__writeback_single_inode'
> >   b'writeback_sb_inodes'
> >   b'__writeback_inodes_wb'
> >   b'wb_writeback'
> >   b'wb_workfn'
> >   b'process_one_work'
> >   b'worker_thread'
> >   b'kthread'
> >   b'ret_from_fork'
> >     5763
> > 
> > Thanks,
> > Ming
> > 
> 
> Is there any update on this? Sorry if I am making noise, but I would
> like to help to improve the kernel (or fix it) if I can help.
> Otherwise, please let me know how to consider this case,

IMO, the extra write path from exit_to_usermode_loop() isn't expected,
that should be the reason why write IO order is changed, then performance
drops on your USB storage.

We need our fs/ext4 experts to take a look.

Or can you reproduce the issue on xfs or btrfs?

Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-10  8:05                                                                                             ` Ming Lei
@ 2019-12-11  2:41                                                                                               ` Theodore Y. Ts'o
  2019-12-11  4:00                                                                                                 ` Ming Lei
  0 siblings, 1 reply; 104+ messages in thread
From: Theodore Y. Ts'o @ 2019-12-11  2:41 UTC (permalink / raw)
  To: Ming Lei
  Cc: Andrea Vai, Schmid, Carsten, Finn Thain, Damien Le Moal,
	Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

On Tue, Dec 10, 2019 at 04:05:50PM +0800, Ming Lei wrote:
> > > The path[2] is expected behaviour. Not sure path [1] is correct,
> > > given
> > > ext4_release_file() is supposed to be called when this inode is
> > > released. That means the file is closed 4358 times during 1GB file
> > > copying to usb storage.
> > > 
> > > [1] insert requests when returning to user mode from syscall
> > > 
> > >   b'blk_mq_sched_request_inserted'
> > >   b'blk_mq_sched_request_inserted'
> > >   b'dd_insert_requests'
> > >   b'blk_mq_sched_insert_requests'
> > >   b'blk_mq_flush_plug_list'
> > >   b'blk_flush_plug_list'
> > >   b'io_schedule_prepare'
> > >   b'io_schedule'
> > >   b'rq_qos_wait'
> > >   b'wbt_wait'
> > >   b'__rq_qos_throttle'
> > >   b'blk_mq_make_request'
> > >   b'generic_make_request'
> > >   b'submit_bio'
> > >   b'ext4_io_submit'
> > >   b'ext4_writepages'
> > >   b'do_writepages'
> > >   b'__filemap_fdatawrite_range'
> > >   b'ext4_release_file'
> > >   b'__fput'
> > >   b'task_work_run'
> > >   b'exit_to_usermode_loop'
> > >   b'do_syscall_64'
> > >   b'entry_SYSCALL_64_after_hwframe'
> > >     4358

I'm guessing that your workload is repeatedly truncating a file (or
calling open with O_TRUNC) and then writing data to it.  When you do
this, then when the file is closed, we assume that since you were
replacing the previous contents of a file with new contents, that you
would be unhappy if the file contents was replaced by a zero length
file after a crash.  That's because ten years, ago there were a *huge*
number of crappy applications that would replace a file by reading it
into memory, truncating it, and then write out the new contents of the
file.  This could be a high score file for a game, or a KDE or GNOME
state file, etc.

So if someone does open, truncate, write, close, we still immediately
writing out the data on the close, assuming that the programmer really
wanted open, truncate, write, fsync, close, but was too careless to
actually do the right thing.

Some workaround[1] like this is done by all of the major file systems,
and was fallout the agreement from the "O_PONIES"[2] controversy.
This was discussed and agreed to at the 2009 LSF/MM workshop.  (See
the "rename, fsync, and ponies" section.)

[1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/45
[2] https://blahg.josefsipek.net/?p=364
[3] https://lwn.net/Articles/327601/

So if you're seeing a call to filemap_fdatawrite_range as the result
of a fput, that's why.

In any case, this behavior has been around for a decade, and it
appears to be incidental to your performance difficulties with your
USB thumbdrive and block-mq.

						- Ted

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-11  2:41                                                                                               ` Theodore Y. Ts'o
@ 2019-12-11  4:00                                                                                                 ` Ming Lei
  2019-12-11 16:07                                                                                                   ` Theodore Y. Ts'o
  0 siblings, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-12-11  4:00 UTC (permalink / raw)
  To: Theodore Y. Ts'o
  Cc: Andrea Vai, Schmid, Carsten, Finn Thain, Damien Le Moal,
	Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

On Tue, Dec 10, 2019 at 09:41:37PM -0500, Theodore Y. Ts'o wrote:
> On Tue, Dec 10, 2019 at 04:05:50PM +0800, Ming Lei wrote:
> > > > The path[2] is expected behaviour. Not sure path [1] is correct,
> > > > given
> > > > ext4_release_file() is supposed to be called when this inode is
> > > > released. That means the file is closed 4358 times during 1GB file
> > > > copying to usb storage.
> > > > 
> > > > [1] insert requests when returning to user mode from syscall
> > > > 
> > > >   b'blk_mq_sched_request_inserted'
> > > >   b'blk_mq_sched_request_inserted'
> > > >   b'dd_insert_requests'
> > > >   b'blk_mq_sched_insert_requests'
> > > >   b'blk_mq_flush_plug_list'
> > > >   b'blk_flush_plug_list'
> > > >   b'io_schedule_prepare'
> > > >   b'io_schedule'
> > > >   b'rq_qos_wait'
> > > >   b'wbt_wait'
> > > >   b'__rq_qos_throttle'
> > > >   b'blk_mq_make_request'
> > > >   b'generic_make_request'
> > > >   b'submit_bio'
> > > >   b'ext4_io_submit'
> > > >   b'ext4_writepages'
> > > >   b'do_writepages'
> > > >   b'__filemap_fdatawrite_range'
> > > >   b'ext4_release_file'
> > > >   b'__fput'
> > > >   b'task_work_run'
> > > >   b'exit_to_usermode_loop'
> > > >   b'do_syscall_64'
> > > >   b'entry_SYSCALL_64_after_hwframe'
> > > >     4358
> 
> I'm guessing that your workload is repeatedly truncating a file (or
> calling open with O_TRUNC) and then writing data to it.  When you do
> this, then when the file is closed, we assume that since you were
> replacing the previous contents of a file with new contents, that you
> would be unhappy if the file contents was replaced by a zero length
> file after a crash.  That's because ten years, ago there were a *huge*
> number of crappy applications that would replace a file by reading it
> into memory, truncating it, and then write out the new contents of the
> file.  This could be a high score file for a game, or a KDE or GNOME
> state file, etc.
> 
> So if someone does open, truncate, write, close, we still immediately
> writing out the data on the close, assuming that the programmer really
> wanted open, truncate, write, fsync, close, but was too careless to
> actually do the right thing.
> 
> Some workaround[1] like this is done by all of the major file systems,
> and was fallout the agreement from the "O_PONIES"[2] controversy.
> This was discussed and agreed to at the 2009 LSF/MM workshop.  (See
> the "rename, fsync, and ponies" section.)
> 
> [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/45
> [2] https://blahg.josefsipek.net/?p=364
> [3] https://lwn.net/Articles/327601/
> 
> So if you're seeing a call to filemap_fdatawrite_range as the result
> of a fput, that's why.
> 
> In any case, this behavior has been around for a decade, and it
> appears to be incidental to your performance difficulties with your
> USB thumbdrive and block-mq.

I didn't reproduce the issue in my test environment, and follows
Andrea's test commands[1]:

  mount UUID=$uuid /mnt/pendrive 2>&1 |tee -a $logfile
  SECONDS=0
  cp $testfile /mnt/pendrive 2>&1 |tee -a $logfile
  umount /mnt/pendrive 2>&1 |tee -a $logfile

The 'cp' command supposes to open/close the file just once, however
ext4_release_file() & write pages is observed to run for 4358 times
when executing the above 'cp' test.


[1] https://marc.info/?l=linux-kernel&m=157486689806734&w=2


Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-11  4:00                                                                                                 ` Ming Lei
@ 2019-12-11 16:07                                                                                                   ` Theodore Y. Ts'o
  2019-12-11 21:33                                                                                                     ` Ming Lei
  0 siblings, 1 reply; 104+ messages in thread
From: Theodore Y. Ts'o @ 2019-12-11 16:07 UTC (permalink / raw)
  To: Ming Lei
  Cc: Andrea Vai, Schmid, Carsten, Finn Thain, Damien Le Moal,
	Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

On Wed, Dec 11, 2019 at 12:00:58PM +0800, Ming Lei wrote:
> I didn't reproduce the issue in my test environment, and follows
> Andrea's test commands[1]:
> 
>   mount UUID=$uuid /mnt/pendrive 2>&1 |tee -a $logfile
>   SECONDS=0
>   cp $testfile /mnt/pendrive 2>&1 |tee -a $logfile
>   umount /mnt/pendrive 2>&1 |tee -a $logfile
> 
> The 'cp' command supposes to open/close the file just once, however
> ext4_release_file() & write pages is observed to run for 4358 times
> when executing the above 'cp' test.

Why are we sure the ext4_release_file() / _fput() is coming from the
cp command, as opposed to something else that might be running on the
system under test?  _fput() is called by the kernel when the last
reference to a struct file is released.  (Specifically, if you have a
fd which is dup'ed, it's only when the last fd corresponding to the
struct file is closed, and the struct file is about to be released,
does the file system's f_ops->release function get called.)

So the first question I'd ask is whether there is anything else going
on the system, and whether the writes are happening to the USB thumb
drive, or to some other storage device.  And if there is something
else which is writing to the pendrive, maybe that's why no one else
has been able to reproduce the OP's complaint....

    	      	 	    	     - Ted

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-11 16:07                                                                                                   ` Theodore Y. Ts'o
@ 2019-12-11 21:33                                                                                                     ` Ming Lei
  2019-12-12  7:34                                                                                                       ` Andrea Vai
  2019-12-18  8:25                                                                                                       ` Andrea Vai
  0 siblings, 2 replies; 104+ messages in thread
From: Ming Lei @ 2019-12-11 21:33 UTC (permalink / raw)
  To: Theodore Y. Ts'o
  Cc: Andrea Vai, Schmid, Carsten, Finn Thain, Damien Le Moal,
	Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

On Wed, Dec 11, 2019 at 11:07:45AM -0500, Theodore Y. Ts'o wrote:
> On Wed, Dec 11, 2019 at 12:00:58PM +0800, Ming Lei wrote:
> > I didn't reproduce the issue in my test environment, and follows
> > Andrea's test commands[1]:
> > 
> >   mount UUID=$uuid /mnt/pendrive 2>&1 |tee -a $logfile
> >   SECONDS=0
> >   cp $testfile /mnt/pendrive 2>&1 |tee -a $logfile
> >   umount /mnt/pendrive 2>&1 |tee -a $logfile
> > 
> > The 'cp' command supposes to open/close the file just once, however
> > ext4_release_file() & write pages is observed to run for 4358 times
> > when executing the above 'cp' test.
> 
> Why are we sure the ext4_release_file() / _fput() is coming from the
> cp command, as opposed to something else that might be running on the
> system under test?  _fput() is called by the kernel when the last

Please see the log:

https://lore.kernel.org/linux-scsi/3af3666920e7d46f8f0c6d88612f143ffabc743c.camel@unipv.it/2-log_ming.zip

Which is collected by:

#!/bin/sh
MAJ=$1
MIN=$2
MAJ=$(( $MAJ << 20 ))
DEV=$(( $MAJ | $MIN ))

/usr/share/bcc/tools/trace -t -C \
    't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args->rwbs, args->sector, args->nr_sector' \
    't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d", args->rwbs, args->sector, args->nr_sector'

$MAJ:$MIN points to the USB storage disk.

From the above IO trace, there are two write paths, one is from cp,
another is from writeback wq.

The stackcount trace[1] is consistent with the IO trace log since it
only shows two IO paths, that is why I concluded that the write done via
ext4_release_file() is from 'cp'.

[1] https://lore.kernel.org/linux-scsi/320b315b9c87543d4fb919ecbdf841596c8fbcea.camel@unipv.it/2-log_ming_20191129_150609.zip

> reference to a struct file is released.  (Specifically, if you have a
> fd which is dup'ed, it's only when the last fd corresponding to the
> struct file is closed, and the struct file is about to be released,
> does the file system's f_ops->release function get called.)
> 
> So the first question I'd ask is whether there is anything else going
> on the system, and whether the writes are happening to the USB thumb
> drive, or to some other storage device.  And if there is something
> else which is writing to the pendrive, maybe that's why no one else
> has been able to reproduce the OP's complaint....

OK, we can ask Andrea to confirm that via the following trace, which
will add pid/comm info in the stack trace:

/usr/share/bcc/tools/stackcount  blk_mq_sched_request_inserted

Andrew, could you collect the above log again when running new/bad
kernel for confirming if the write done by ext4_release_file() is from
the 'cp' process?

Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-11 21:33                                                                                                     ` Ming Lei
@ 2019-12-12  7:34                                                                                                       ` Andrea Vai
  2019-12-18  8:25                                                                                                       ` Andrea Vai
  1 sibling, 0 replies; 104+ messages in thread
From: Andrea Vai @ 2019-12-12  7:34 UTC (permalink / raw)
  To: Ming Lei, Theodore Y. Ts'o
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

Il giorno gio, 12/12/2019 alle 05.33 +0800, Ming Lei ha scritto:
> On Wed, Dec 11, 2019 at 11:07:45AM -0500, Theodore Y. Ts'o wrote:
> > On Wed, Dec 11, 2019 at 12:00:58PM +0800, Ming Lei wrote:
> > > I didn't reproduce the issue in my test environment, and follows
> > > Andrea's test commands[1]:
> > > 
> > >   mount UUID=$uuid /mnt/pendrive 2>&1 |tee -a $logfile
> > >   SECONDS=0
> > >   cp $testfile /mnt/pendrive 2>&1 |tee -a $logfile
> > >   umount /mnt/pendrive 2>&1 |tee -a $logfile
> > > 
> > > The 'cp' command supposes to open/close the file just once,
> however
> > > ext4_release_file() & write pages is observed to run for 4358
> times
> > > when executing the above 'cp' test.
> > 
> > Why are we sure the ext4_release_file() / _fput() is coming from
> the
> > cp command, as opposed to something else that might be running on
> the
> > system under test?  _fput() is called by the kernel when the last
> 
> Please see the log:
> 
> https://lore.kernel.org/linux-scsi/3af3666920e7d46f8f0c6d88612f143ffabc743c.camel@unipv.it/2-log_ming.zip
> 
> Which is collected by:
> 
> #!/bin/sh
> MAJ=$1
> MIN=$2
> MAJ=$(( $MAJ << 20 ))
> DEV=$(( $MAJ | $MIN ))
> 
> /usr/share/bcc/tools/trace -t -C \
>     't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args-
> >rwbs, args->sector, args->nr_sector' \
>     't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d", args-
> >rwbs, args->sector, args->nr_sector'
> 
> $MAJ:$MIN points to the USB storage disk.
> 
> From the above IO trace, there are two write paths, one is from cp,
> another is from writeback wq.
> 
> The stackcount trace[1] is consistent with the IO trace log since it
> only shows two IO paths, that is why I concluded that the write done
> via
> ext4_release_file() is from 'cp'.
> 
> [1] 
> https://lore.kernel.org/linux-scsi/320b315b9c87543d4fb919ecbdf841596c8fbcea.camel@unipv.it/2-log_ming_20191129_150609.zip
> 
> > reference to a struct file is released.  (Specifically, if you
> have a
> > fd which is dup'ed, it's only when the last fd corresponding to
> the
> > struct file is closed, and the struct file is about to be
> released,
> > does the file system's f_ops->release function get called.)
> > 
> > So the first question I'd ask is whether there is anything else
> going
> > on the system, and whether the writes are happening to the USB
> thumb
> > drive, or to some other storage device.  And if there is something
> > else which is writing to the pendrive, maybe that's why no one
> else
> > has been able to reproduce the OP's complaint....
> 
> OK, we can ask Andrea to confirm that via the following trace, which
> will add pid/comm info in the stack trace:
> 
> /usr/share/bcc/tools/stackcount  blk_mq_sched_request_inserted
> 
> Andrea, could you collect the above log again when running new/bad
> kernel for confirming if the write done by ext4_release_file() is
> from
> the 'cp' process?

Yes, I will try to do it as soon as possible and let you know.
I will also try xfs or btrfs, as you suggested in another message.

Thanks, and bye
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-11 21:33                                                                                                     ` Ming Lei
  2019-12-12  7:34                                                                                                       ` Andrea Vai
@ 2019-12-18  8:25                                                                                                       ` Andrea Vai
  2019-12-18  9:48                                                                                                         ` Ming Lei
  1 sibling, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-12-18  8:25 UTC (permalink / raw)
  To: Ming Lei, Theodore Y. Ts'o
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

[-- Attachment #1: Type: text/plain, Size: 3636 bytes --]

Il giorno gio, 12/12/2019 alle 05.33 +0800, Ming Lei ha scritto:
> On Wed, Dec 11, 2019 at 11:07:45AM -0500, Theodore Y. Ts'o wrote:
> > On Wed, Dec 11, 2019 at 12:00:58PM +0800, Ming Lei wrote:
> > > I didn't reproduce the issue in my test environment, and follows
> > > Andrea's test commands[1]:
> > > 
> > >   mount UUID=$uuid /mnt/pendrive 2>&1 |tee -a $logfile
> > >   SECONDS=0
> > >   cp $testfile /mnt/pendrive 2>&1 |tee -a $logfile
> > >   umount /mnt/pendrive 2>&1 |tee -a $logfile
> > > 
> > > The 'cp' command supposes to open/close the file just once,
> however
> > > ext4_release_file() & write pages is observed to run for 4358
> times
> > > when executing the above 'cp' test.
> > 
> > Why are we sure the ext4_release_file() / _fput() is coming from
> the
> > cp command, as opposed to something else that might be running on
> the
> > system under test?  _fput() is called by the kernel when the last
> 
> Please see the log:
> 
> https://lore.kernel.org/linux-scsi/3af3666920e7d46f8f0c6d88612f143ffabc743c.camel@unipv.it/2-log_ming.zip
> 
> Which is collected by:
> 
> #!/bin/sh
> MAJ=$1
> MIN=$2
> MAJ=$(( $MAJ << 20 ))
> DEV=$(( $MAJ | $MIN ))
> 
> /usr/share/bcc/tools/trace -t -C \
>     't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args-
> >rwbs, args->sector, args->nr_sector' \
>     't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d", args-
> >rwbs, args->sector, args->nr_sector'
> 
> $MAJ:$MIN points to the USB storage disk.
> 
> From the above IO trace, there are two write paths, one is from cp,
> another is from writeback wq.
> 
> The stackcount trace[1] is consistent with the IO trace log since it
> only shows two IO paths, that is why I concluded that the write done
> via
> ext4_release_file() is from 'cp'.
> 
> [1] 
> https://lore.kernel.org/linux-scsi/320b315b9c87543d4fb919ecbdf841596c8fbcea.camel@unipv.it/2-log_ming_20191129_150609.zip
> 
> > reference to a struct file is released.  (Specifically, if you
> have a
> > fd which is dup'ed, it's only when the last fd corresponding to
> the
> > struct file is closed, and the struct file is about to be
> released,
> > does the file system's f_ops->release function get called.)
> > 
> > So the first question I'd ask is whether there is anything else
> going
> > on the system, and whether the writes are happening to the USB
> thumb
> > drive, or to some other storage device.  And if there is something
> > else which is writing to the pendrive, maybe that's why no one
> else
> > has been able to reproduce the OP's complaint....
> 
> OK, we can ask Andrea to confirm that via the following trace, which
> will add pid/comm info in the stack trace:
> 
> /usr/share/bcc/tools/stackcount blk_mq_sched_request_inserted
> 
> Andrew, could you collect the above log again when running new/bad
> kernel for confirming if the write done by ext4_release_file() is
> from
> the 'cp' process?

You can find the stackcount log attached. It has been produced by:

- /usr/share/bcc/tools/stackcount blk_mq_sched_request_inserted > trace.log
- wait some seconds
- run the test (1 copy trial), wait for the test to finish, wait some seconds
- stop the trace (ctrl+C)

The test took 1994 seconds to complete.

I also tried the usual test with btrfs and xfs. Btrfs behavior looks
"good". xfs seems sometimes better, sometimes worse, I would say. I
don't know if it matters, anyway you can also find the results of the
two tests (100 trials each). Basically, btrfs is always between 68 and
89 seconds, with a cyclicity (?) with "period=2 trials". xfs looks
almost always very good (63-65s), but sometimes "bad" (>300s).

Thanks,
Andrea

[-- Attachment #2: test_btrfs_20191217.txt --]
[-- Type: text/plain, Size: 11367 bytes --]

*** test btrfs *** -> test_btrfs_20191217.txt

Starting 100 tries with:
Linux angus.unipv.it 5.4.0+ #1 SMP Mon Nov 25 11:31:34 CET 2019 x86_64 x86_64 x86_64 GNU/Linux
-rw-r--r--. 1 root root 1,0G 25 nov 13.29 /NoBackup/testfile
/dev/sda1: LABEL="Fedora30" UUID="a7ca2491-c807-4b10-b33f-ef425699148d" TYPE="ext4" PARTUUID="8b16fbdd-01"
/dev/sda2: LABEL="Swap_4GB" UUID="ba020b1e-4cdc-4f94-b92c-bdc11613388d" TYPE="swap" PARTUUID="8b16fbdd-02"
/dev/sdf1: LABEL="BAK_ANDVAI" UUID="6ddfec28-3d9a-4676-a726-927fd3fb21e7" UUID_SUB="581c69ab-6758-4662-999a-b6dfe6ee5e69" TYPE="btrfs" PARTUUID="09066b88-01"
/dev/sdg1: LABEL="BAK_ANDVAI" UUID="df777e33-8890-4cee-a718-42233f4cafae" TYPE="ext4" PARTUUID="75265898-01"
cat /sys/block/sdf/queue/scheduler --> [mq-deadline] none
Inizio: mar 17 dic 2019, 13:31:00, CET...fine: mar 17 dic 2019, 13:32:26, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 13:32:26, CET...fine: mar 17 dic 2019, 13:33:36, CET --> ci ho messo 70 secondi!
Inizio: mar 17 dic 2019, 13:33:36, CET...fine: mar 17 dic 2019, 13:35:02, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 13:35:02, CET...fine: mar 17 dic 2019, 13:36:14, CET --> ci ho messo 72 secondi!
Inizio: mar 17 dic 2019, 13:36:14, CET...fine: mar 17 dic 2019, 13:37:40, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 13:37:40, CET...fine: mar 17 dic 2019, 13:38:51, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 13:38:51, CET...fine: mar 17 dic 2019, 13:40:18, CET --> ci ho messo 87 secondi!
Inizio: mar 17 dic 2019, 13:40:18, CET...fine: mar 17 dic 2019, 13:41:29, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 13:41:29, CET...fine: mar 17 dic 2019, 13:42:58, CET --> ci ho messo 89 secondi!
Inizio: mar 17 dic 2019, 13:42:58, CET...fine: mar 17 dic 2019, 13:44:11, CET --> ci ho messo 73 secondi!
Inizio: mar 17 dic 2019, 13:44:11, CET...fine: mar 17 dic 2019, 13:45:40, CET --> ci ho messo 89 secondi!
Inizio: mar 17 dic 2019, 13:45:40, CET...fine: mar 17 dic 2019, 13:46:49, CET --> ci ho messo 69 secondi!
Inizio: mar 17 dic 2019, 13:46:49, CET...fine: mar 17 dic 2019, 13:48:16, CET --> ci ho messo 87 secondi!
Inizio: mar 17 dic 2019, 13:48:16, CET...fine: mar 17 dic 2019, 13:49:27, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 13:49:27, CET...fine: mar 17 dic 2019, 13:50:53, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 13:50:53, CET...fine: mar 17 dic 2019, 13:52:04, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 13:52:04, CET...fine: mar 17 dic 2019, 13:53:30, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 13:53:30, CET...fine: mar 17 dic 2019, 13:54:41, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 13:54:41, CET...fine: mar 17 dic 2019, 13:56:07, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 13:56:07, CET...fine: mar 17 dic 2019, 13:57:18, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 13:57:18, CET...fine: mar 17 dic 2019, 13:58:46, CET --> ci ho messo 88 secondi!
Inizio: mar 17 dic 2019, 13:58:46, CET...fine: mar 17 dic 2019, 13:59:57, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 13:59:57, CET...fine: mar 17 dic 2019, 14:01:23, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:01:23, CET...fine: mar 17 dic 2019, 14:02:34, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:02:34, CET...fine: mar 17 dic 2019, 14:04:00, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:04:00, CET...fine: mar 17 dic 2019, 14:05:11, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:05:11, CET...fine: mar 17 dic 2019, 14:06:38, CET --> ci ho messo 87 secondi!
Inizio: mar 17 dic 2019, 14:06:38, CET...fine: mar 17 dic 2019, 14:07:49, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:07:49, CET...fine: mar 17 dic 2019, 14:09:15, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:09:15, CET...fine: mar 17 dic 2019, 14:10:26, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:10:26, CET...fine: mar 17 dic 2019, 14:11:52, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:11:52, CET...fine: mar 17 dic 2019, 14:13:03, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:13:03, CET...fine: mar 17 dic 2019, 14:14:29, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:14:29, CET...fine: mar 17 dic 2019, 14:15:41, CET --> ci ho messo 72 secondi!
Inizio: mar 17 dic 2019, 14:15:41, CET...fine: mar 17 dic 2019, 14:17:07, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:17:07, CET...fine: mar 17 dic 2019, 14:18:18, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:18:18, CET...fine: mar 17 dic 2019, 14:19:44, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:19:44, CET...fine: mar 17 dic 2019, 14:20:55, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:20:55, CET...fine: mar 17 dic 2019, 14:22:21, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:22:21, CET...fine: mar 17 dic 2019, 14:23:33, CET --> ci ho messo 72 secondi!
Inizio: mar 17 dic 2019, 14:23:33, CET...fine: mar 17 dic 2019, 14:24:59, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:24:59, CET...fine: mar 17 dic 2019, 14:26:10, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:26:10, CET...fine: mar 17 dic 2019, 14:27:36, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:27:36, CET...fine: mar 17 dic 2019, 14:28:46, CET --> ci ho messo 70 secondi!
Inizio: mar 17 dic 2019, 14:28:46, CET...fine: mar 17 dic 2019, 14:30:12, CET --> ci ho messo 85 secondi!
Inizio: mar 17 dic 2019, 14:30:12, CET...fine: mar 17 dic 2019, 14:31:23, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:31:23, CET...fine: mar 17 dic 2019, 14:32:49, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:32:49, CET...fine: mar 17 dic 2019, 14:34:01, CET --> ci ho messo 72 secondi!
Inizio: mar 17 dic 2019, 14:34:01, CET...fine: mar 17 dic 2019, 14:35:27, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:35:27, CET...fine: mar 17 dic 2019, 14:36:38, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:36:38, CET...fine: mar 17 dic 2019, 14:38:04, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:38:04, CET...fine: mar 17 dic 2019, 14:39:15, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:39:15, CET...fine: mar 17 dic 2019, 14:40:41, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:40:41, CET...fine: mar 17 dic 2019, 14:41:52, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:41:52, CET...fine: mar 17 dic 2019, 14:43:18, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:43:18, CET...fine: mar 17 dic 2019, 14:44:29, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:44:29, CET...fine: mar 17 dic 2019, 14:45:55, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:45:55, CET...fine: mar 17 dic 2019, 14:47:06, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:47:06, CET...fine: mar 17 dic 2019, 14:48:32, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:48:32, CET...fine: mar 17 dic 2019, 14:49:43, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:49:43, CET...fine: mar 17 dic 2019, 14:51:10, CET --> ci ho messo 87 secondi!
Inizio: mar 17 dic 2019, 14:51:10, CET...fine: mar 17 dic 2019, 14:52:23, CET --> ci ho messo 73 secondi!
Inizio: mar 17 dic 2019, 14:52:23, CET...fine: mar 17 dic 2019, 14:53:49, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:53:49, CET...fine: mar 17 dic 2019, 14:55:00, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:55:00, CET...fine: mar 17 dic 2019, 14:56:26, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:56:26, CET...fine: mar 17 dic 2019, 14:57:37, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 14:57:37, CET...fine: mar 17 dic 2019, 14:59:03, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 14:59:03, CET...fine: mar 17 dic 2019, 15:00:14, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 15:00:14, CET...fine: mar 17 dic 2019, 15:01:40, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 15:01:40, CET...fine: mar 17 dic 2019, 15:02:51, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 15:02:51, CET...fine: mar 17 dic 2019, 15:04:17, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 15:04:17, CET...fine: mar 17 dic 2019, 15:05:28, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 15:05:28, CET...fine: mar 17 dic 2019, 15:06:53, CET --> ci ho messo 85 secondi!
Inizio: mar 17 dic 2019, 15:06:53, CET...fine: mar 17 dic 2019, 15:08:04, CET --> ci ho messo 70 secondi!
Inizio: mar 17 dic 2019, 15:08:04, CET...fine: mar 17 dic 2019, 15:09:30, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 15:09:30, CET...fine: mar 17 dic 2019, 15:10:41, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 15:10:41, CET...fine: mar 17 dic 2019, 15:12:07, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 15:12:07, CET...fine: mar 17 dic 2019, 15:13:18, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 15:13:18, CET...fine: mar 17 dic 2019, 15:14:44, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 15:14:44, CET...fine: mar 17 dic 2019, 15:15:55, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 15:15:55, CET...fine: mar 17 dic 2019, 15:17:21, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 15:17:21, CET...fine: mar 17 dic 2019, 15:18:32, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 15:18:32, CET...fine: mar 17 dic 2019, 15:19:58, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 15:19:58, CET...fine: mar 17 dic 2019, 15:21:08, CET --> ci ho messo 70 secondi!
Inizio: mar 17 dic 2019, 15:21:08, CET...fine: mar 17 dic 2019, 15:22:34, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 15:22:34, CET...fine: mar 17 dic 2019, 15:23:45, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 15:23:45, CET...fine: mar 17 dic 2019, 15:25:11, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 15:25:11, CET...fine: mar 17 dic 2019, 15:26:22, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 15:26:22, CET...fine: mar 17 dic 2019, 15:27:48, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 15:27:48, CET...fine: mar 17 dic 2019, 15:28:59, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 15:28:59, CET...fine: mar 17 dic 2019, 15:30:25, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 15:30:25, CET...fine: mar 17 dic 2019, 15:31:35, CET --> ci ho messo 70 secondi!
Inizio: mar 17 dic 2019, 15:31:35, CET...fine: mar 17 dic 2019, 15:33:03, CET --> ci ho messo 87 secondi!
Inizio: mar 17 dic 2019, 15:33:03, CET...fine: mar 17 dic 2019, 15:34:14, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 15:34:14, CET...fine: mar 17 dic 2019, 15:35:40, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 15:35:40, CET...fine: mar 17 dic 2019, 15:36:51, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 15:36:51, CET...fine: mar 17 dic 2019, 15:38:17, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 15:38:17, CET...fine: mar 17 dic 2019, 15:39:28, CET --> ci ho messo 71 secondi!
Inizio: mar 17 dic 2019, 15:39:28, CET...fine: mar 17 dic 2019, 15:40:54, CET --> ci ho messo 86 secondi!
Inizio: mar 17 dic 2019, 15:40:54, CET...fine: mar 17 dic 2019, 15:42:05, CET --> ci ho messo 71 secondi!

[-- Attachment #3: test_xfs_20191217.txt --]
[-- Type: text/plain, Size: 11206 bytes --]

*** TEST XFS: *** -> test_xfs_20191217.txt

Starting 100 tries with:
Linux angus.unipv.it 5.4.0+ #1 SMP Mon Nov 25 11:31:34 CET 2019 x86_64 x86_64 x86_64 GNU/Linux
-rw-r--r--. 1 root root 1,0G 25 nov 13.29 /NoBackup/testfile
/dev/sda1: LABEL="Fedora30" UUID="a7ca2491-c807-4b10-b33f-ef425699148d" TYPE="ext4" PARTUUID="8b16fbdd-01"
/dev/sda2: LABEL="Swap_4GB" UUID="ba020b1e-4cdc-4f94-b92c-bdc11613388d" TYPE="swap" PARTUUID="8b16fbdd-02"
/dev/sdf1: UUID="eb5a4791-5b26-44b6-871e-efd464a3adc5" TYPE="xfs" PARTUUID="09066b88-01"
cat /sys/block/sdf/queue/scheduler --> [mq-deadline] none
Inizio: mar 17 dic 2019, 23:58:22, CET...fine: mar 17 dic 2019, 23:59:28, CET --> ci ho messo 64 secondi!
Inizio: mar 17 dic 2019, 23:59:28, CET...fine: mer 18 dic 2019, 00:00:33, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 00:00:33, CET...fine: mer 18 dic 2019, 00:01:39, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 00:01:39, CET...fine: mer 18 dic 2019, 00:06:35, CET --> ci ho messo 294 secondi!
Inizio: mer 18 dic 2019, 00:06:35, CET...fine: mer 18 dic 2019, 00:07:41, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 00:07:41, CET...fine: mer 18 dic 2019, 00:08:46, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 00:08:46, CET...fine: mer 18 dic 2019, 00:09:52, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 00:09:52, CET...fine: mer 18 dic 2019, 00:10:57, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 00:10:57, CET...fine: mer 18 dic 2019, 00:12:03, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 00:12:03, CET...fine: mer 18 dic 2019, 00:13:08, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 00:13:08, CET...fine: mer 18 dic 2019, 00:14:14, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 00:14:14, CET...fine: mer 18 dic 2019, 00:15:19, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 00:15:19, CET...fine: mer 18 dic 2019, 00:21:39, CET --> ci ho messo 379 secondi!
Inizio: mer 18 dic 2019, 00:21:39, CET...fine: mer 18 dic 2019, 00:22:44, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 00:22:44, CET...fine: mer 18 dic 2019, 00:23:50, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 00:23:50, CET...fine: mer 18 dic 2019, 00:29:16, CET --> ci ho messo 325 secondi!
Inizio: mer 18 dic 2019, 00:29:16, CET...fine: mer 18 dic 2019, 00:30:22, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 00:30:22, CET...fine: mer 18 dic 2019, 00:34:50, CET --> ci ho messo 266 secondi!
Inizio: mer 18 dic 2019, 00:34:50, CET...fine: mer 18 dic 2019, 00:35:56, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 00:35:56, CET...fine: mer 18 dic 2019, 00:37:01, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 00:37:01, CET...fine: mer 18 dic 2019, 00:43:39, CET --> ci ho messo 397 secondi!
Inizio: mer 18 dic 2019, 00:43:39, CET...fine: mer 18 dic 2019, 00:48:31, CET --> ci ho messo 291 secondi!
Inizio: mer 18 dic 2019, 00:48:31, CET...fine: mer 18 dic 2019, 00:49:37, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 00:49:37, CET...fine: mer 18 dic 2019, 00:50:42, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 00:50:42, CET...fine: mer 18 dic 2019, 00:55:39, CET --> ci ho messo 296 secondi!
Inizio: mer 18 dic 2019, 00:55:39, CET...fine: mer 18 dic 2019, 00:56:44, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 00:56:44, CET...fine: mer 18 dic 2019, 00:57:50, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 00:57:50, CET...fine: mer 18 dic 2019, 00:58:54, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 00:58:54, CET...fine: mer 18 dic 2019, 01:00:01, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 01:00:01, CET...fine: mer 18 dic 2019, 01:01:05, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 01:01:05, CET...fine: mer 18 dic 2019, 01:02:11, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 01:02:11, CET...fine: mer 18 dic 2019, 01:03:16, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 01:03:16, CET...fine: mer 18 dic 2019, 01:04:22, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 01:04:22, CET...fine: mer 18 dic 2019, 01:05:27, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 01:05:27, CET...fine: mer 18 dic 2019, 01:11:38, CET --> ci ho messo 369 secondi!
Inizio: mer 18 dic 2019, 01:11:38, CET...fine: mer 18 dic 2019, 01:12:43, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 01:12:43, CET...fine: mer 18 dic 2019, 01:18:20, CET --> ci ho messo 336 secondi!
Inizio: mer 18 dic 2019, 01:18:20, CET...fine: mer 18 dic 2019, 01:19:25, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 01:19:25, CET...fine: mer 18 dic 2019, 01:21:01, CET --> ci ho messo 95 secondi!
Inizio: mer 18 dic 2019, 01:21:01, CET...fine: mer 18 dic 2019, 01:22:06, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 01:22:06, CET...fine: mer 18 dic 2019, 01:23:12, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 01:23:12, CET...fine: mer 18 dic 2019, 01:29:43, CET --> ci ho messo 390 secondi!
Inizio: mer 18 dic 2019, 01:29:43, CET...fine: mer 18 dic 2019, 01:30:49, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 01:30:49, CET...fine: mer 18 dic 2019, 01:35:36, CET --> ci ho messo 285 secondi!
Inizio: mer 18 dic 2019, 01:35:36, CET...fine: mer 18 dic 2019, 01:36:44, CET --> ci ho messo 66 secondi!
Inizio: mer 18 dic 2019, 01:36:44, CET...fine: mer 18 dic 2019, 01:37:48, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 01:37:48, CET...fine: mer 18 dic 2019, 01:38:55, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 01:38:55, CET...fine: mer 18 dic 2019, 01:44:04, CET --> ci ho messo 308 secondi!
Inizio: mer 18 dic 2019, 01:44:04, CET...fine: mer 18 dic 2019, 01:45:10, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 01:45:10, CET...fine: mer 18 dic 2019, 01:49:42, CET --> ci ho messo 270 secondi!
Inizio: mer 18 dic 2019, 01:49:42, CET...fine: mer 18 dic 2019, 01:50:48, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 01:50:48, CET...fine: mer 18 dic 2019, 01:51:53, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 01:51:53, CET...fine: mer 18 dic 2019, 01:58:18, CET --> ci ho messo 383 secondi!
Inizio: mer 18 dic 2019, 01:58:18, CET...fine: mer 18 dic 2019, 01:59:23, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 01:59:23, CET...fine: mer 18 dic 2019, 02:00:29, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 02:00:29, CET...fine: mer 18 dic 2019, 02:01:34, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 02:01:34, CET...fine: mer 18 dic 2019, 02:02:40, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:02:40, CET...fine: mer 18 dic 2019, 02:03:45, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 02:03:45, CET...fine: mer 18 dic 2019, 02:04:51, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:04:51, CET...fine: mer 18 dic 2019, 02:05:56, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 02:05:56, CET...fine: mer 18 dic 2019, 02:07:02, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:07:02, CET...fine: mer 18 dic 2019, 02:08:07, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 02:08:07, CET...fine: mer 18 dic 2019, 02:09:14, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 02:09:14, CET...fine: mer 18 dic 2019, 02:13:44, CET --> ci ho messo 269 secondi!
Inizio: mer 18 dic 2019, 02:13:44, CET...fine: mer 18 dic 2019, 02:14:51, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 02:14:51, CET...fine: mer 18 dic 2019, 02:15:56, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:15:56, CET...fine: mer 18 dic 2019, 02:17:02, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 02:17:02, CET...fine: mer 18 dic 2019, 02:18:07, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:18:07, CET...fine: mer 18 dic 2019, 02:19:13, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 02:19:13, CET...fine: mer 18 dic 2019, 02:20:18, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:20:18, CET...fine: mer 18 dic 2019, 02:21:24, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 02:21:24, CET...fine: mer 18 dic 2019, 02:22:29, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:22:29, CET...fine: mer 18 dic 2019, 02:23:35, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 02:23:35, CET...fine: mer 18 dic 2019, 02:24:40, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:24:40, CET...fine: mer 18 dic 2019, 02:25:46, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 02:25:46, CET...fine: mer 18 dic 2019, 02:26:51, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:26:51, CET...fine: mer 18 dic 2019, 02:27:57, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 02:27:57, CET...fine: mer 18 dic 2019, 02:29:02, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:29:02, CET...fine: mer 18 dic 2019, 02:30:08, CET --> ci ho messo 65 secondi!
Inizio: mer 18 dic 2019, 02:30:08, CET...fine: mer 18 dic 2019, 02:31:13, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 02:31:13, CET...fine: mer 18 dic 2019, 02:32:19, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:32:19, CET...fine: mer 18 dic 2019, 02:40:06, CET --> ci ho messo 465 secondi!
Inizio: mer 18 dic 2019, 02:40:06, CET...fine: mer 18 dic 2019, 02:41:12, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:41:12, CET...fine: mer 18 dic 2019, 02:42:17, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 02:42:17, CET...fine: mer 18 dic 2019, 02:43:23, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:43:23, CET...fine: mer 18 dic 2019, 02:44:28, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 02:44:28, CET...fine: mer 18 dic 2019, 02:45:34, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:45:34, CET...fine: mer 18 dic 2019, 02:46:39, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 02:46:39, CET...fine: mer 18 dic 2019, 02:47:45, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:47:45, CET...fine: mer 18 dic 2019, 02:48:50, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 02:48:50, CET...fine: mer 18 dic 2019, 02:54:26, CET --> ci ho messo 334 secondi!
Inizio: mer 18 dic 2019, 02:54:26, CET...fine: mer 18 dic 2019, 02:55:31, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 02:55:31, CET...fine: mer 18 dic 2019, 02:57:11, CET --> ci ho messo 98 secondi!
Inizio: mer 18 dic 2019, 02:57:11, CET...fine: mer 18 dic 2019, 02:58:16, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 02:58:16, CET...fine: mer 18 dic 2019, 02:59:22, CET --> ci ho messo 64 secondi!
Inizio: mer 18 dic 2019, 02:59:22, CET...fine: mer 18 dic 2019, 03:00:27, CET --> ci ho messo 63 secondi!
Inizio: mer 18 dic 2019, 03:00:27, CET...fine: mer 18 dic 2019, 03:05:55, CET --> ci ho messo 326 secondi!
Inizio: mer 18 dic 2019, 03:05:55, CET...fine: mer 18 dic 2019, 03:11:49, CET --> ci ho messo 352 secondi!
Inizio: mer 18 dic 2019, 03:11:49, CET...fine: mer 18 dic 2019, 03:12:56, CET --> ci ho messo 66 secondi!
Inizio: mer 18 dic 2019, 03:12:56, CET...fine: mer 18 dic 2019, 03:14:01, CET --> ci ho messo 63 secondi!


[-- Attachment #4: trace_20191217.zip --]
[-- Type: application/zip, Size: 14385 bytes --]

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-18  8:25                                                                                                       ` Andrea Vai
@ 2019-12-18  9:48                                                                                                         ` Ming Lei
       [not found]                                                                                                           ` <b1b6a0e9d690ecd9432025acd2db4ac09f834040.camel@unipv.it>
  0 siblings, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-12-18  9:48 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Theodore Y. Ts'o, Schmid, Carsten, Finn Thain,
	Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list, linux-ext4,
	linux-fsdevel

On Wed, Dec 18, 2019 at 09:25:02AM +0100, Andrea Vai wrote:
> Il giorno gio, 12/12/2019 alle 05.33 +0800, Ming Lei ha scritto:
> > On Wed, Dec 11, 2019 at 11:07:45AM -0500, Theodore Y. Ts'o wrote:
> > > On Wed, Dec 11, 2019 at 12:00:58PM +0800, Ming Lei wrote:
> > > > I didn't reproduce the issue in my test environment, and follows
> > > > Andrea's test commands[1]:
> > > > 
> > > >   mount UUID=$uuid /mnt/pendrive 2>&1 |tee -a $logfile
> > > >   SECONDS=0
> > > >   cp $testfile /mnt/pendrive 2>&1 |tee -a $logfile
> > > >   umount /mnt/pendrive 2>&1 |tee -a $logfile
> > > > 
> > > > The 'cp' command supposes to open/close the file just once,
> > however
> > > > ext4_release_file() & write pages is observed to run for 4358
> > times
> > > > when executing the above 'cp' test.
> > > 
> > > Why are we sure the ext4_release_file() / _fput() is coming from
> > the
> > > cp command, as opposed to something else that might be running on
> > the
> > > system under test?  _fput() is called by the kernel when the last
> > 
> > Please see the log:
> > 
> > https://lore.kernel.org/linux-scsi/3af3666920e7d46f8f0c6d88612f143ffabc743c.camel@unipv.it/2-log_ming.zip
> > 
> > Which is collected by:
> > 
> > #!/bin/sh
> > MAJ=$1
> > MIN=$2
> > MAJ=$(( $MAJ << 20 ))
> > DEV=$(( $MAJ | $MIN ))
> > 
> > /usr/share/bcc/tools/trace -t -C \
> >     't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args-
> > >rwbs, args->sector, args->nr_sector' \
> >     't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d", args-
> > >rwbs, args->sector, args->nr_sector'
> > 
> > $MAJ:$MIN points to the USB storage disk.
> > 
> > From the above IO trace, there are two write paths, one is from cp,
> > another is from writeback wq.
> > 
> > The stackcount trace[1] is consistent with the IO trace log since it
> > only shows two IO paths, that is why I concluded that the write done
> > via
> > ext4_release_file() is from 'cp'.
> > 
> > [1] 
> > https://lore.kernel.org/linux-scsi/320b315b9c87543d4fb919ecbdf841596c8fbcea.camel@unipv.it/2-log_ming_20191129_150609.zip
> > 
> > > reference to a struct file is released.  (Specifically, if you
> > have a
> > > fd which is dup'ed, it's only when the last fd corresponding to
> > the
> > > struct file is closed, and the struct file is about to be
> > released,
> > > does the file system's f_ops->release function get called.)
> > > 
> > > So the first question I'd ask is whether there is anything else
> > going
> > > on the system, and whether the writes are happening to the USB
> > thumb
> > > drive, or to some other storage device.  And if there is something
> > > else which is writing to the pendrive, maybe that's why no one
> > else
> > > has been able to reproduce the OP's complaint....
> > 
> > OK, we can ask Andrea to confirm that via the following trace, which
> > will add pid/comm info in the stack trace:
> > 
> > /usr/share/bcc/tools/stackcount blk_mq_sched_request_inserted
> > 
> > Andrew, could you collect the above log again when running new/bad
> > kernel for confirming if the write done by ext4_release_file() is
> > from
> > the 'cp' process?
> 
> You can find the stackcount log attached. It has been produced by:
> 
> - /usr/share/bcc/tools/stackcount blk_mq_sched_request_inserted > trace.log
> - wait some seconds
> - run the test (1 copy trial), wait for the test to finish, wait some seconds
> - stop the trace (ctrl+C)

Thanks for collecting the log, looks your 'stackcount' doesn't include
comm/pid info, seems there is difference between your bcc and
my bcc in fedora 30.

Could you collect above log again via the following command?

/usr/share/bcc/tools/stackcount -P -K t:block:block_rq_insert

which will show the comm/pid info.

Sorry for not seeing the bcc difference.

> 
> The test took 1994 seconds to complete.
> 
> I also tried the usual test with btrfs and xfs. Btrfs behavior looks
> "good". xfs seems sometimes better, sometimes worse, I would say. I
> don't know if it matters, anyway you can also find the results of the
> two tests (100 trials each). Basically, btrfs is always between 68 and
> 89 seconds, with a cyclicity (?) with "period=2 trials". xfs looks
> almost always very good (63-65s), but sometimes "bad" (>300s).

If you are interested in digging into this one, the following trace
should be helpful:

https://lore.kernel.org/linux-scsi/f38db337cf26390f7c7488a0bc2076633737775b.camel@unipv.it/T/#m5aa008626e07913172ad40e1eb8e5f2ffd560fc6


Thanks,
Ming

> 
> Thanks,
> Andrea

> *** test btrfs *** -> test_btrfs_20191217.txt
> 
> Starting 100 tries with:
> Linux angus.unipv.it 5.4.0+ #1 SMP Mon Nov 25 11:31:34 CET 2019 x86_64 x86_64 x86_64 GNU/Linux
> -rw-r--r--. 1 root root 1,0G 25 nov 13.29 /NoBackup/testfile
> /dev/sda1: LABEL="Fedora30" UUID="a7ca2491-c807-4b10-b33f-ef425699148d" TYPE="ext4" PARTUUID="8b16fbdd-01"
> /dev/sda2: LABEL="Swap_4GB" UUID="ba020b1e-4cdc-4f94-b92c-bdc11613388d" TYPE="swap" PARTUUID="8b16fbdd-02"
> /dev/sdf1: LABEL="BAK_ANDVAI" UUID="6ddfec28-3d9a-4676-a726-927fd3fb21e7" UUID_SUB="581c69ab-6758-4662-999a-b6dfe6ee5e69" TYPE="btrfs" PARTUUID="09066b88-01"
> /dev/sdg1: LABEL="BAK_ANDVAI" UUID="df777e33-8890-4cee-a718-42233f4cafae" TYPE="ext4" PARTUUID="75265898-01"
> cat /sys/block/sdf/queue/scheduler --> [mq-deadline] none
> Inizio: mar 17 dic 2019, 13:31:00, CET...fine: mar 17 dic 2019, 13:32:26, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 13:32:26, CET...fine: mar 17 dic 2019, 13:33:36, CET --> ci ho messo 70 secondi!
> Inizio: mar 17 dic 2019, 13:33:36, CET...fine: mar 17 dic 2019, 13:35:02, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 13:35:02, CET...fine: mar 17 dic 2019, 13:36:14, CET --> ci ho messo 72 secondi!
> Inizio: mar 17 dic 2019, 13:36:14, CET...fine: mar 17 dic 2019, 13:37:40, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 13:37:40, CET...fine: mar 17 dic 2019, 13:38:51, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 13:38:51, CET...fine: mar 17 dic 2019, 13:40:18, CET --> ci ho messo 87 secondi!
> Inizio: mar 17 dic 2019, 13:40:18, CET...fine: mar 17 dic 2019, 13:41:29, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 13:41:29, CET...fine: mar 17 dic 2019, 13:42:58, CET --> ci ho messo 89 secondi!
> Inizio: mar 17 dic 2019, 13:42:58, CET...fine: mar 17 dic 2019, 13:44:11, CET --> ci ho messo 73 secondi!
> Inizio: mar 17 dic 2019, 13:44:11, CET...fine: mar 17 dic 2019, 13:45:40, CET --> ci ho messo 89 secondi!
> Inizio: mar 17 dic 2019, 13:45:40, CET...fine: mar 17 dic 2019, 13:46:49, CET --> ci ho messo 69 secondi!
> Inizio: mar 17 dic 2019, 13:46:49, CET...fine: mar 17 dic 2019, 13:48:16, CET --> ci ho messo 87 secondi!
> Inizio: mar 17 dic 2019, 13:48:16, CET...fine: mar 17 dic 2019, 13:49:27, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 13:49:27, CET...fine: mar 17 dic 2019, 13:50:53, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 13:50:53, CET...fine: mar 17 dic 2019, 13:52:04, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 13:52:04, CET...fine: mar 17 dic 2019, 13:53:30, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 13:53:30, CET...fine: mar 17 dic 2019, 13:54:41, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 13:54:41, CET...fine: mar 17 dic 2019, 13:56:07, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 13:56:07, CET...fine: mar 17 dic 2019, 13:57:18, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 13:57:18, CET...fine: mar 17 dic 2019, 13:58:46, CET --> ci ho messo 88 secondi!
> Inizio: mar 17 dic 2019, 13:58:46, CET...fine: mar 17 dic 2019, 13:59:57, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 13:59:57, CET...fine: mar 17 dic 2019, 14:01:23, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:01:23, CET...fine: mar 17 dic 2019, 14:02:34, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:02:34, CET...fine: mar 17 dic 2019, 14:04:00, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:04:00, CET...fine: mar 17 dic 2019, 14:05:11, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:05:11, CET...fine: mar 17 dic 2019, 14:06:38, CET --> ci ho messo 87 secondi!
> Inizio: mar 17 dic 2019, 14:06:38, CET...fine: mar 17 dic 2019, 14:07:49, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:07:49, CET...fine: mar 17 dic 2019, 14:09:15, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:09:15, CET...fine: mar 17 dic 2019, 14:10:26, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:10:26, CET...fine: mar 17 dic 2019, 14:11:52, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:11:52, CET...fine: mar 17 dic 2019, 14:13:03, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:13:03, CET...fine: mar 17 dic 2019, 14:14:29, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:14:29, CET...fine: mar 17 dic 2019, 14:15:41, CET --> ci ho messo 72 secondi!
> Inizio: mar 17 dic 2019, 14:15:41, CET...fine: mar 17 dic 2019, 14:17:07, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:17:07, CET...fine: mar 17 dic 2019, 14:18:18, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:18:18, CET...fine: mar 17 dic 2019, 14:19:44, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:19:44, CET...fine: mar 17 dic 2019, 14:20:55, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:20:55, CET...fine: mar 17 dic 2019, 14:22:21, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:22:21, CET...fine: mar 17 dic 2019, 14:23:33, CET --> ci ho messo 72 secondi!
> Inizio: mar 17 dic 2019, 14:23:33, CET...fine: mar 17 dic 2019, 14:24:59, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:24:59, CET...fine: mar 17 dic 2019, 14:26:10, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:26:10, CET...fine: mar 17 dic 2019, 14:27:36, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:27:36, CET...fine: mar 17 dic 2019, 14:28:46, CET --> ci ho messo 70 secondi!
> Inizio: mar 17 dic 2019, 14:28:46, CET...fine: mar 17 dic 2019, 14:30:12, CET --> ci ho messo 85 secondi!
> Inizio: mar 17 dic 2019, 14:30:12, CET...fine: mar 17 dic 2019, 14:31:23, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:31:23, CET...fine: mar 17 dic 2019, 14:32:49, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:32:49, CET...fine: mar 17 dic 2019, 14:34:01, CET --> ci ho messo 72 secondi!
> Inizio: mar 17 dic 2019, 14:34:01, CET...fine: mar 17 dic 2019, 14:35:27, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:35:27, CET...fine: mar 17 dic 2019, 14:36:38, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:36:38, CET...fine: mar 17 dic 2019, 14:38:04, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:38:04, CET...fine: mar 17 dic 2019, 14:39:15, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:39:15, CET...fine: mar 17 dic 2019, 14:40:41, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:40:41, CET...fine: mar 17 dic 2019, 14:41:52, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:41:52, CET...fine: mar 17 dic 2019, 14:43:18, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:43:18, CET...fine: mar 17 dic 2019, 14:44:29, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:44:29, CET...fine: mar 17 dic 2019, 14:45:55, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:45:55, CET...fine: mar 17 dic 2019, 14:47:06, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:47:06, CET...fine: mar 17 dic 2019, 14:48:32, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:48:32, CET...fine: mar 17 dic 2019, 14:49:43, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:49:43, CET...fine: mar 17 dic 2019, 14:51:10, CET --> ci ho messo 87 secondi!
> Inizio: mar 17 dic 2019, 14:51:10, CET...fine: mar 17 dic 2019, 14:52:23, CET --> ci ho messo 73 secondi!
> Inizio: mar 17 dic 2019, 14:52:23, CET...fine: mar 17 dic 2019, 14:53:49, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:53:49, CET...fine: mar 17 dic 2019, 14:55:00, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:55:00, CET...fine: mar 17 dic 2019, 14:56:26, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:56:26, CET...fine: mar 17 dic 2019, 14:57:37, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 14:57:37, CET...fine: mar 17 dic 2019, 14:59:03, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 14:59:03, CET...fine: mar 17 dic 2019, 15:00:14, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 15:00:14, CET...fine: mar 17 dic 2019, 15:01:40, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 15:01:40, CET...fine: mar 17 dic 2019, 15:02:51, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 15:02:51, CET...fine: mar 17 dic 2019, 15:04:17, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 15:04:17, CET...fine: mar 17 dic 2019, 15:05:28, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 15:05:28, CET...fine: mar 17 dic 2019, 15:06:53, CET --> ci ho messo 85 secondi!
> Inizio: mar 17 dic 2019, 15:06:53, CET...fine: mar 17 dic 2019, 15:08:04, CET --> ci ho messo 70 secondi!
> Inizio: mar 17 dic 2019, 15:08:04, CET...fine: mar 17 dic 2019, 15:09:30, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 15:09:30, CET...fine: mar 17 dic 2019, 15:10:41, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 15:10:41, CET...fine: mar 17 dic 2019, 15:12:07, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 15:12:07, CET...fine: mar 17 dic 2019, 15:13:18, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 15:13:18, CET...fine: mar 17 dic 2019, 15:14:44, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 15:14:44, CET...fine: mar 17 dic 2019, 15:15:55, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 15:15:55, CET...fine: mar 17 dic 2019, 15:17:21, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 15:17:21, CET...fine: mar 17 dic 2019, 15:18:32, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 15:18:32, CET...fine: mar 17 dic 2019, 15:19:58, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 15:19:58, CET...fine: mar 17 dic 2019, 15:21:08, CET --> ci ho messo 70 secondi!
> Inizio: mar 17 dic 2019, 15:21:08, CET...fine: mar 17 dic 2019, 15:22:34, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 15:22:34, CET...fine: mar 17 dic 2019, 15:23:45, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 15:23:45, CET...fine: mar 17 dic 2019, 15:25:11, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 15:25:11, CET...fine: mar 17 dic 2019, 15:26:22, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 15:26:22, CET...fine: mar 17 dic 2019, 15:27:48, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 15:27:48, CET...fine: mar 17 dic 2019, 15:28:59, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 15:28:59, CET...fine: mar 17 dic 2019, 15:30:25, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 15:30:25, CET...fine: mar 17 dic 2019, 15:31:35, CET --> ci ho messo 70 secondi!
> Inizio: mar 17 dic 2019, 15:31:35, CET...fine: mar 17 dic 2019, 15:33:03, CET --> ci ho messo 87 secondi!
> Inizio: mar 17 dic 2019, 15:33:03, CET...fine: mar 17 dic 2019, 15:34:14, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 15:34:14, CET...fine: mar 17 dic 2019, 15:35:40, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 15:35:40, CET...fine: mar 17 dic 2019, 15:36:51, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 15:36:51, CET...fine: mar 17 dic 2019, 15:38:17, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 15:38:17, CET...fine: mar 17 dic 2019, 15:39:28, CET --> ci ho messo 71 secondi!
> Inizio: mar 17 dic 2019, 15:39:28, CET...fine: mar 17 dic 2019, 15:40:54, CET --> ci ho messo 86 secondi!
> Inizio: mar 17 dic 2019, 15:40:54, CET...fine: mar 17 dic 2019, 15:42:05, CET --> ci ho messo 71 secondi!

> *** TEST XFS: *** -> test_xfs_20191217.txt
> 
> Starting 100 tries with:
> Linux angus.unipv.it 5.4.0+ #1 SMP Mon Nov 25 11:31:34 CET 2019 x86_64 x86_64 x86_64 GNU/Linux
> -rw-r--r--. 1 root root 1,0G 25 nov 13.29 /NoBackup/testfile
> /dev/sda1: LABEL="Fedora30" UUID="a7ca2491-c807-4b10-b33f-ef425699148d" TYPE="ext4" PARTUUID="8b16fbdd-01"
> /dev/sda2: LABEL="Swap_4GB" UUID="ba020b1e-4cdc-4f94-b92c-bdc11613388d" TYPE="swap" PARTUUID="8b16fbdd-02"
> /dev/sdf1: UUID="eb5a4791-5b26-44b6-871e-efd464a3adc5" TYPE="xfs" PARTUUID="09066b88-01"
> cat /sys/block/sdf/queue/scheduler --> [mq-deadline] none
> Inizio: mar 17 dic 2019, 23:58:22, CET...fine: mar 17 dic 2019, 23:59:28, CET --> ci ho messo 64 secondi!
> Inizio: mar 17 dic 2019, 23:59:28, CET...fine: mer 18 dic 2019, 00:00:33, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 00:00:33, CET...fine: mer 18 dic 2019, 00:01:39, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 00:01:39, CET...fine: mer 18 dic 2019, 00:06:35, CET --> ci ho messo 294 secondi!
> Inizio: mer 18 dic 2019, 00:06:35, CET...fine: mer 18 dic 2019, 00:07:41, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 00:07:41, CET...fine: mer 18 dic 2019, 00:08:46, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 00:08:46, CET...fine: mer 18 dic 2019, 00:09:52, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 00:09:52, CET...fine: mer 18 dic 2019, 00:10:57, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 00:10:57, CET...fine: mer 18 dic 2019, 00:12:03, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 00:12:03, CET...fine: mer 18 dic 2019, 00:13:08, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 00:13:08, CET...fine: mer 18 dic 2019, 00:14:14, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 00:14:14, CET...fine: mer 18 dic 2019, 00:15:19, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 00:15:19, CET...fine: mer 18 dic 2019, 00:21:39, CET --> ci ho messo 379 secondi!
> Inizio: mer 18 dic 2019, 00:21:39, CET...fine: mer 18 dic 2019, 00:22:44, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 00:22:44, CET...fine: mer 18 dic 2019, 00:23:50, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 00:23:50, CET...fine: mer 18 dic 2019, 00:29:16, CET --> ci ho messo 325 secondi!
> Inizio: mer 18 dic 2019, 00:29:16, CET...fine: mer 18 dic 2019, 00:30:22, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 00:30:22, CET...fine: mer 18 dic 2019, 00:34:50, CET --> ci ho messo 266 secondi!
> Inizio: mer 18 dic 2019, 00:34:50, CET...fine: mer 18 dic 2019, 00:35:56, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 00:35:56, CET...fine: mer 18 dic 2019, 00:37:01, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 00:37:01, CET...fine: mer 18 dic 2019, 00:43:39, CET --> ci ho messo 397 secondi!
> Inizio: mer 18 dic 2019, 00:43:39, CET...fine: mer 18 dic 2019, 00:48:31, CET --> ci ho messo 291 secondi!
> Inizio: mer 18 dic 2019, 00:48:31, CET...fine: mer 18 dic 2019, 00:49:37, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 00:49:37, CET...fine: mer 18 dic 2019, 00:50:42, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 00:50:42, CET...fine: mer 18 dic 2019, 00:55:39, CET --> ci ho messo 296 secondi!
> Inizio: mer 18 dic 2019, 00:55:39, CET...fine: mer 18 dic 2019, 00:56:44, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 00:56:44, CET...fine: mer 18 dic 2019, 00:57:50, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 00:57:50, CET...fine: mer 18 dic 2019, 00:58:54, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 00:58:54, CET...fine: mer 18 dic 2019, 01:00:01, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 01:00:01, CET...fine: mer 18 dic 2019, 01:01:05, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 01:01:05, CET...fine: mer 18 dic 2019, 01:02:11, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 01:02:11, CET...fine: mer 18 dic 2019, 01:03:16, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 01:03:16, CET...fine: mer 18 dic 2019, 01:04:22, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 01:04:22, CET...fine: mer 18 dic 2019, 01:05:27, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 01:05:27, CET...fine: mer 18 dic 2019, 01:11:38, CET --> ci ho messo 369 secondi!
> Inizio: mer 18 dic 2019, 01:11:38, CET...fine: mer 18 dic 2019, 01:12:43, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 01:12:43, CET...fine: mer 18 dic 2019, 01:18:20, CET --> ci ho messo 336 secondi!
> Inizio: mer 18 dic 2019, 01:18:20, CET...fine: mer 18 dic 2019, 01:19:25, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 01:19:25, CET...fine: mer 18 dic 2019, 01:21:01, CET --> ci ho messo 95 secondi!
> Inizio: mer 18 dic 2019, 01:21:01, CET...fine: mer 18 dic 2019, 01:22:06, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 01:22:06, CET...fine: mer 18 dic 2019, 01:23:12, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 01:23:12, CET...fine: mer 18 dic 2019, 01:29:43, CET --> ci ho messo 390 secondi!
> Inizio: mer 18 dic 2019, 01:29:43, CET...fine: mer 18 dic 2019, 01:30:49, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 01:30:49, CET...fine: mer 18 dic 2019, 01:35:36, CET --> ci ho messo 285 secondi!
> Inizio: mer 18 dic 2019, 01:35:36, CET...fine: mer 18 dic 2019, 01:36:44, CET --> ci ho messo 66 secondi!
> Inizio: mer 18 dic 2019, 01:36:44, CET...fine: mer 18 dic 2019, 01:37:48, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 01:37:48, CET...fine: mer 18 dic 2019, 01:38:55, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 01:38:55, CET...fine: mer 18 dic 2019, 01:44:04, CET --> ci ho messo 308 secondi!
> Inizio: mer 18 dic 2019, 01:44:04, CET...fine: mer 18 dic 2019, 01:45:10, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 01:45:10, CET...fine: mer 18 dic 2019, 01:49:42, CET --> ci ho messo 270 secondi!
> Inizio: mer 18 dic 2019, 01:49:42, CET...fine: mer 18 dic 2019, 01:50:48, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 01:50:48, CET...fine: mer 18 dic 2019, 01:51:53, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 01:51:53, CET...fine: mer 18 dic 2019, 01:58:18, CET --> ci ho messo 383 secondi!
> Inizio: mer 18 dic 2019, 01:58:18, CET...fine: mer 18 dic 2019, 01:59:23, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 01:59:23, CET...fine: mer 18 dic 2019, 02:00:29, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 02:00:29, CET...fine: mer 18 dic 2019, 02:01:34, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 02:01:34, CET...fine: mer 18 dic 2019, 02:02:40, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:02:40, CET...fine: mer 18 dic 2019, 02:03:45, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 02:03:45, CET...fine: mer 18 dic 2019, 02:04:51, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:04:51, CET...fine: mer 18 dic 2019, 02:05:56, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 02:05:56, CET...fine: mer 18 dic 2019, 02:07:02, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:07:02, CET...fine: mer 18 dic 2019, 02:08:07, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 02:08:07, CET...fine: mer 18 dic 2019, 02:09:14, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 02:09:14, CET...fine: mer 18 dic 2019, 02:13:44, CET --> ci ho messo 269 secondi!
> Inizio: mer 18 dic 2019, 02:13:44, CET...fine: mer 18 dic 2019, 02:14:51, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 02:14:51, CET...fine: mer 18 dic 2019, 02:15:56, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:15:56, CET...fine: mer 18 dic 2019, 02:17:02, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 02:17:02, CET...fine: mer 18 dic 2019, 02:18:07, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:18:07, CET...fine: mer 18 dic 2019, 02:19:13, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 02:19:13, CET...fine: mer 18 dic 2019, 02:20:18, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:20:18, CET...fine: mer 18 dic 2019, 02:21:24, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 02:21:24, CET...fine: mer 18 dic 2019, 02:22:29, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:22:29, CET...fine: mer 18 dic 2019, 02:23:35, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 02:23:35, CET...fine: mer 18 dic 2019, 02:24:40, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:24:40, CET...fine: mer 18 dic 2019, 02:25:46, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 02:25:46, CET...fine: mer 18 dic 2019, 02:26:51, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:26:51, CET...fine: mer 18 dic 2019, 02:27:57, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 02:27:57, CET...fine: mer 18 dic 2019, 02:29:02, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:29:02, CET...fine: mer 18 dic 2019, 02:30:08, CET --> ci ho messo 65 secondi!
> Inizio: mer 18 dic 2019, 02:30:08, CET...fine: mer 18 dic 2019, 02:31:13, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 02:31:13, CET...fine: mer 18 dic 2019, 02:32:19, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:32:19, CET...fine: mer 18 dic 2019, 02:40:06, CET --> ci ho messo 465 secondi!
> Inizio: mer 18 dic 2019, 02:40:06, CET...fine: mer 18 dic 2019, 02:41:12, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:41:12, CET...fine: mer 18 dic 2019, 02:42:17, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 02:42:17, CET...fine: mer 18 dic 2019, 02:43:23, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:43:23, CET...fine: mer 18 dic 2019, 02:44:28, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 02:44:28, CET...fine: mer 18 dic 2019, 02:45:34, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:45:34, CET...fine: mer 18 dic 2019, 02:46:39, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 02:46:39, CET...fine: mer 18 dic 2019, 02:47:45, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:47:45, CET...fine: mer 18 dic 2019, 02:48:50, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 02:48:50, CET...fine: mer 18 dic 2019, 02:54:26, CET --> ci ho messo 334 secondi!
> Inizio: mer 18 dic 2019, 02:54:26, CET...fine: mer 18 dic 2019, 02:55:31, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 02:55:31, CET...fine: mer 18 dic 2019, 02:57:11, CET --> ci ho messo 98 secondi!
> Inizio: mer 18 dic 2019, 02:57:11, CET...fine: mer 18 dic 2019, 02:58:16, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 02:58:16, CET...fine: mer 18 dic 2019, 02:59:22, CET --> ci ho messo 64 secondi!
> Inizio: mer 18 dic 2019, 02:59:22, CET...fine: mer 18 dic 2019, 03:00:27, CET --> ci ho messo 63 secondi!
> Inizio: mer 18 dic 2019, 03:00:27, CET...fine: mer 18 dic 2019, 03:05:55, CET --> ci ho messo 326 secondi!
> Inizio: mer 18 dic 2019, 03:05:55, CET...fine: mer 18 dic 2019, 03:11:49, CET --> ci ho messo 352 secondi!
> Inizio: mer 18 dic 2019, 03:11:49, CET...fine: mer 18 dic 2019, 03:12:56, CET --> ci ho messo 66 secondi!
> Inizio: mer 18 dic 2019, 03:12:56, CET...fine: mer 18 dic 2019, 03:14:01, CET --> ci ho messo 63 secondi!
> 



-- 
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
       [not found]                                                                                                           ` <b1b6a0e9d690ecd9432025acd2db4ac09f834040.camel@unipv.it>
@ 2019-12-23 13:08                                                                                                             ` Ming Lei
  2019-12-23 14:02                                                                                                               ` Andrea Vai
  2019-12-23 16:26                                                                                                               ` Theodore Y. Ts'o
  0 siblings, 2 replies; 104+ messages in thread
From: Ming Lei @ 2019-12-23 13:08 UTC (permalink / raw)
  To: Andrea Vai, Theodore Y. Ts'o
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

On Mon, Dec 23, 2019 at 12:22:45PM +0100, Andrea Vai wrote:
> Il giorno mer, 18/12/2019 alle 17.48 +0800, Ming Lei ha scritto:
> > On Wed, Dec 18, 2019 at 09:25:02AM +0100, Andrea Vai wrote:
> > > Il giorno gio, 12/12/2019 alle 05.33 +0800, Ming Lei ha scritto:
> > > > On Wed, Dec 11, 2019 at 11:07:45AM -0500, Theodore Y. Ts'o
> > wrote:
> > > > > On Wed, Dec 11, 2019 at 12:00:58PM +0800, Ming Lei wrote:
> > > > > > I didn't reproduce the issue in my test environment, and
> > follows
> > > > > > Andrea's test commands[1]:
> > > > > > 
> > > > > >   mount UUID=$uuid /mnt/pendrive 2>&1 |tee -a $logfile
> > > > > >   SECONDS=0
> > > > > >   cp $testfile /mnt/pendrive 2>&1 |tee -a $logfile
> > > > > >   umount /mnt/pendrive 2>&1 |tee -a $logfile
> > > > > > 
> > > > > > The 'cp' command supposes to open/close the file just once,
> > > > however
> > > > > > ext4_release_file() & write pages is observed to run for
> > 4358
> > > > times
> > > > > > when executing the above 'cp' test.
> > > > > 
> > > > > Why are we sure the ext4_release_file() / _fput() is coming
> > from
> > > > the
> > > > > cp command, as opposed to something else that might be running
> > on
> > > > the
> > > > > system under test?  _fput() is called by the kernel when the
> > last
> > > > 
> > > > Please see the log:
> > > > 
> > > > 
> > https://lore.kernel.org/linux-scsi/3af3666920e7d46f8f0c6d88612f143ffabc743c.camel@unipv.it/2-log_ming.zip
> > > > 
> > > > Which is collected by:
> > > > 
> > > > #!/bin/sh
> > > > MAJ=$1
> > > > MIN=$2
> > > > MAJ=$(( $MAJ << 20 ))
> > > > DEV=$(( $MAJ | $MIN ))
> > > > 
> > > > /usr/share/bcc/tools/trace -t -C \
> > > >     't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d",
> > args-
> > > > >rwbs, args->sector, args->nr_sector' \
> > > >     't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d",
> > args-
> > > > >rwbs, args->sector, args->nr_sector'
> > > > 
> > > > $MAJ:$MIN points to the USB storage disk.
> > > > 
> > > > From the above IO trace, there are two write paths, one is from
> > cp,
> > > > another is from writeback wq.
> > > > 
> > > > The stackcount trace[1] is consistent with the IO trace log
> > since it
> > > > only shows two IO paths, that is why I concluded that the write
> > done
> > > > via
> > > > ext4_release_file() is from 'cp'.
> > > > 
> > > > [1] 
> > > > 
> > https://lore.kernel.org/linux-scsi/320b315b9c87543d4fb919ecbdf841596c8fbcea.camel@unipv.it/2-log_ming_20191129_150609.zip
> > > > 
> > > > > reference to a struct file is released.  (Specifically, if you
> > > > have a
> > > > > fd which is dup'ed, it's only when the last fd corresponding
> > to
> > > > the
> > > > > struct file is closed, and the struct file is about to be
> > > > released,
> > > > > does the file system's f_ops->release function get called.)
> > > > > 
> > > > > So the first question I'd ask is whether there is anything
> > else
> > > > going
> > > > > on the system, and whether the writes are happening to the USB
> > > > thumb
> > > > > drive, or to some other storage device.  And if there is
> > something
> > > > > else which is writing to the pendrive, maybe that's why no one
> > > > else
> > > > > has been able to reproduce the OP's complaint....
> > > > 
> > > > OK, we can ask Andrea to confirm that via the following trace,
> > which
> > > > will add pid/comm info in the stack trace:
> > > > 
> > > > /usr/share/bcc/tools/stackcount blk_mq_sched_request_inserted
> > > > 
> > > > Andrew, could you collect the above log again when running
> > new/bad
> > > > kernel for confirming if the write done by ext4_release_file()
> > is
> > > > from
> > > > the 'cp' process?
> > > 
> > > You can find the stackcount log attached. It has been produced by:
> > > 
> > > - /usr/share/bcc/tools/stackcount blk_mq_sched_request_inserted >
> > trace.log
> > > - wait some seconds
> > > - run the test (1 copy trial), wait for the test to finish, wait
> > some seconds
> > > - stop the trace (ctrl+C)
> > 
> > Thanks for collecting the log, looks your 'stackcount' doesn't
> > include
> > comm/pid info, seems there is difference between your bcc and
> > my bcc in fedora 30.
> > 
> > Could you collect above log again via the following command?
> > 
> > /usr/share/bcc/tools/stackcount -P -K t:block:block_rq_insert
> > 
> > which will show the comm/pid info.
> 
> ok, attached (trace_20191219.txt), the test (1 trial) took 3684
> seconds.

From the above trace:

  b'blk_mq_sched_request_inserted'
  b'blk_mq_sched_request_inserted'
  b'dd_insert_requests'
  b'blk_mq_sched_insert_requests'
  b'blk_mq_flush_plug_list'
  b'blk_flush_plug_list'
  b'io_schedule_prepare'
  b'io_schedule'
  b'rq_qos_wait'
  b'wbt_wait'
  b'__rq_qos_throttle'
  b'blk_mq_make_request'
  b'generic_make_request'
  b'submit_bio'
  b'ext4_io_submit'
  b'ext4_writepages'
  b'do_writepages'
  b'__filemap_fdatawrite_range'
  b'ext4_release_file'
  b'__fput'
  b'task_work_run'
  b'exit_to_usermode_loop'
  b'do_syscall_64'
  b'entry_SYSCALL_64_after_hwframe'
    b'cp' [19863]
    4400

So this write is clearly from 'cp' process, and it should be one
ext4 fs issue.

Ted, can you take a look at this issue?

> 
> > > I also tried the usual test with btrfs and xfs. Btrfs behavior
> > looks
> > > "good". xfs seems sometimes better, sometimes worse, I would say.
> > I
> > > don't know if it matters, anyway you can also find the results of
> > the
> > > two tests (100 trials each). Basically, btrfs is always between 68
> > and
> > > 89 seconds, with a cyclicity (?) with "period=2 trials". xfs looks
> > > almost always very good (63-65s), but sometimes "bad" (>300s).
> > 
> > If you are interested in digging into this one, the following trace
> > should be helpful:
> > 
> > https://lore.kernel.org/linux-scsi/f38db337cf26390f7c7488a0bc2076633737775b.camel@unipv.it/T/#m5aa008626e07913172ad40e1eb8e5f2ffd560fc6
> > 
> 
> Attached:
> - trace_xfs_20191223.txt (7 trials, then aborted while doing the 8th),
> times to complete:
> 64s
> 63s
> 64s
> 833s
> 1105s
> 63s
> 64s

oops, looks we have to collect io insert trace with the following bcc script
on xfs for confirming if there is similar issue with ext4, could you run
it again on xfs? And only post the trace done in case of slow 'cp'.


#!/bin/sh

MAJ=$1
MIN=$2
MAJ=$(( $MAJ << 20 ))
DEV=$(( $MAJ | $MIN ))

/usr/share/bcc/tools/trace -t -C \
    't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args->rwbs, args->sector, args->nr_sector' \
    't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d", args->rwbs, args->sector, args->nr_sector'


Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-23 13:08                                                                                                             ` Ming Lei
@ 2019-12-23 14:02                                                                                                               ` Andrea Vai
  2019-12-24  1:32                                                                                                                 ` Ming Lei
  2019-12-23 16:26                                                                                                               ` Theodore Y. Ts'o
  1 sibling, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-12-23 14:02 UTC (permalink / raw)
  To: Ming Lei, Theodore Y. Ts'o
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

[-- Attachment #1: Type: text/plain, Size: 7172 bytes --]

Il giorno lun, 23/12/2019 alle 21.08 +0800, Ming Lei ha scritto:
> On Mon, Dec 23, 2019 at 12:22:45PM +0100, Andrea Vai wrote:
> > Il giorno mer, 18/12/2019 alle 17.48 +0800, Ming Lei ha scritto:
> > > On Wed, Dec 18, 2019 at 09:25:02AM +0100, Andrea Vai wrote:
> > > > Il giorno gio, 12/12/2019 alle 05.33 +0800, Ming Lei ha
> scritto:
> > > > > On Wed, Dec 11, 2019 at 11:07:45AM -0500, Theodore Y. Ts'o
> > > wrote:
> > > > > > On Wed, Dec 11, 2019 at 12:00:58PM +0800, Ming Lei wrote:
> > > > > > > I didn't reproduce the issue in my test environment, and
> > > follows
> > > > > > > Andrea's test commands[1]:
> > > > > > > 
> > > > > > >   mount UUID=$uuid /mnt/pendrive 2>&1 |tee -a $logfile
> > > > > > >   SECONDS=0
> > > > > > >   cp $testfile /mnt/pendrive 2>&1 |tee -a $logfile
> > > > > > >   umount /mnt/pendrive 2>&1 |tee -a $logfile
> > > > > > > 
> > > > > > > The 'cp' command supposes to open/close the file just
> once,
> > > > > however
> > > > > > > ext4_release_file() & write pages is observed to run for
> > > 4358
> > > > > times
> > > > > > > when executing the above 'cp' test.
> > > > > > 
> > > > > > Why are we sure the ext4_release_file() / _fput() is
> coming
> > > from
> > > > > the
> > > > > > cp command, as opposed to something else that might be
> running
> > > on
> > > > > the
> > > > > > system under test?  _fput() is called by the kernel when
> the
> > > last
> > > > > 
> > > > > Please see the log:
> > > > > 
> > > > > 
> > > 
> https://lore.kernel.org/linux-scsi/3af3666920e7d46f8f0c6d88612f143ffabc743c.camel@unipv.it/2-log_ming.zip
> > > > > 
> > > > > Which is collected by:
> > > > > 
> > > > > #!/bin/sh
> > > > > MAJ=$1
> > > > > MIN=$2
> > > > > MAJ=$(( $MAJ << 20 ))
> > > > > DEV=$(( $MAJ | $MIN ))
> > > > > 
> > > > > /usr/share/bcc/tools/trace -t -C \
> > > > >     't:block:block_rq_issue (args->dev == '$DEV') "%s %d
> %d",
> > > args-
> > > > > >rwbs, args->sector, args->nr_sector' \
> > > > >     't:block:block_rq_insert (args->dev == '$DEV') "%s %d
> %d",
> > > args-
> > > > > >rwbs, args->sector, args->nr_sector'
> > > > > 
> > > > > $MAJ:$MIN points to the USB storage disk.
> > > > > 
> > > > > From the above IO trace, there are two write paths, one is
> from
> > > cp,
> > > > > another is from writeback wq.
> > > > > 
> > > > > The stackcount trace[1] is consistent with the IO trace log
> > > since it
> > > > > only shows two IO paths, that is why I concluded that the
> write
> > > done
> > > > > via
> > > > > ext4_release_file() is from 'cp'.
> > > > > 
> > > > > [1] 
> > > > > 
> > > 
> https://lore.kernel.org/linux-scsi/320b315b9c87543d4fb919ecbdf841596c8fbcea.camel@unipv.it/2-log_ming_20191129_150609.zip
> > > > > 
> > > > > > reference to a struct file is released.  (Specifically, if
> you
> > > > > have a
> > > > > > fd which is dup'ed, it's only when the last fd
> corresponding
> > > to
> > > > > the
> > > > > > struct file is closed, and the struct file is about to be
> > > > > released,
> > > > > > does the file system's f_ops->release function get
> called.)
> > > > > > 
> > > > > > So the first question I'd ask is whether there is anything
> > > else
> > > > > going
> > > > > > on the system, and whether the writes are happening to the
> USB
> > > > > thumb
> > > > > > drive, or to some other storage device.  And if there is
> > > something
> > > > > > else which is writing to the pendrive, maybe that's why no
> one
> > > > > else
> > > > > > has been able to reproduce the OP's complaint....
> > > > > 
> > > > > OK, we can ask Andrea to confirm that via the following
> trace,
> > > which
> > > > > will add pid/comm info in the stack trace:
> > > > > 
> > > > > /usr/share/bcc/tools/stackcount
> blk_mq_sched_request_inserted
> > > > > 
> > > > > Andrew, could you collect the above log again when running
> > > new/bad
> > > > > kernel for confirming if the write done by
> ext4_release_file()
> > > is
> > > > > from
> > > > > the 'cp' process?
> > > > 
> > > > You can find the stackcount log attached. It has been produced
> by:
> > > > 
> > > > - /usr/share/bcc/tools/stackcount
> blk_mq_sched_request_inserted >
> > > trace.log
> > > > - wait some seconds
> > > > - run the test (1 copy trial), wait for the test to finish,
> wait
> > > some seconds
> > > > - stop the trace (ctrl+C)
> > > 
> > > Thanks for collecting the log, looks your 'stackcount' doesn't
> > > include
> > > comm/pid info, seems there is difference between your bcc and
> > > my bcc in fedora 30.
> > > 
> > > Could you collect above log again via the following command?
> > > 
> > > /usr/share/bcc/tools/stackcount -P -K t:block:block_rq_insert
> > > 
> > > which will show the comm/pid info.
> > 
> > ok, attached (trace_20191219.txt), the test (1 trial) took 3684
> > seconds.
> 
> From the above trace:
> 
>   b'blk_mq_sched_request_inserted'
>   b'blk_mq_sched_request_inserted'
>   b'dd_insert_requests'
>   b'blk_mq_sched_insert_requests'
>   b'blk_mq_flush_plug_list'
>   b'blk_flush_plug_list'
>   b'io_schedule_prepare'
>   b'io_schedule'
>   b'rq_qos_wait'
>   b'wbt_wait'
>   b'__rq_qos_throttle'
>   b'blk_mq_make_request'
>   b'generic_make_request'
>   b'submit_bio'
>   b'ext4_io_submit'
>   b'ext4_writepages'
>   b'do_writepages'
>   b'__filemap_fdatawrite_range'
>   b'ext4_release_file'
>   b'__fput'
>   b'task_work_run'
>   b'exit_to_usermode_loop'
>   b'do_syscall_64'
>   b'entry_SYSCALL_64_after_hwframe'
>     b'cp' [19863]
>     4400
> 
> So this write is clearly from 'cp' process, and it should be one
> ext4 fs issue.
> 
> Ted, can you take a look at this issue?
> 
> > 
> > > > I also tried the usual test with btrfs and xfs. Btrfs behavior
> > > looks
> > > > "good". xfs seems sometimes better, sometimes worse, I would
> say.
> > > I
> > > > don't know if it matters, anyway you can also find the results
> of
> > > the
> > > > two tests (100 trials each). Basically, btrfs is always
> between 68
> > > and
> > > > 89 seconds, with a cyclicity (?) with "period=2 trials". xfs
> looks
> > > > almost always very good (63-65s), but sometimes "bad" (>300s).
> > > 
> > > If you are interested in digging into this one, the following
> trace
> > > should be helpful:
> > > 
> > > 
> https://lore.kernel.org/linux-scsi/f38db337cf26390f7c7488a0bc2076633737775b.camel@unipv.it/T/#m5aa008626e07913172ad40e1eb8e5f2ffd560fc6
> > > 
> > 
> > Attached:
> > - trace_xfs_20191223.txt (7 trials, then aborted while doing the
> 8th),
> > times to complete:
> > 64s
> > 63s
> > 64s
> > 833s
> > 1105s
> > 63s
> > 64s
> 
> oops, looks we have to collect io insert trace with the following
> bcc script
> on xfs for confirming if there is similar issue with ext4, could you
> run
> it again on xfs? And only post the trace done in case of slow 'cp'.
> 
> 
> #!/bin/sh
> 
> MAJ=$1
> MIN=$2
> MAJ=$(( $MAJ << 20 ))
> DEV=$(( $MAJ | $MIN ))
> 
> /usr/share/bcc/tools/trace -t -C \
>     't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args-
> >rwbs, args->sector, args->nr_sector' \
>     't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d", args-
> >rwbs, args->sector, args->nr_sector'
> 
> 
here it is (1 trial, 313 seconds to finish)

Thanks,
Andrea

[-- Attachment #2: trace_20191223_xfs_new.zip --]
[-- Type: application/zip, Size: 129317 bytes --]

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-23 13:08                                                                                                             ` Ming Lei
  2019-12-23 14:02                                                                                                               ` Andrea Vai
@ 2019-12-23 16:26                                                                                                               ` Theodore Y. Ts'o
  2019-12-23 16:29                                                                                                                 ` Andrea Vai
  1 sibling, 1 reply; 104+ messages in thread
From: Theodore Y. Ts'o @ 2019-12-23 16:26 UTC (permalink / raw)
  To: Ming Lei
  Cc: Andrea Vai, Schmid, Carsten, Finn Thain, Damien Le Moal,
	Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

On Mon, Dec 23, 2019 at 09:08:28PM +0800, Ming Lei wrote:
> 
> From the above trace:
> 
>   b'blk_mq_sched_request_inserted'
>   b'blk_mq_sched_request_inserted'
>   b'dd_insert_requests'
>   b'blk_mq_sched_insert_requests'
>   b'blk_mq_flush_plug_list'
>   b'blk_flush_plug_list'
>   b'io_schedule_prepare'
>   b'io_schedule'
>   b'rq_qos_wait'
>   b'wbt_wait'
>   b'__rq_qos_throttle'
>   b'blk_mq_make_request'
>   b'generic_make_request'
>   b'submit_bio'
>   b'ext4_io_submit'
>   b'ext4_writepages'
>   b'do_writepages'
>   b'__filemap_fdatawrite_range'
>   b'ext4_release_file'
>   b'__fput'
>   b'task_work_run'
>   b'exit_to_usermode_loop'
>   b'do_syscall_64'
>   b'entry_SYSCALL_64_after_hwframe'
>     b'cp' [19863]
>     4400
> 
> So this write is clearly from 'cp' process, and it should be one
> ext4 fs issue.

We need a system call trace of the cp process, to understand what
system call is resulting in fput, (eg., I assume it's close(2) but
let's be sure), and often it's calling that system call.

What cp process is it?  Is it from shellutils?  Is it from busybox?

     		   	      	   		- Ted

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-23 16:26                                                                                                               ` Theodore Y. Ts'o
@ 2019-12-23 16:29                                                                                                                 ` Andrea Vai
  2019-12-23 17:22                                                                                                                   ` Theodore Y. Ts'o
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-12-23 16:29 UTC (permalink / raw)
  To: Theodore Y. Ts'o, Ming Lei
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

Il giorno lun, 23/12/2019 alle 11.26 -0500, Theodore Y. Ts'o ha
scritto:
> On Mon, Dec 23, 2019 at 09:08:28PM +0800, Ming Lei wrote:
> > 
> > From the above trace:
> > 
> >   b'blk_mq_sched_request_inserted'
> >   b'blk_mq_sched_request_inserted'
> >   b'dd_insert_requests'
> >   b'blk_mq_sched_insert_requests'
> >   b'blk_mq_flush_plug_list'
> >   b'blk_flush_plug_list'
> >   b'io_schedule_prepare'
> >   b'io_schedule'
> >   b'rq_qos_wait'
> >   b'wbt_wait'
> >   b'__rq_qos_throttle'
> >   b'blk_mq_make_request'
> >   b'generic_make_request'
> >   b'submit_bio'
> >   b'ext4_io_submit'
> >   b'ext4_writepages'
> >   b'do_writepages'
> >   b'__filemap_fdatawrite_range'
> >   b'ext4_release_file'
> >   b'__fput'
> >   b'task_work_run'
> >   b'exit_to_usermode_loop'
> >   b'do_syscall_64'
> >   b'entry_SYSCALL_64_after_hwframe'
> >     b'cp' [19863]
> >     4400
> > 
> > So this write is clearly from 'cp' process, and it should be one
> > ext4 fs issue.
> 
> We need a system call trace of the cp process, to understand what
> system call is resulting in fput, (eg., I assume it's close(2) but
> let's be sure), and often it's calling that system call.
> 
> What cp process is it?  Is it from shellutils?  Is it from busybox?
> 
>      		   	      	   		- Ted

I run the cp command from a bash script, or from a bash shell. I don't
know if this answer your question, otherwise feel free to tell me a
way to find the answer to give you.

Thanks,
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-23 16:29                                                                                                                 ` Andrea Vai
@ 2019-12-23 17:22                                                                                                                   ` Theodore Y. Ts'o
  2019-12-23 18:45                                                                                                                     ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Theodore Y. Ts'o @ 2019-12-23 17:22 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Ming Lei, Schmid, Carsten, Finn Thain, Damien Le Moal,
	Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

On Mon, Dec 23, 2019 at 05:29:27PM +0100, Andrea Vai wrote:
> I run the cp command from a bash script, or from a bash shell. I don't
> know if this answer your question, otherwise feel free to tell me a
> way to find the answer to give you.

What distro are you using, and/or what package is the cp command
coming from, and what is the package name and version?

Also, can you remind me what the bash script is and how many files you are copying?

Can you change the script so that the cp command is prefixed by:

"strace -tTf -o /tmp/st "

e.g.,

	strace -tTf -o /tmp/st cp <args>

And then send me the /tmp/st file.  This will significantly change the
time, so don't do this for measuring performance.  I just want to see
what the /bin/cp command is *doing*.

      	      	     	     	      - Ted

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-23 17:22                                                                                                                   ` Theodore Y. Ts'o
@ 2019-12-23 18:45                                                                                                                     ` Andrea Vai
  2019-12-23 19:53                                                                                                                       ` Theodore Y. Ts'o
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-12-23 18:45 UTC (permalink / raw)
  To: Theodore Y. Ts'o
  Cc: Ming Lei, Schmid, Carsten, Finn Thain, Damien Le Moal,
	Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

[-- Attachment #1: Type: text/plain, Size: 1537 bytes --]

Il giorno lun, 23/12/2019 alle 12.22 -0500, Theodore Y. Ts'o ha
scritto:
> On Mon, Dec 23, 2019 at 05:29:27PM +0100, Andrea Vai wrote:
> > I run the cp command from a bash script, or from a bash shell. I
> don't
> > know if this answer your question, otherwise feel free to tell me
> a
> > way to find the answer to give you.
> 
> What distro are you using, and/or what package is the cp command
> coming from, and what is the package name and version?

Fedora 30

$ rpm -qf `which cp`
coreutils-8.31-6.fc30.x86_64

> 
> Also, can you remind me what the bash script is and how many files
> you are copying?

basically, it's:

  mount UUID=$uuid /mnt/pendrive
  SECONDS=0
  cp $testfile /mnt/pendrive
  umount /mnt/pendrive
  tempo=$SECONDS

and it copies one file only. Anyway, you can find the whole script
attached.


> 
> Can you change the script so that the cp command is prefixed by:
> 
> "strace -tTf -o /tmp/st "
> 
> e.g.,
> 
> 	strace -tTf -o /tmp/st cp <args>
> 
> And then send me
btw, please tell me if "me" means only you or I cc: all the
recipients, as usual

>  the /tmp/st file.  This will significantly change the
> time, so don't do this for measuring performance.  I just want to
> see
> what the /bin/cp command is *doing*.

I will do it, but I have a doubt. Since the problem doesn't happen
every time, is it useful to give you a trace of a "fast" run? And, if
it's not, I think I should measure performance with the trace command
prefix, to identify a "slow" run to report you. Does it make sense?

Thanks,
Andrea

[-- Attachment #2: test --]
[-- Type: application/x-shellscript, Size: 1403 bytes --]

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-23 18:45                                                                                                                     ` Andrea Vai
@ 2019-12-23 19:53                                                                                                                       ` Theodore Y. Ts'o
  2019-12-24  1:27                                                                                                                         ` Ming Lei
  0 siblings, 1 reply; 104+ messages in thread
From: Theodore Y. Ts'o @ 2019-12-23 19:53 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Ming Lei, Schmid, Carsten, Finn Thain, Damien Le Moal,
	Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

On Mon, Dec 23, 2019 at 07:45:57PM +0100, Andrea Vai wrote:
> basically, it's:
> 
>   mount UUID=$uuid /mnt/pendrive
>   SECONDS=0
>   cp $testfile /mnt/pendrive
>   umount /mnt/pendrive
>   tempo=$SECONDS
> 
> and it copies one file only. Anyway, you can find the whole script
> attached.

OK, so whether we are doing the writeback at the end of cp, or when
you do the umount, it's probably not going to make any difference.  We
can get rid of the stack trace in question by changing the script to
be basically:

mount UUID=$uuid /mnt/pendrive
SECONDS=0
rm -f /mnt/pendrive/$testfile
cp $testfile /mnt/pendrive
umount /mnt/pendrive
tempo=$SECONDS

I predict if you do that, you'll see that all of the time is spent in
the umount, when we are trying to write back the file.

I really don't think then this is a file system problem at all.  It's
just that USB I/O is slow, for whatever reason.  We'll see a stack
trace in the writeback code waiting for the I/O to be completed, but
that doesn't mean that the root cause is in the writeback code or in
the file system which is triggering the writeback.

I suspect the next step is use a blktrace, to see what kind of I/O is
being sent to the USB drive, and how long it takes for the I/O to
complete.  You might also try to capture the output of "iostat -x 1"
while the script is running, and see what the difference might be
between a kernel version that has the problem and one that doesn't,
and see if that gives us a clue.

> > And then send me
> btw, please tell me if "me" means only you or I cc: all the
> recipients, as usual

Well, I don't think we know what the root cause is.  Ming is focusing
on that stack trace, but I think it's a red herring.....  And if it's
not a file system problem, then other people will be best suited to
debug the issue.

   	      	     	   	      	    - Ted

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-23 19:53                                                                                                                       ` Theodore Y. Ts'o
@ 2019-12-24  1:27                                                                                                                         ` Ming Lei
  2019-12-24  6:49                                                                                                                           ` Andrea Vai
                                                                                                                                             ` (2 more replies)
  0 siblings, 3 replies; 104+ messages in thread
From: Ming Lei @ 2019-12-24  1:27 UTC (permalink / raw)
  To: Theodore Y. Ts'o
  Cc: Andrea Vai, Schmid, Carsten, Finn Thain, Damien Le Moal,
	Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

Hi Ted,

On Mon, Dec 23, 2019 at 02:53:01PM -0500, Theodore Y. Ts'o wrote:
> On Mon, Dec 23, 2019 at 07:45:57PM +0100, Andrea Vai wrote:
> > basically, it's:
> > 
> >   mount UUID=$uuid /mnt/pendrive
> >   SECONDS=0
> >   cp $testfile /mnt/pendrive
> >   umount /mnt/pendrive
> >   tempo=$SECONDS
> > 
> > and it copies one file only. Anyway, you can find the whole script
> > attached.
> 
> OK, so whether we are doing the writeback at the end of cp, or when
> you do the umount, it's probably not going to make any difference.  We
> can get rid of the stack trace in question by changing the script to
> be basically:
> 
> mount UUID=$uuid /mnt/pendrive
> SECONDS=0
> rm -f /mnt/pendrive/$testfile
> cp $testfile /mnt/pendrive
> umount /mnt/pendrive
> tempo=$SECONDS
> 
> I predict if you do that, you'll see that all of the time is spent in
> the umount, when we are trying to write back the file.
> 
> I really don't think then this is a file system problem at all.  It's
> just that USB I/O is slow, for whatever reason.  We'll see a stack
> trace in the writeback code waiting for the I/O to be completed, but
> that doesn't mean that the root cause is in the writeback code or in
> the file system which is triggering the writeback.

Wrt. the slow write on this usb storage, it is caused by two writeback
path, one is the writeback wq, another is from ext4_release_file() which
is triggered from exit_to_usermode_loop().

When the two write path is run concurrently, the sequential write order
is broken, then write performance drops much on this particular usb
storage.

The ext4_release_file() should be run from read() or write() syscall if
Fedora 30's 'cp' is implemented correctly. IMO, it isn't expected behavior
for ext4_release_file() to be run thousands of times when just
running 'cp' once, see comment of ext4_release_file():

	/*
	 * Called when an inode is released. Note that this is different
	 * from ext4_file_open: open gets called at every open, but release
	 * gets called only when /all/ the files are closed.
	 */
	static int ext4_release_file(struct inode *inode, struct file *filp)

> 
> I suspect the next step is use a blktrace, to see what kind of I/O is
> being sent to the USB drive, and how long it takes for the I/O to
> complete.  You might also try to capture the output of "iostat -x 1"
> while the script is running, and see what the difference might be
> between a kernel version that has the problem and one that doesn't,
> and see if that gives us a clue.

That isn't necessary, given we have concluded that the bad write
performance is caused by broken write order.

> 
> > > And then send me
> > btw, please tell me if "me" means only you or I cc: all the
> > recipients, as usual
> 
> Well, I don't think we know what the root cause is.  Ming is focusing
> on that stack trace, but I think it's a red herring.....  And if it's
> not a file system problem, then other people will be best suited to
> debug the issue.

So far, the reason points to the extra writeback path from exit_to_usermode_loop().
If it is not from close() syscall, the issue should be related with file reference
count. If it is from close() syscall, the issue might be in 'cp''s
implementation.

Andrea, please collect the following log or the strace log requested by Ted, then
we can confirm if the extra writeback is from close() or read/write() syscall:

# pass PID of 'cp' to this script
#!/bin/sh
PID=$1
/usr/share/bcc/tools/trace -P $PID  -t -C \
    't:block:block_rq_insert "%s %d %d", args->rwbs, args->sector, args->nr_sector' \
    't:syscalls:sys_exit_close ' \
    't:syscalls:sys_exit_read ' \
    't:syscalls:sys_exit_write '


Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-23 14:02                                                                                                               ` Andrea Vai
@ 2019-12-24  1:32                                                                                                                 ` Ming Lei
  2019-12-24  8:04                                                                                                                   ` Andrea Vai
  0 siblings, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-12-24  1:32 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Theodore Y. Ts'o, Schmid, Carsten, Finn Thain,
	Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list, linux-ext4,
	linux-fsdevel

On Mon, Dec 23, 2019 at 03:02:35PM +0100, Andrea Vai wrote:
> Il giorno lun, 23/12/2019 alle 21.08 +0800, Ming Lei ha scritto:
> > On Mon, Dec 23, 2019 at 12:22:45PM +0100, Andrea Vai wrote:
> > > Il giorno mer, 18/12/2019 alle 17.48 +0800, Ming Lei ha scritto:
> > > > On Wed, Dec 18, 2019 at 09:25:02AM +0100, Andrea Vai wrote:
> > > > > Il giorno gio, 12/12/2019 alle 05.33 +0800, Ming Lei ha
> > scritto:
> > > > > > On Wed, Dec 11, 2019 at 11:07:45AM -0500, Theodore Y. Ts'o
> > > > wrote:
> > > > > > > On Wed, Dec 11, 2019 at 12:00:58PM +0800, Ming Lei wrote:
> > > > > > > > I didn't reproduce the issue in my test environment, and
> > > > follows
> > > > > > > > Andrea's test commands[1]:
> > > > > > > > 
> > > > > > > >   mount UUID=$uuid /mnt/pendrive 2>&1 |tee -a $logfile
> > > > > > > >   SECONDS=0
> > > > > > > >   cp $testfile /mnt/pendrive 2>&1 |tee -a $logfile
> > > > > > > >   umount /mnt/pendrive 2>&1 |tee -a $logfile
> > > > > > > > 
> > > > > > > > The 'cp' command supposes to open/close the file just
> > once,
> > > > > > however
> > > > > > > > ext4_release_file() & write pages is observed to run for
> > > > 4358
> > > > > > times
> > > > > > > > when executing the above 'cp' test.
> > > > > > > 
> > > > > > > Why are we sure the ext4_release_file() / _fput() is
> > coming
> > > > from
> > > > > > the
> > > > > > > cp command, as opposed to something else that might be
> > running
> > > > on
> > > > > > the
> > > > > > > system under test?  _fput() is called by the kernel when
> > the
> > > > last
> > > > > > 
> > > > > > Please see the log:
> > > > > > 
> > > > > > 
> > > > 
> > https://lore.kernel.org/linux-scsi/3af3666920e7d46f8f0c6d88612f143ffabc743c.camel@unipv.it/2-log_ming.zip
> > > > > > 
> > > > > > Which is collected by:
> > > > > > 
> > > > > > #!/bin/sh
> > > > > > MAJ=$1
> > > > > > MIN=$2
> > > > > > MAJ=$(( $MAJ << 20 ))
> > > > > > DEV=$(( $MAJ | $MIN ))
> > > > > > 
> > > > > > /usr/share/bcc/tools/trace -t -C \
> > > > > >     't:block:block_rq_issue (args->dev == '$DEV') "%s %d
> > %d",
> > > > args-
> > > > > > >rwbs, args->sector, args->nr_sector' \
> > > > > >     't:block:block_rq_insert (args->dev == '$DEV') "%s %d
> > %d",
> > > > args-
> > > > > > >rwbs, args->sector, args->nr_sector'
> > > > > > 
> > > > > > $MAJ:$MIN points to the USB storage disk.
> > > > > > 
> > > > > > From the above IO trace, there are two write paths, one is
> > from
> > > > cp,
> > > > > > another is from writeback wq.
> > > > > > 
> > > > > > The stackcount trace[1] is consistent with the IO trace log
> > > > since it
> > > > > > only shows two IO paths, that is why I concluded that the
> > write
> > > > done
> > > > > > via
> > > > > > ext4_release_file() is from 'cp'.
> > > > > > 
> > > > > > [1] 
> > > > > > 
> > > > 
> > https://lore.kernel.org/linux-scsi/320b315b9c87543d4fb919ecbdf841596c8fbcea.camel@unipv.it/2-log_ming_20191129_150609.zip
> > > > > > 
> > > > > > > reference to a struct file is released.  (Specifically, if
> > you
> > > > > > have a
> > > > > > > fd which is dup'ed, it's only when the last fd
> > corresponding
> > > > to
> > > > > > the
> > > > > > > struct file is closed, and the struct file is about to be
> > > > > > released,
> > > > > > > does the file system's f_ops->release function get
> > called.)
> > > > > > > 
> > > > > > > So the first question I'd ask is whether there is anything
> > > > else
> > > > > > going
> > > > > > > on the system, and whether the writes are happening to the
> > USB
> > > > > > thumb
> > > > > > > drive, or to some other storage device.  And if there is
> > > > something
> > > > > > > else which is writing to the pendrive, maybe that's why no
> > one
> > > > > > else
> > > > > > > has been able to reproduce the OP's complaint....
> > > > > > 
> > > > > > OK, we can ask Andrea to confirm that via the following
> > trace,
> > > > which
> > > > > > will add pid/comm info in the stack trace:
> > > > > > 
> > > > > > /usr/share/bcc/tools/stackcount
> > blk_mq_sched_request_inserted
> > > > > > 
> > > > > > Andrew, could you collect the above log again when running
> > > > new/bad
> > > > > > kernel for confirming if the write done by
> > ext4_release_file()
> > > > is
> > > > > > from
> > > > > > the 'cp' process?
> > > > > 
> > > > > You can find the stackcount log attached. It has been produced
> > by:
> > > > > 
> > > > > - /usr/share/bcc/tools/stackcount
> > blk_mq_sched_request_inserted >
> > > > trace.log
> > > > > - wait some seconds
> > > > > - run the test (1 copy trial), wait for the test to finish,
> > wait
> > > > some seconds
> > > > > - stop the trace (ctrl+C)
> > > > 
> > > > Thanks for collecting the log, looks your 'stackcount' doesn't
> > > > include
> > > > comm/pid info, seems there is difference between your bcc and
> > > > my bcc in fedora 30.
> > > > 
> > > > Could you collect above log again via the following command?
> > > > 
> > > > /usr/share/bcc/tools/stackcount -P -K t:block:block_rq_insert
> > > > 
> > > > which will show the comm/pid info.
> > > 
> > > ok, attached (trace_20191219.txt), the test (1 trial) took 3684
> > > seconds.
> > 
> > From the above trace:
> > 
> >   b'blk_mq_sched_request_inserted'
> >   b'blk_mq_sched_request_inserted'
> >   b'dd_insert_requests'
> >   b'blk_mq_sched_insert_requests'
> >   b'blk_mq_flush_plug_list'
> >   b'blk_flush_plug_list'
> >   b'io_schedule_prepare'
> >   b'io_schedule'
> >   b'rq_qos_wait'
> >   b'wbt_wait'
> >   b'__rq_qos_throttle'
> >   b'blk_mq_make_request'
> >   b'generic_make_request'
> >   b'submit_bio'
> >   b'ext4_io_submit'
> >   b'ext4_writepages'
> >   b'do_writepages'
> >   b'__filemap_fdatawrite_range'
> >   b'ext4_release_file'
> >   b'__fput'
> >   b'task_work_run'
> >   b'exit_to_usermode_loop'
> >   b'do_syscall_64'
> >   b'entry_SYSCALL_64_after_hwframe'
> >     b'cp' [19863]
> >     4400
> > 
> > So this write is clearly from 'cp' process, and it should be one
> > ext4 fs issue.
> > 
> > Ted, can you take a look at this issue?
> > 
> > > 
> > > > > I also tried the usual test with btrfs and xfs. Btrfs behavior
> > > > looks
> > > > > "good". xfs seems sometimes better, sometimes worse, I would
> > say.
> > > > I
> > > > > don't know if it matters, anyway you can also find the results
> > of
> > > > the
> > > > > two tests (100 trials each). Basically, btrfs is always
> > between 68
> > > > and
> > > > > 89 seconds, with a cyclicity (?) with "period=2 trials". xfs
> > looks
> > > > > almost always very good (63-65s), but sometimes "bad" (>300s).
> > > > 
> > > > If you are interested in digging into this one, the following
> > trace
> > > > should be helpful:
> > > > 
> > > > 
> > https://lore.kernel.org/linux-scsi/f38db337cf26390f7c7488a0bc2076633737775b.camel@unipv.it/T/#m5aa008626e07913172ad40e1eb8e5f2ffd560fc6
> > > > 
> > > 
> > > Attached:
> > > - trace_xfs_20191223.txt (7 trials, then aborted while doing the
> > 8th),
> > > times to complete:
> > > 64s
> > > 63s
> > > 64s
> > > 833s
> > > 1105s
> > > 63s
> > > 64s
> > 
> > oops, looks we have to collect io insert trace with the following
> > bcc script
> > on xfs for confirming if there is similar issue with ext4, could you
> > run
> > it again on xfs? And only post the trace done in case of slow 'cp'.
> > 
> > 
> > #!/bin/sh
> > 
> > MAJ=$1
> > MIN=$2
> > MAJ=$(( $MAJ << 20 ))
> > DEV=$(( $MAJ | $MIN ))
> > 
> > /usr/share/bcc/tools/trace -t -C \
> >     't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args-
> > >rwbs, args->sector, args->nr_sector' \
> >     't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d", args-
> > >rwbs, args->sector, args->nr_sector'
> > 
> > 
> here it is (1 trial, 313 seconds to finish)

The above log shows similar issue with ext4 since there is another
writeback IO path from 'cp' process. And the following trace can show if
it is same with ext4's issue:

/usr/share/bcc/tools/stackcount -P -K t:block:block_rq_insert


Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-24  1:27                                                                                                                         ` Ming Lei
@ 2019-12-24  6:49                                                                                                                           ` Andrea Vai
  2019-12-24  8:51                                                                                                                           ` Andrea Vai
  2019-12-25  5:17                                                                                                                           ` Theodore Y. Ts'o
  2 siblings, 0 replies; 104+ messages in thread
From: Andrea Vai @ 2019-12-24  6:49 UTC (permalink / raw)
  To: Ming Lei, Theodore Y. Ts'o
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

Il giorno mar, 24/12/2019 alle 09.27 +0800, Ming Lei ha scritto:
> Hi Ted,
> 
> On Mon, Dec 23, 2019 at 02:53:01PM -0500, Theodore Y. Ts'o wrote:
> > On Mon, Dec 23, 2019 at 07:45:57PM +0100, Andrea Vai wrote:
> > > basically, it's:
> > > 
> > >   mount UUID=$uuid /mnt/pendrive
> > >   SECONDS=0
> > >   cp $testfile /mnt/pendrive
> > >   umount /mnt/pendrive
> > >   tempo=$SECONDS
> > > 
> > > and it copies one file only. Anyway, you can find the whole
> script
> > > attached.
> > 
> > OK, so whether we are doing the writeback at the end of cp, or
> when
> > you do the umount, it's probably not going to make any
> difference.  We
> > can get rid of the stack trace in question by changing the script
> to
> > be basically:
> > 
> > mount UUID=$uuid /mnt/pendrive
> > SECONDS=0
> > rm -f /mnt/pendrive/$testfile
> > cp $testfile /mnt/pendrive
> > umount /mnt/pendrive
> > tempo=$SECONDS
> > 
> > I predict if you do that, you'll see that all of the time is spent
> in
> > the umount, when we are trying to write back the file.
> > 
> > I really don't think then this is a file system problem at
> all.  It's
> > just that USB I/O is slow, for whatever reason.  We'll see a stack
> > trace in the writeback code waiting for the I/O to be completed,
> but
> > that doesn't mean that the root cause is in the writeback code or
> in
> > the file system which is triggering the writeback.
> 
> Wrt. the slow write on this usb storage, it is caused by two
> writeback
> path, one is the writeback wq, another is from ext4_release_file()
> which
> is triggered from exit_to_usermode_loop().
> 
> When the two write path is run concurrently, the sequential write
> order
> is broken, then write performance drops much on this particular usb
> storage.
> 
> The ext4_release_file() should be run from read() or write() syscall
> if
> Fedora 30's 'cp' is implemented correctly. IMO, it isn't expected
> behavior
> for ext4_release_file() to be run thousands of times when just
> running 'cp' once, see comment of ext4_release_file():
> 
> 	/*
> 	 * Called when an inode is released. Note that this is
> different
> 	 * from ext4_file_open: open gets called at every open, but
> release
> 	 * gets called only when /all/ the files are closed.
> 	 */
> 	static int ext4_release_file(struct inode *inode, struct file
> *filp)
> 
> > 
> > I suspect the next step is use a blktrace, to see what kind of I/O
> is
> > being sent to the USB drive, and how long it takes for the I/O to
> > complete.  You might also try to capture the output of "iostat -x
> 1"
> > while the script is running, and see what the difference might be
> > between a kernel version that has the problem and one that
> doesn't,
> > and see if that gives us a clue.
> 
> That isn't necessary, given we have concluded that the bad write
> performance is caused by broken write order.
> 
> > 
> > > > And then send me
> > > btw, please tell me if "me" means only you or I cc: all the
> > > recipients, as usual
> > 
> > Well, I don't think we know what the root cause is.  Ming is
> focusing
> > on that stack trace, but I think it's a red herring.....  And if
> it's
> > not a file system problem, then other people will be best suited
> to
> > debug the issue.
> 
> So far, the reason points to the extra writeback path from
> exit_to_usermode_loop().
> If it is not from close() syscall, the issue should be related with
> file reference
> count. If it is from close() syscall, the issue might be in 'cp''s
> implementation.
> 
> Andrea, please collect the following log or the strace log requested
> by Ted, then
> we can confirm if the extra writeback is from close() or
> read/write() syscall:
> 
> # pass PID of 'cp' to this script
> #!/bin/sh
> PID=$1
> /usr/share/bcc/tools/trace -P $PID  -t -C \
>     't:block:block_rq_insert "%s %d %d", args->rwbs, args->sector,
> args->nr_sector' \
>     't:syscalls:sys_exit_close ' \
>     't:syscalls:sys_exit_read ' \
>     't:syscalls:sys_exit_write '

Sorry if I am a bit confused, should I run it on ext4 or xfs, or
doesn't matter? What if I get it on a "fast" run? Should I throw it
away and try again until I get a slow one, or it doesn't matter?

Thanks,
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-24  1:32                                                                                                                 ` Ming Lei
@ 2019-12-24  8:04                                                                                                                   ` Andrea Vai
  2019-12-24  8:47                                                                                                                     ` Ming Lei
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-12-24  8:04 UTC (permalink / raw)
  To: Ming Lei
  Cc: Theodore Y. Ts'o, Schmid, Carsten, Finn Thain,
	Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list, linux-ext4,
	linux-fsdevel

Il giorno mar, 24/12/2019 alle 09.32 +0800, Ming Lei ha scritto:
> On Mon, Dec 23, 2019 at 03:02:35PM +0100, Andrea Vai wrote:
> > Il giorno lun, 23/12/2019 alle 21.08 +0800, Ming Lei ha scritto:
> > > On Mon, Dec 23, 2019 at 12:22:45PM +0100, Andrea Vai wrote:
> > > > Il giorno mer, 18/12/2019 alle 17.48 +0800, Ming Lei ha
> scritto:
> > > > > On Wed, Dec 18, 2019 at 09:25:02AM +0100, Andrea Vai wrote:
> > > > > > Il giorno gio, 12/12/2019 alle 05.33 +0800, Ming Lei ha
> > > scritto:
> > > > > > > On Wed, Dec 11, 2019 at 11:07:45AM -0500, Theodore Y.
> Ts'o
> > > > > wrote:
> > > > > > > > On Wed, Dec 11, 2019 at 12:00:58PM +0800, Ming Lei
> wrote:
> > > > > > > > > I didn't reproduce the issue in my test environment,
> and
> > > > > follows
> > > > > > > > > Andrea's test commands[1]:
> > > > > > > > > 
> > > > > > > > >   mount UUID=$uuid /mnt/pendrive 2>&1 |tee -a
> $logfile
> > > > > > > > >   SECONDS=0
> > > > > > > > >   cp $testfile /mnt/pendrive 2>&1 |tee -a $logfile
> > > > > > > > >   umount /mnt/pendrive 2>&1 |tee -a $logfile
> > > > > > > > > 
> > > > > > > > > The 'cp' command supposes to open/close the file
> just
> > > once,
> > > > > > > however
> > > > > > > > > ext4_release_file() & write pages is observed to run
> for
> > > > > 4358
> > > > > > > times
> > > > > > > > > when executing the above 'cp' test.
> > > > > > > > 
> > > > > > > > Why are we sure the ext4_release_file() / _fput() is
> > > coming
> > > > > from
> > > > > > > the
> > > > > > > > cp command, as opposed to something else that might be
> > > running
> > > > > on
> > > > > > > the
> > > > > > > > system under test?  _fput() is called by the kernel
> when
> > > the
> > > > > last
> > > > > > > 
> > > > > > > Please see the log:
> > > > > > > 
> > > > > > > 
> > > > > 
> > > 
> https://lore.kernel.org/linux-scsi/3af3666920e7d46f8f0c6d88612f143ffabc743c.camel@unipv.it/2-log_ming.zip
> > > > > > > 
> > > > > > > Which is collected by:
> > > > > > > 
> > > > > > > #!/bin/sh
> > > > > > > MAJ=$1
> > > > > > > MIN=$2
> > > > > > > MAJ=$(( $MAJ << 20 ))
> > > > > > > DEV=$(( $MAJ | $MIN ))
> > > > > > > 
> > > > > > > /usr/share/bcc/tools/trace -t -C \
> > > > > > >     't:block:block_rq_issue (args->dev == '$DEV') "%s %d
> > > %d",
> > > > > args-
> > > > > > > >rwbs, args->sector, args->nr_sector' \
> > > > > > >     't:block:block_rq_insert (args->dev == '$DEV') "%s
> %d
> > > %d",
> > > > > args-
> > > > > > > >rwbs, args->sector, args->nr_sector'
> > > > > > > 
> > > > > > > $MAJ:$MIN points to the USB storage disk.
> > > > > > > 
> > > > > > > From the above IO trace, there are two write paths, one
> is
> > > from
> > > > > cp,
> > > > > > > another is from writeback wq.
> > > > > > > 
> > > > > > > The stackcount trace[1] is consistent with the IO trace
> log
> > > > > since it
> > > > > > > only shows two IO paths, that is why I concluded that
> the
> > > write
> > > > > done
> > > > > > > via
> > > > > > > ext4_release_file() is from 'cp'.
> > > > > > > 
> > > > > > > [1] 
> > > > > > > 
> > > > > 
> > > 
> https://lore.kernel.org/linux-scsi/320b315b9c87543d4fb919ecbdf841596c8fbcea.camel@unipv.it/2-log_ming_20191129_150609.zip
> > > > > > > 
> > > > > > > > reference to a struct file is
> released.  (Specifically, if
> > > you
> > > > > > > have a
> > > > > > > > fd which is dup'ed, it's only when the last fd
> > > corresponding
> > > > > to
> > > > > > > the
> > > > > > > > struct file is closed, and the struct file is about to
> be
> > > > > > > released,
> > > > > > > > does the file system's f_ops->release function get
> > > called.)
> > > > > > > > 
> > > > > > > > So the first question I'd ask is whether there is
> anything
> > > > > else
> > > > > > > going
> > > > > > > > on the system, and whether the writes are happening to
> the
> > > USB
> > > > > > > thumb
> > > > > > > > drive, or to some other storage device.  And if there
> is
> > > > > something
> > > > > > > > else which is writing to the pendrive, maybe that's
> why no
> > > one
> > > > > > > else
> > > > > > > > has been able to reproduce the OP's complaint....
> > > > > > > 
> > > > > > > OK, we can ask Andrea to confirm that via the following
> > > trace,
> > > > > which
> > > > > > > will add pid/comm info in the stack trace:
> > > > > > > 
> > > > > > > /usr/share/bcc/tools/stackcount
> > > blk_mq_sched_request_inserted
> > > > > > > 
> > > > > > > Andrew, could you collect the above log again when
> running
> > > > > new/bad
> > > > > > > kernel for confirming if the write done by
> > > ext4_release_file()
> > > > > is
> > > > > > > from
> > > > > > > the 'cp' process?
> > > > > > 
> > > > > > You can find the stackcount log attached. It has been
> produced
> > > by:
> > > > > > 
> > > > > > - /usr/share/bcc/tools/stackcount
> > > blk_mq_sched_request_inserted >
> > > > > trace.log
> > > > > > - wait some seconds
> > > > > > - run the test (1 copy trial), wait for the test to
> finish,
> > > wait
> > > > > some seconds
> > > > > > - stop the trace (ctrl+C)
> > > > > 
> > > > > Thanks for collecting the log, looks your 'stackcount'
> doesn't
> > > > > include
> > > > > comm/pid info, seems there is difference between your bcc
> and
> > > > > my bcc in fedora 30.
> > > > > 
> > > > > Could you collect above log again via the following command?
> > > > > 
> > > > > /usr/share/bcc/tools/stackcount -P -K
> t:block:block_rq_insert
> > > > > 
> > > > > which will show the comm/pid info.
> > > > 
> > > > ok, attached (trace_20191219.txt), the test (1 trial) took
> 3684
> > > > seconds.
> > > 
> > > From the above trace:
> > > 
> > >   b'blk_mq_sched_request_inserted'
> > >   b'blk_mq_sched_request_inserted'
> > >   b'dd_insert_requests'
> > >   b'blk_mq_sched_insert_requests'
> > >   b'blk_mq_flush_plug_list'
> > >   b'blk_flush_plug_list'
> > >   b'io_schedule_prepare'
> > >   b'io_schedule'
> > >   b'rq_qos_wait'
> > >   b'wbt_wait'
> > >   b'__rq_qos_throttle'
> > >   b'blk_mq_make_request'
> > >   b'generic_make_request'
> > >   b'submit_bio'
> > >   b'ext4_io_submit'
> > >   b'ext4_writepages'
> > >   b'do_writepages'
> > >   b'__filemap_fdatawrite_range'
> > >   b'ext4_release_file'
> > >   b'__fput'
> > >   b'task_work_run'
> > >   b'exit_to_usermode_loop'
> > >   b'do_syscall_64'
> > >   b'entry_SYSCALL_64_after_hwframe'
> > >     b'cp' [19863]
> > >     4400
> > > 
> > > So this write is clearly from 'cp' process, and it should be one
> > > ext4 fs issue.
> > > 
> > > Ted, can you take a look at this issue?
> > > 
> > > > 
> > > > > > I also tried the usual test with btrfs and xfs. Btrfs
> behavior
> > > > > looks
> > > > > > "good". xfs seems sometimes better, sometimes worse, I
> would
> > > say.
> > > > > I
> > > > > > don't know if it matters, anyway you can also find the
> results
> > > of
> > > > > the
> > > > > > two tests (100 trials each). Basically, btrfs is always
> > > between 68
> > > > > and
> > > > > > 89 seconds, with a cyclicity (?) with "period=2 trials".
> xfs
> > > looks
> > > > > > almost always very good (63-65s), but sometimes "bad"
> (>300s).
> > > > > 
> > > > > If you are interested in digging into this one, the
> following
> > > trace
> > > > > should be helpful:
> > > > > 
> > > > > 
> > > 
> https://lore.kernel.org/linux-scsi/f38db337cf26390f7c7488a0bc2076633737775b.camel@unipv.it/T/#m5aa008626e07913172ad40e1eb8e5f2ffd560fc6
> > > > > 
> > > > 
> > > > Attached:
> > > > - trace_xfs_20191223.txt (7 trials, then aborted while doing
> the
> > > 8th),
> > > > times to complete:
> > > > 64s
> > > > 63s
> > > > 64s
> > > > 833s
> > > > 1105s
> > > > 63s
> > > > 64s
> > > 
> > > oops, looks we have to collect io insert trace with the
> following
> > > bcc script
> > > on xfs for confirming if there is similar issue with ext4, could
> you
> > > run
> > > it again on xfs? And only post the trace done in case of slow
> 'cp'.
> > > 
> > > 
> > > #!/bin/sh
> > > 
> > > MAJ=$1
> > > MIN=$2
> > > MAJ=$(( $MAJ << 20 ))
> > > DEV=$(( $MAJ | $MIN ))
> > > 
> > > /usr/share/bcc/tools/trace -t -C \
> > >     't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d",
> args-
> > > >rwbs, args->sector, args->nr_sector' \
> > >     't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d",
> args-
> > > >rwbs, args->sector, args->nr_sector'
> > > 
> > > 
> > here it is (1 trial, 313 seconds to finish)
> 
> The above log shows similar issue with ext4 since there is another
> writeback IO path from 'cp' process. And the following trace can
> show if
> it is same with ext4's issue:
> 
> /usr/share/bcc/tools/stackcount -P -K t:block:block_rq_insert

sorry, also here please tell me which conditions should I use to run
the test (ext4 or xfs? slow run or not important?)

Thanks,
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-24  8:04                                                                                                                   ` Andrea Vai
@ 2019-12-24  8:47                                                                                                                     ` Ming Lei
  0 siblings, 0 replies; 104+ messages in thread
From: Ming Lei @ 2019-12-24  8:47 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Theodore Y. Ts'o, Schmid, Carsten, Finn Thain,
	Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list, linux-ext4,
	linux-fsdevel

On Tue, Dec 24, 2019 at 09:04:10AM +0100, Andrea Vai wrote:
> Il giorno mar, 24/12/2019 alle 09.32 +0800, Ming Lei ha scritto:
> > On Mon, Dec 23, 2019 at 03:02:35PM +0100, Andrea Vai wrote:
> > > Il giorno lun, 23/12/2019 alle 21.08 +0800, Ming Lei ha scritto:
> > > > On Mon, Dec 23, 2019 at 12:22:45PM +0100, Andrea Vai wrote:
> > > > > Il giorno mer, 18/12/2019 alle 17.48 +0800, Ming Lei ha
> > scritto:
> > > > > > On Wed, Dec 18, 2019 at 09:25:02AM +0100, Andrea Vai wrote:
> > > > > > > Il giorno gio, 12/12/2019 alle 05.33 +0800, Ming Lei ha
> > > > scritto:
> > > > > > > > On Wed, Dec 11, 2019 at 11:07:45AM -0500, Theodore Y.
> > Ts'o
> > > > > > wrote:
> > > > > > > > > On Wed, Dec 11, 2019 at 12:00:58PM +0800, Ming Lei
> > wrote:
> > > > > > > > > > I didn't reproduce the issue in my test environment,
> > and
> > > > > > follows
> > > > > > > > > > Andrea's test commands[1]:
> > > > > > > > > > 
> > > > > > > > > >   mount UUID=$uuid /mnt/pendrive 2>&1 |tee -a
> > $logfile
> > > > > > > > > >   SECONDS=0
> > > > > > > > > >   cp $testfile /mnt/pendrive 2>&1 |tee -a $logfile
> > > > > > > > > >   umount /mnt/pendrive 2>&1 |tee -a $logfile
> > > > > > > > > > 
> > > > > > > > > > The 'cp' command supposes to open/close the file
> > just
> > > > once,
> > > > > > > > however
> > > > > > > > > > ext4_release_file() & write pages is observed to run
> > for
> > > > > > 4358
> > > > > > > > times
> > > > > > > > > > when executing the above 'cp' test.
> > > > > > > > > 
> > > > > > > > > Why are we sure the ext4_release_file() / _fput() is
> > > > coming
> > > > > > from
> > > > > > > > the
> > > > > > > > > cp command, as opposed to something else that might be
> > > > running
> > > > > > on
> > > > > > > > the
> > > > > > > > > system under test?  _fput() is called by the kernel
> > when
> > > > the
> > > > > > last
> > > > > > > > 
> > > > > > > > Please see the log:
> > > > > > > > 
> > > > > > > > 
> > > > > > 
> > > > 
> > https://lore.kernel.org/linux-scsi/3af3666920e7d46f8f0c6d88612f143ffabc743c.camel@unipv.it/2-log_ming.zip
> > > > > > > > 
> > > > > > > > Which is collected by:
> > > > > > > > 
> > > > > > > > #!/bin/sh
> > > > > > > > MAJ=$1
> > > > > > > > MIN=$2
> > > > > > > > MAJ=$(( $MAJ << 20 ))
> > > > > > > > DEV=$(( $MAJ | $MIN ))
> > > > > > > > 
> > > > > > > > /usr/share/bcc/tools/trace -t -C \
> > > > > > > >     't:block:block_rq_issue (args->dev == '$DEV') "%s %d
> > > > %d",
> > > > > > args-
> > > > > > > > >rwbs, args->sector, args->nr_sector' \
> > > > > > > >     't:block:block_rq_insert (args->dev == '$DEV') "%s
> > %d
> > > > %d",
> > > > > > args-
> > > > > > > > >rwbs, args->sector, args->nr_sector'
> > > > > > > > 
> > > > > > > > $MAJ:$MIN points to the USB storage disk.
> > > > > > > > 
> > > > > > > > From the above IO trace, there are two write paths, one
> > is
> > > > from
> > > > > > cp,
> > > > > > > > another is from writeback wq.
> > > > > > > > 
> > > > > > > > The stackcount trace[1] is consistent with the IO trace
> > log
> > > > > > since it
> > > > > > > > only shows two IO paths, that is why I concluded that
> > the
> > > > write
> > > > > > done
> > > > > > > > via
> > > > > > > > ext4_release_file() is from 'cp'.
> > > > > > > > 
> > > > > > > > [1] 
> > > > > > > > 
> > > > > > 
> > > > 
> > https://lore.kernel.org/linux-scsi/320b315b9c87543d4fb919ecbdf841596c8fbcea.camel@unipv.it/2-log_ming_20191129_150609.zip
> > > > > > > > 
> > > > > > > > > reference to a struct file is
> > released.  (Specifically, if
> > > > you
> > > > > > > > have a
> > > > > > > > > fd which is dup'ed, it's only when the last fd
> > > > corresponding
> > > > > > to
> > > > > > > > the
> > > > > > > > > struct file is closed, and the struct file is about to
> > be
> > > > > > > > released,
> > > > > > > > > does the file system's f_ops->release function get
> > > > called.)
> > > > > > > > > 
> > > > > > > > > So the first question I'd ask is whether there is
> > anything
> > > > > > else
> > > > > > > > going
> > > > > > > > > on the system, and whether the writes are happening to
> > the
> > > > USB
> > > > > > > > thumb
> > > > > > > > > drive, or to some other storage device.  And if there
> > is
> > > > > > something
> > > > > > > > > else which is writing to the pendrive, maybe that's
> > why no
> > > > one
> > > > > > > > else
> > > > > > > > > has been able to reproduce the OP's complaint....
> > > > > > > > 
> > > > > > > > OK, we can ask Andrea to confirm that via the following
> > > > trace,
> > > > > > which
> > > > > > > > will add pid/comm info in the stack trace:
> > > > > > > > 
> > > > > > > > /usr/share/bcc/tools/stackcount
> > > > blk_mq_sched_request_inserted
> > > > > > > > 
> > > > > > > > Andrew, could you collect the above log again when
> > running
> > > > > > new/bad
> > > > > > > > kernel for confirming if the write done by
> > > > ext4_release_file()
> > > > > > is
> > > > > > > > from
> > > > > > > > the 'cp' process?
> > > > > > > 
> > > > > > > You can find the stackcount log attached. It has been
> > produced
> > > > by:
> > > > > > > 
> > > > > > > - /usr/share/bcc/tools/stackcount
> > > > blk_mq_sched_request_inserted >
> > > > > > trace.log
> > > > > > > - wait some seconds
> > > > > > > - run the test (1 copy trial), wait for the test to
> > finish,
> > > > wait
> > > > > > some seconds
> > > > > > > - stop the trace (ctrl+C)
> > > > > > 
> > > > > > Thanks for collecting the log, looks your 'stackcount'
> > doesn't
> > > > > > include
> > > > > > comm/pid info, seems there is difference between your bcc
> > and
> > > > > > my bcc in fedora 30.
> > > > > > 
> > > > > > Could you collect above log again via the following command?
> > > > > > 
> > > > > > /usr/share/bcc/tools/stackcount -P -K
> > t:block:block_rq_insert
> > > > > > 
> > > > > > which will show the comm/pid info.
> > > > > 
> > > > > ok, attached (trace_20191219.txt), the test (1 trial) took
> > 3684
> > > > > seconds.
> > > > 
> > > > From the above trace:
> > > > 
> > > >   b'blk_mq_sched_request_inserted'
> > > >   b'blk_mq_sched_request_inserted'
> > > >   b'dd_insert_requests'
> > > >   b'blk_mq_sched_insert_requests'
> > > >   b'blk_mq_flush_plug_list'
> > > >   b'blk_flush_plug_list'
> > > >   b'io_schedule_prepare'
> > > >   b'io_schedule'
> > > >   b'rq_qos_wait'
> > > >   b'wbt_wait'
> > > >   b'__rq_qos_throttle'
> > > >   b'blk_mq_make_request'
> > > >   b'generic_make_request'
> > > >   b'submit_bio'
> > > >   b'ext4_io_submit'
> > > >   b'ext4_writepages'
> > > >   b'do_writepages'
> > > >   b'__filemap_fdatawrite_range'
> > > >   b'ext4_release_file'
> > > >   b'__fput'
> > > >   b'task_work_run'
> > > >   b'exit_to_usermode_loop'
> > > >   b'do_syscall_64'
> > > >   b'entry_SYSCALL_64_after_hwframe'
> > > >     b'cp' [19863]
> > > >     4400
> > > > 
> > > > So this write is clearly from 'cp' process, and it should be one
> > > > ext4 fs issue.
> > > > 
> > > > Ted, can you take a look at this issue?
> > > > 
> > > > > 
> > > > > > > I also tried the usual test with btrfs and xfs. Btrfs
> > behavior
> > > > > > looks
> > > > > > > "good". xfs seems sometimes better, sometimes worse, I
> > would
> > > > say.
> > > > > > I
> > > > > > > don't know if it matters, anyway you can also find the
> > results
> > > > of
> > > > > > the
> > > > > > > two tests (100 trials each). Basically, btrfs is always
> > > > between 68
> > > > > > and
> > > > > > > 89 seconds, with a cyclicity (?) with "period=2 trials".
> > xfs
> > > > looks
> > > > > > > almost always very good (63-65s), but sometimes "bad"
> > (>300s).
> > > > > > 
> > > > > > If you are interested in digging into this one, the
> > following
> > > > trace
> > > > > > should be helpful:
> > > > > > 
> > > > > > 
> > > > 
> > https://lore.kernel.org/linux-scsi/f38db337cf26390f7c7488a0bc2076633737775b.camel@unipv.it/T/#m5aa008626e07913172ad40e1eb8e5f2ffd560fc6
> > > > > > 
> > > > > 
> > > > > Attached:
> > > > > - trace_xfs_20191223.txt (7 trials, then aborted while doing
> > the
> > > > 8th),
> > > > > times to complete:
> > > > > 64s
> > > > > 63s
> > > > > 64s
> > > > > 833s
> > > > > 1105s
> > > > > 63s
> > > > > 64s
> > > > 
> > > > oops, looks we have to collect io insert trace with the
> > following
> > > > bcc script
> > > > on xfs for confirming if there is similar issue with ext4, could
> > you
> > > > run
> > > > it again on xfs? And only post the trace done in case of slow
> > 'cp'.
> > > > 
> > > > 
> > > > #!/bin/sh
> > > > 
> > > > MAJ=$1
> > > > MIN=$2
> > > > MAJ=$(( $MAJ << 20 ))
> > > > DEV=$(( $MAJ | $MIN ))
> > > > 
> > > > /usr/share/bcc/tools/trace -t -C \
> > > >     't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d",
> > args-
> > > > >rwbs, args->sector, args->nr_sector' \
> > > >     't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d",
> > args-
> > > > >rwbs, args->sector, args->nr_sector'
> > > > 
> > > > 
> > > here it is (1 trial, 313 seconds to finish)
> > 
> > The above log shows similar issue with ext4 since there is another
> > writeback IO path from 'cp' process. And the following trace can
> > show if
> > it is same with ext4's issue:
> > 
> > /usr/share/bcc/tools/stackcount -P -K t:block:block_rq_insert
> 
> sorry, also here please tell me which conditions should I use to run
> the test (ext4 or xfs? slow run or not important?)

Maybe not needed.

After thinking the issue further, looks it is highly related with
removing ioc_batching and BDI congestion by blk-mq.

When there are two writeback paths, the original block layer(legacy)
can set 'cp' process which writes pages during close() as 'batching',
then write pages from writeback wq context is blocked. That said there
is actually single writeback IO path even though two are writing pages,
so write order can be maintained, see the following comment in original
__get_request():

	/*
	 * The queue will fill after this allocation, so set
	 * it as full, and mark this process as "batching".
	 * This process will be allowed to complete a batch of
	 * requests, others will be blocked.
	 */

This behavior can be shown in the IO trace done in old kernel with
legacy block IO path:

https://lore.kernel.org/linux-scsi/f82fd5129e3dcacae703a689be60b20a7fedadf6.camel@unipv.it/2-log_ming_20191128_182751.zip

IMO, we need to figure out one solution in blk-mq to fix this issue
since HDD. performance might be hurt under this situation.

Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-24  1:27                                                                                                                         ` Ming Lei
  2019-12-24  6:49                                                                                                                           ` Andrea Vai
@ 2019-12-24  8:51                                                                                                                           ` Andrea Vai
  2019-12-24  9:35                                                                                                                             ` Ming Lei
  2019-12-25  5:17                                                                                                                           ` Theodore Y. Ts'o
  2 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-12-24  8:51 UTC (permalink / raw)
  To: Ming Lei, Theodore Y. Ts'o
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

[-- Attachment #1: Type: text/plain, Size: 4324 bytes --]

Il giorno mar, 24/12/2019 alle 09.27 +0800, Ming Lei ha scritto:
> Hi Ted,
> 
> On Mon, Dec 23, 2019 at 02:53:01PM -0500, Theodore Y. Ts'o wrote:
> > On Mon, Dec 23, 2019 at 07:45:57PM +0100, Andrea Vai wrote:
> > > basically, it's:
> > > 
> > >   mount UUID=$uuid /mnt/pendrive
> > >   SECONDS=0
> > >   cp $testfile /mnt/pendrive
> > >   umount /mnt/pendrive
> > >   tempo=$SECONDS
> > > 
> > > and it copies one file only. Anyway, you can find the whole
> script
> > > attached.
> > 
> > OK, so whether we are doing the writeback at the end of cp, or
> when
> > you do the umount, it's probably not going to make any
> difference.  We
> > can get rid of the stack trace in question by changing the script
> to
> > be basically:
> > 
> > mount UUID=$uuid /mnt/pendrive
> > SECONDS=0
> > rm -f /mnt/pendrive/$testfile
> > cp $testfile /mnt/pendrive
> > umount /mnt/pendrive
> > tempo=$SECONDS
> > 
> > I predict if you do that, you'll see that all of the time is spent
> in
> > the umount, when we are trying to write back the file.
> > 
> > I really don't think then this is a file system problem at
> all.  It's
> > just that USB I/O is slow, for whatever reason.  We'll see a stack
> > trace in the writeback code waiting for the I/O to be completed,
> but
> > that doesn't mean that the root cause is in the writeback code or
> in
> > the file system which is triggering the writeback.
> 
> Wrt. the slow write on this usb storage, it is caused by two
> writeback
> path, one is the writeback wq, another is from ext4_release_file()
> which
> is triggered from exit_to_usermode_loop().
> 
> When the two write path is run concurrently, the sequential write
> order
> is broken, then write performance drops much on this particular usb
> storage.
> 
> The ext4_release_file() should be run from read() or write() syscall
> if
> Fedora 30's 'cp' is implemented correctly. IMO, it isn't expected
> behavior
> for ext4_release_file() to be run thousands of times when just
> running 'cp' once, see comment of ext4_release_file():
> 
> 	/*
> 	 * Called when an inode is released. Note that this is
> different
> 	 * from ext4_file_open: open gets called at every open, but
> release
> 	 * gets called only when /all/ the files are closed.
> 	 */
> 	static int ext4_release_file(struct inode *inode, struct file
> *filp)
> 
> > 
> > I suspect the next step is use a blktrace, to see what kind of I/O
> is
> > being sent to the USB drive, and how long it takes for the I/O to
> > complete.  You might also try to capture the output of "iostat -x
> 1"
> > while the script is running, and see what the difference might be
> > between a kernel version that has the problem and one that
> doesn't,
> > and see if that gives us a clue.
> 
> That isn't necessary, given we have concluded that the bad write
> performance is caused by broken write order.
> 
> > 
> > > > And then send me
> > > btw, please tell me if "me" means only you or I cc: all the
> > > recipients, as usual
> > 
> > Well, I don't think we know what the root cause is.  Ming is
> focusing
> > on that stack trace, but I think it's a red herring.....  And if
> it's
> > not a file system problem, then other people will be best suited
> to
> > debug the issue.
> 
> So far, the reason points to the extra writeback path from
> exit_to_usermode_loop().
> If it is not from close() syscall, the issue should be related with
> file reference
> count. If it is from close() syscall, the issue might be in 'cp''s
> implementation.
> 
> Andrea, please collect the following log or the strace log requested
> by Ted, then
> we can confirm if the extra writeback is from close() or
> read/write() syscall:
> 
> # pass PID of 'cp' to this script
> #!/bin/sh
> PID=$1
> /usr/share/bcc/tools/trace -P $PID  -t -C \
>     't:block:block_rq_insert "%s %d %d", args->rwbs, args->sector,
> args->nr_sector' \
>     't:syscalls:sys_exit_close ' \
>     't:syscalls:sys_exit_read ' \
>     't:syscalls:sys_exit_write '

Meanwhile, I tried to run the test and obtained an error (...usage:
trace [-h] [-b BUFFER_PAGES] [-p PID]...), so assumed the "-P" should
be "-p", corrected and obtained the attached log with ext4 and a slow
copy (2482 seconds) by doing:

- start the test
- look at the cp pid
- run the trace
- wait for the test to finish
- stop the trace.

Thanks,
Andrea

[-- Attachment #2: 20191224_test_ming.zip --]
[-- Type: application/zip, Size: 20830 bytes --]

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-24  8:51                                                                                                                           ` Andrea Vai
@ 2019-12-24  9:35                                                                                                                             ` Ming Lei
  0 siblings, 0 replies; 104+ messages in thread
From: Ming Lei @ 2019-12-24  9:35 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Theodore Y. Ts'o, Schmid, Carsten, Finn Thain,
	Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list, linux-ext4,
	linux-fsdevel

On Tue, Dec 24, 2019 at 09:51:16AM +0100, Andrea Vai wrote:
> Il giorno mar, 24/12/2019 alle 09.27 +0800, Ming Lei ha scritto:
> > Hi Ted,
> > 
> > On Mon, Dec 23, 2019 at 02:53:01PM -0500, Theodore Y. Ts'o wrote:
> > > On Mon, Dec 23, 2019 at 07:45:57PM +0100, Andrea Vai wrote:
> > > > basically, it's:
> > > > 
> > > >   mount UUID=$uuid /mnt/pendrive
> > > >   SECONDS=0
> > > >   cp $testfile /mnt/pendrive
> > > >   umount /mnt/pendrive
> > > >   tempo=$SECONDS
> > > > 
> > > > and it copies one file only. Anyway, you can find the whole
> > script
> > > > attached.
> > > 
> > > OK, so whether we are doing the writeback at the end of cp, or
> > when
> > > you do the umount, it's probably not going to make any
> > difference.  We
> > > can get rid of the stack trace in question by changing the script
> > to
> > > be basically:
> > > 
> > > mount UUID=$uuid /mnt/pendrive
> > > SECONDS=0
> > > rm -f /mnt/pendrive/$testfile
> > > cp $testfile /mnt/pendrive
> > > umount /mnt/pendrive
> > > tempo=$SECONDS
> > > 
> > > I predict if you do that, you'll see that all of the time is spent
> > in
> > > the umount, when we are trying to write back the file.
> > > 
> > > I really don't think then this is a file system problem at
> > all.  It's
> > > just that USB I/O is slow, for whatever reason.  We'll see a stack
> > > trace in the writeback code waiting for the I/O to be completed,
> > but
> > > that doesn't mean that the root cause is in the writeback code or
> > in
> > > the file system which is triggering the writeback.
> > 
> > Wrt. the slow write on this usb storage, it is caused by two
> > writeback
> > path, one is the writeback wq, another is from ext4_release_file()
> > which
> > is triggered from exit_to_usermode_loop().
> > 
> > When the two write path is run concurrently, the sequential write
> > order
> > is broken, then write performance drops much on this particular usb
> > storage.
> > 
> > The ext4_release_file() should be run from read() or write() syscall
> > if
> > Fedora 30's 'cp' is implemented correctly. IMO, it isn't expected
> > behavior
> > for ext4_release_file() to be run thousands of times when just
> > running 'cp' once, see comment of ext4_release_file():
> > 
> > 	/*
> > 	 * Called when an inode is released. Note that this is
> > different
> > 	 * from ext4_file_open: open gets called at every open, but
> > release
> > 	 * gets called only when /all/ the files are closed.
> > 	 */
> > 	static int ext4_release_file(struct inode *inode, struct file
> > *filp)
> > 
> > > 
> > > I suspect the next step is use a blktrace, to see what kind of I/O
> > is
> > > being sent to the USB drive, and how long it takes for the I/O to
> > > complete.  You might also try to capture the output of "iostat -x
> > 1"
> > > while the script is running, and see what the difference might be
> > > between a kernel version that has the problem and one that
> > doesn't,
> > > and see if that gives us a clue.
> > 
> > That isn't necessary, given we have concluded that the bad write
> > performance is caused by broken write order.
> > 
> > > 
> > > > > And then send me
> > > > btw, please tell me if "me" means only you or I cc: all the
> > > > recipients, as usual
> > > 
> > > Well, I don't think we know what the root cause is.  Ming is
> > focusing
> > > on that stack trace, but I think it's a red herring.....  And if
> > it's
> > > not a file system problem, then other people will be best suited
> > to
> > > debug the issue.
> > 
> > So far, the reason points to the extra writeback path from
> > exit_to_usermode_loop().
> > If it is not from close() syscall, the issue should be related with
> > file reference
> > count. If it is from close() syscall, the issue might be in 'cp''s
> > implementation.
> > 
> > Andrea, please collect the following log or the strace log requested
> > by Ted, then
> > we can confirm if the extra writeback is from close() or
> > read/write() syscall:
> > 
> > # pass PID of 'cp' to this script
> > #!/bin/sh
> > PID=$1
> > /usr/share/bcc/tools/trace -P $PID  -t -C \
> >     't:block:block_rq_insert "%s %d %d", args->rwbs, args->sector,
> > args->nr_sector' \
> >     't:syscalls:sys_exit_close ' \
> >     't:syscalls:sys_exit_read ' \
> >     't:syscalls:sys_exit_write '
> 
> Meanwhile, I tried to run the test and obtained an error (...usage:
> trace [-h] [-b BUFFER_PAGES] [-p PID]...), so assumed the "-P" should
> be "-p", corrected and obtained the attached log with ext4 and a slow
> copy (2482 seconds) by doing:
> 
> - start the test
> - look at the cp pid
> - run the trace
> - wait for the test to finish
> - stop the trace.

The log shows all io submission is from close() syscall, so fs code
is fine, and I have provided the reason of this issue in last email:

https://lore.kernel.org/linux-scsi/e3dc2a3e0221c0a0beb91172ba2bff1f6acc0cb7.camel@unipv.it/T/#m845caca2969da5676516c35dc0c3528a79beb886

Thanks, 
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-24  1:27                                                                                                                         ` Ming Lei
  2019-12-24  6:49                                                                                                                           ` Andrea Vai
  2019-12-24  8:51                                                                                                                           ` Andrea Vai
@ 2019-12-25  5:17                                                                                                                           ` Theodore Y. Ts'o
  2019-12-26  2:27                                                                                                                             ` Ming Lei
  2 siblings, 1 reply; 104+ messages in thread
From: Theodore Y. Ts'o @ 2019-12-25  5:17 UTC (permalink / raw)
  To: Ming Lei
  Cc: Andrea Vai, Schmid, Carsten, Finn Thain, Damien Le Moal,
	Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

On Tue, Dec 24, 2019 at 09:27:07AM +0800, Ming Lei wrote:
> The ext4_release_file() should be run from read() or write() syscall if
> Fedora 30's 'cp' is implemented correctly. IMO, it isn't expected behavior
> for ext4_release_file() to be run thousands of times when just
> running 'cp' once, see comment of ext4_release_file():

What's your evidence of that?  As opposed to the writeback taking a
long time, leading to the *one* call of ext4_release_file taking a
long time?  If it's a big file, we might very well be calliing
ext4_writepages multiple times, from a single call to
__filemap_fdatawrite_range().

You confused mightily from that assertion, and that caused me to make
assumptions that cp was doing something crazy.  But I'm quite conviced
now that this is almost certainly not what is happening.

> > I suspect the next step is use a blktrace, to see what kind of I/O is
> > being sent to the USB drive, and how long it takes for the I/O to
> > complete.  You might also try to capture the output of "iostat -x 1"
> > while the script is running, and see what the difference might be
> > between a kernel version that has the problem and one that doesn't,
> > and see if that gives us a clue.
> 
> That isn't necessary, given we have concluded that the bad write
> performance is caused by broken write order.

I didn't see any evidence of that from what I had in my inbox, so I
went back to the mailing list archives to figure out what you were
talking about.  Part of the problem is this has been a very
long-spanning thread, and I had deleted from my inbox all of the parts
relating to the MQ scheduler since that was clearly Not My Problem.  :-)

So, summarizing the most of the thread.  The problem started when we
removed the legacy I/O scheduler, since we are now only using the MQ
scheduler.  What the kernel is sending is long writes (240 sectors),
but it is being sent as an interleaved stream of two sequential
writes.  This particular pendrive can't handle this workload, because
it has a very simplistic Flash Translation Layer.  Now, this is not
*broken*, from a storage perspective; it's just that it's more than
the simple little brain of this particular pen drive can handle.

Previously, with a single queue, and specially since the queue depth
supported by this pen drive is 1, the elevator algorithm would sort
the I/O requests so that it would be mostly sequential, and this
wouldn't be much of a problem.  However, once the legacy I/O stack was
removed, the MQ stack is designed so that we don't have to take a
global lock in order to submit an I/O request.  That also means that
we can't do a full elevator sort since that would require locking all
of the queues.

This is not a problem, since HDD's generally have a 16 deep queue, and
SSD's have a super-deep queue depth since they get their speed via
parallel writes to different flash chips.  Unfortunately, it *is* a
problem for super primitive USB sticks.

> So far, the reason points to the extra writeback path from exit_to_usermode_loop().
> If it is not from close() syscall, the issue should be related with file reference
> count. If it is from close() syscall, the issue might be in 'cp''s
> implementation.

Oh, it's probably from the close system call; and it's *only* from a
single close system call.  Because there is the auto delayed
allocation resolution to protect against buggy userspace, under
certain circumstances, as I explained earlier, we force a full
writeout on a close for a file decsriptor which was opened with an
O_TRUNC.  This is by *design*, since we are trying to protect against
buggy userspace (application programmers vastly outnumber file system
programmers, and far too many of them want O_PONY).  This is Working
As Intended.

You can disable it by deleting the test file before the cp:

    rm -f /mnt/pendrive/$testfile

Or you can disable the protection against stupid userspace by using
the noauto_da_alloc mount option.  (But then if you have a buggy game
program which writes the top-ten score file by using open(2) w/
O_TRUNC, and then said program closes the OpenGL library, and the
proprietary 3rd party binary-only video driver wedges the X server
requiring a hard reset to recover, and the top-ten score file becomes
a zero-length file, don't come crying to me...  Or if a graphical text
editor forgets to use fsync(2) before saving a source file you spent
hours working on, and then the system crashes at exactly the wrong
moment and your source file becomes zero-length, against, don't come
crying to me.  Blame the stupid application programmer which wrote
your text editor who decided to skip the fsync(2), or who decided that
copying the ACL's and xattrs was Too Hard(tm), and so opening the file
with O_TRUNC and rewriting the file in place was easier for the
application programmer.)

In any case, I think this is all working all as intended.  The MQ I/O
stack is optimized for modern HDD and SSD's, and especially SSD's.
And the file system assumes that parallel sequential writes,
especially if they are large, is really not a big deal, since that's
what NCQ or massive parallelism of pretty much all SSD's want.
(Again, ignoring the legacy of crappy flash drives.

You can argue with storage stack folks about whether we need to have
super-dumb mode for slow, crappy flash which uses a global lock and a
global elevator scheduler for super-crappy flash if you want.  I'm
going to stay out of that argument.

					- Ted

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-25  5:17                                                                                                                           ` Theodore Y. Ts'o
@ 2019-12-26  2:27                                                                                                                             ` Ming Lei
  2019-12-26  3:30                                                                                                                               ` Theodore Y. Ts'o
  0 siblings, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-12-26  2:27 UTC (permalink / raw)
  To: Theodore Y. Ts'o
  Cc: Andrea Vai, Schmid, Carsten, Finn Thain, Damien Le Moal,
	Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

On Wed, Dec 25, 2019 at 12:17:22AM -0500, Theodore Y. Ts'o wrote:
> On Tue, Dec 24, 2019 at 09:27:07AM +0800, Ming Lei wrote:
> > The ext4_release_file() should be run from read() or write() syscall if
> > Fedora 30's 'cp' is implemented correctly. IMO, it isn't expected behavior
> > for ext4_release_file() to be run thousands of times when just
> > running 'cp' once, see comment of ext4_release_file():
> 
> What's your evidence of that?  As opposed to the writeback taking a
> long time, leading to the *one* call of ext4_release_file taking a
> long time?  If it's a big file, we might very well be calliing
> ext4_writepages multiple times, from a single call to
> __filemap_fdatawrite_range().
> 
> You confused mightily from that assertion, and that caused me to make
> assumptions that cp was doing something crazy.  But I'm quite conviced
> now that this is almost certainly not what is happening.
> 
> > > I suspect the next step is use a blktrace, to see what kind of I/O is
> > > being sent to the USB drive, and how long it takes for the I/O to
> > > complete.  You might also try to capture the output of "iostat -x 1"
> > > while the script is running, and see what the difference might be
> > > between a kernel version that has the problem and one that doesn't,
> > > and see if that gives us a clue.
> > 
> > That isn't necessary, given we have concluded that the bad write
> > performance is caused by broken write order.
> 
> I didn't see any evidence of that from what I had in my inbox, so I
> went back to the mailing list archives to figure out what you were
> talking about.  Part of the problem is this has been a very
> long-spanning thread, and I had deleted from my inbox all of the parts
> relating to the MQ scheduler since that was clearly Not My Problem.  :-)
> 
> So, summarizing the most of the thread.  The problem started when we
> removed the legacy I/O scheduler, since we are now only using the MQ
> scheduler.  What the kernel is sending is long writes (240 sectors),
> but it is being sent as an interleaved stream of two sequential
> writes.  This particular pendrive can't handle this workload, because
> it has a very simplistic Flash Translation Layer.  Now, this is not
> *broken*, from a storage perspective; it's just that it's more than
> the simple little brain of this particular pen drive can handle.
> 
> Previously, with a single queue, and specially since the queue depth
> supported by this pen drive is 1, the elevator algorithm would sort
> the I/O requests so that it would be mostly sequential, and this
> wouldn't be much of a problem.  However, once the legacy I/O stack was
> removed, the MQ stack is designed so that we don't have to take a
> global lock in order to submit an I/O request.  That also means that
> we can't do a full elevator sort since that would require locking all
> of the queues.
> 
> This is not a problem, since HDD's generally have a 16 deep queue, and
> SSD's have a super-deep queue depth since they get their speed via
> parallel writes to different flash chips.  Unfortunately, it *is* a
> problem for super primitive USB sticks.
> 
> > So far, the reason points to the extra writeback path from exit_to_usermode_loop().
> > If it is not from close() syscall, the issue should be related with file reference
> > count. If it is from close() syscall, the issue might be in 'cp''s
> > implementation.
> 
> Oh, it's probably from the close system call; and it's *only* from a
> single close system call.  Because there is the auto delayed

Right. Looks I mis-interpreted the stackcount log, IOs are submitted
from single close syscall.

> allocation resolution to protect against buggy userspace, under
> certain circumstances, as I explained earlier, we force a full
> writeout on a close for a file decsriptor which was opened with an
> O_TRUNC.  This is by *design*, since we are trying to protect against
> buggy userspace (application programmers vastly outnumber file system
> programmers, and far too many of them want O_PONY).  This is Working
> As Intended.
> 
> You can disable it by deleting the test file before the cp:
> 
>     rm -f /mnt/pendrive/$testfile
> 
> Or you can disable the protection against stupid userspace by using
> the noauto_da_alloc mount option.  (But then if you have a buggy game
> program which writes the top-ten score file by using open(2) w/
> O_TRUNC, and then said program closes the OpenGL library, and the
> proprietary 3rd party binary-only video driver wedges the X server
> requiring a hard reset to recover, and the top-ten score file becomes
> a zero-length file, don't come crying to me...  Or if a graphical text
> editor forgets to use fsync(2) before saving a source file you spent
> hours working on, and then the system crashes at exactly the wrong
> moment and your source file becomes zero-length, against, don't come
> crying to me.  Blame the stupid application programmer which wrote
> your text editor who decided to skip the fsync(2), or who decided that
> copying the ACL's and xattrs was Too Hard(tm), and so opening the file
> with O_TRUNC and rewriting the file in place was easier for the
> application programmer.)
> 
> In any case, I think this is all working all as intended.  The MQ I/O
> stack is optimized for modern HDD and SSD's, and especially SSD's.
> And the file system assumes that parallel sequential writes,
> especially if they are large, is really not a big deal, since that's
> what NCQ or massive parallelism of pretty much all SSD's want.
> (Again, ignoring the legacy of crappy flash drives.
> 
> You can argue with storage stack folks about whether we need to have
> super-dumb mode for slow, crappy flash which uses a global lock and a
> global elevator scheduler for super-crappy flash if you want.  I'm
> going to stay out of that argument.

As I mentioned in the following link:

https://lore.kernel.org/linux-scsi/20191224084721.GA27248@ming.t460p/

The reason is that ioc_batching and BDI congestion is removed by blk-mq.

Then after queue is congested, multiple sequential writes can be done
concurrently at the same time. Before ioc_batching and BDI congestion is
removed, writes are done serialized from multiple processes actually, so
IOs are dispatched to drive in strict sequential order.

This way can't be an issue for SSD.

Maybe we need to be careful for HDD., since the request count in scheduler
queue is double of in-flight request count, and in theory NCQ should only
cover all in-flight 32 requests. I will find a sata HDD., and see if
performance drop can be observed in the similar 'cp' test.


Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-26  2:27                                                                                                                             ` Ming Lei
@ 2019-12-26  3:30                                                                                                                               ` Theodore Y. Ts'o
  2019-12-26  8:37                                                                                                                                 ` Ming Lei
       [not found]                                                                                                                                 ` <20200101074310.10904-1-hdanton@sina.com>
  0 siblings, 2 replies; 104+ messages in thread
From: Theodore Y. Ts'o @ 2019-12-26  3:30 UTC (permalink / raw)
  To: Ming Lei
  Cc: Andrea Vai, Schmid, Carsten, Finn Thain, Damien Le Moal,
	Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

On Thu, Dec 26, 2019 at 10:27:02AM +0800, Ming Lei wrote:
> Maybe we need to be careful for HDD., since the request count in scheduler
> queue is double of in-flight request count, and in theory NCQ should only
> cover all in-flight 32 requests. I will find a sata HDD., and see if
> performance drop can be observed in the similar 'cp' test.

Please try to measure it, but I'd be really surprised if it's
significant with with modern HDD's.  That because they typically have
a queue depth of 16, and a max_sectors_kb of 32767 (e.g., just under
32 MiB).  Sort seeks are typically 1-2 ms, with full stroke seeks
8-10ms.  Typical sequential write speeds on a 7200 RPM drive is
125-150 MiB/s.  So suppose every other request sent to the HDD is from
the other request stream.  The disk will chose the 8 requests from its
queue that are contiguous, and so it will be writing around 256 MiB,
which will take 2-3 seconds.  If it then needs to spend between 1 and
10 ms seeking to another location of the disk, before it writes the
next 256 MiB, the worst case overhead of that seek is 10ms / 2s, or
0.5%.  That may very well be within your measurements' error bars.

And of course, note that in real life, we are very *often* writing to
multiple files in parallel, for example, during a "make -j16" while
building the kernel.  Writing a single large file is certainly
something people do (but even there people who are burning a 4G DVD
rip are often browsing the web while they are waiting for it to
complete, and the browser will be writing cache files, etc.).  So
whether or not this is something where we should be stressing over
this specific workload is going to be quite debateable.

						- Ted

^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-26  3:30                                                                                                                               ` Theodore Y. Ts'o
@ 2019-12-26  8:37                                                                                                                                 ` Ming Lei
  2020-01-07  7:51                                                                                                                                   ` Andrea Vai
       [not found]                                                                                                                                 ` <20200101074310.10904-1-hdanton@sina.com>
  1 sibling, 1 reply; 104+ messages in thread
From: Ming Lei @ 2019-12-26  8:37 UTC (permalink / raw)
  To: Theodore Y. Ts'o
  Cc: Andrea Vai, Schmid, Carsten, Finn Thain, Damien Le Moal,
	Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
	SCSI development list, Himanshu Madhani, Hannes Reinecke,
	Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

On Wed, Dec 25, 2019 at 10:30:57PM -0500, Theodore Y. Ts'o wrote:
> On Thu, Dec 26, 2019 at 10:27:02AM +0800, Ming Lei wrote:
> > Maybe we need to be careful for HDD., since the request count in scheduler
> > queue is double of in-flight request count, and in theory NCQ should only
> > cover all in-flight 32 requests. I will find a sata HDD., and see if
> > performance drop can be observed in the similar 'cp' test.
> 
> Please try to measure it, but I'd be really surprised if it's
> significant with with modern HDD's.

Just find one machine with AHCI SATA, and run the following xfs
overwrite test:

#!/bin/bash
DIR=$1
echo 3 > /proc/sys/vm/drop_caches
fio --readwrite=write --filesize=5g --overwrite=1 --filename=$DIR/fiofile \
        --runtime=60s --time_based --ioengine=psync --direct=0 --bs=4k
		--iodepth=128 --numjobs=2 --group_reporting=1 --name=overwrite

FS is xfs, and disk is LVM over AHCI SATA with NCQ(depth 32), because the
machine is picked up from RH beaker, and it is the only disk in the box.

#lsblk
NAME                            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                               8:0    0 931.5G  0 disk 
├─sda1                            8:1    0     1G  0 part /boot
└─sda2                            8:2    0 930.5G  0 part 
  ├─rhel_hpe--ml10gen9--01-root 253:0    0    50G  0 lvm  /
  ├─rhel_hpe--ml10gen9--01-swap 253:1    0   3.9G  0 lvm  [SWAP]
  └─rhel_hpe--ml10gen9--01-home 253:2    0 876.6G  0 lvm  /home


kernel: 3a7ea2c483a53fc("scsi: provide mq_ops->busy() hook") which is
the previous commit of f664a3cc17b7 ("scsi: kill off the legacy IO path").

            |scsi_mod.use_blk_mq=N |scsi_mod.use_blk_mq=Y |
-----------------------------------------------------------
throughput: |244MB/s               |169MB/s               |
-----------------------------------------------------------

Similar result can be observed on v5.4 kernel(184MB/s) with same test
steps.


> That because they typically have
> a queue depth of 16, and a max_sectors_kb of 32767 (e.g., just under
> 32 MiB).  Sort seeks are typically 1-2 ms, with full stroke seeks
> 8-10ms.  Typical sequential write speeds on a 7200 RPM drive is
> 125-150 MiB/s.  So suppose every other request sent to the HDD is from
> the other request stream.  The disk will chose the 8 requests from its
> queue that are contiguous, and so it will be writing around 256 MiB,
> which will take 2-3 seconds.  If it then needs to spend between 1 and
> 10 ms seeking to another location of the disk, before it writes the
> next 256 MiB, the worst case overhead of that seek is 10ms / 2s, or
> 0.5%.  That may very well be within your measurements' error bars.

Looks you assume that disk seeking just happens once when writing around
256MB. This assumption may not be true, given all data can be in page
cache before writing. So when two tasks are submitting IOs concurrently,
IOs from each single task is sequential, and NCQ may order the current batch
submitted from the two streams. However disk seeking may still be needed
for the next batch handled by NCQ.

> And of course, note that in real life, we are very *often* writing to
> multiple files in parallel, for example, during a "make -j16" while
> building the kernel.  Writing a single large file is certainly
> something people do (but even there people who are burning a 4G DVD
> rip are often browsing the web while they are waiting for it to
> complete, and the browser will be writing cache files, etc.).  So
> whether or not this is something where we should be stressing over
> this specific workload is going to be quite debateable.

Thanks, 
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: slow IO on USB media
       [not found]                                                                                                                                 ` <20200101074310.10904-1-hdanton@sina.com>
@ 2020-01-01 13:53                                                                                                                                   ` Ming Lei
  0 siblings, 0 replies; 104+ messages in thread
From: Ming Lei @ 2020-01-01 13:53 UTC (permalink / raw)
  To: Hillf Danton
  Cc: Theodore Y. Ts'o, Andrea Vai, Schmid, Carsten, Finn Thain,
	Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
	USB list, SCSI development list, Himanshu Madhani,
	Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
	Hans Holmberg, Kernel development list, linux-ext4,
	linux-fsdevel

On Wed, Jan 01, 2020 at 03:43:10PM +0800, Hillf Danton wrote:
> 
> On Thu, 26 Dec 2019 16:37:06 +0800 Ming Lei wrote:
> > On Wed, Dec 25, 2019 at 10:30:57PM -0500, Theodore Y. Ts'o wrote:
> > > On Thu, Dec 26, 2019 at 10:27:02AM +0800, Ming Lei wrote:
> > > > Maybe we need to be careful for HDD., since the request count in scheduler
> > > > queue is double of in-flight request count, and in theory NCQ should only
> > > > cover all in-flight 32 requests. I will find a sata HDD., and see if
> > > > performance drop can be observed in the similar 'cp' test.
> > >
> > > Please try to measure it, but I'd be really surprised if it's
> > > significant with with modern HDD's.
> > 
> > Just find one machine with AHCI SATA, and run the following xfs
> > overwrite test:
> > 
> > #!/bin/bash
> > DIR=$1
> > echo 3 > /proc/sys/vm/drop_caches
> > fio --readwrite=write --filesize=5g --overwrite=1 --filename=$DIR/fiofile \
> >         --runtime=60s --time_based --ioengine=psync --direct=0 --bs=4k
> > 		--iodepth=128 --numjobs=2 --group_reporting=1 --name=overwrite
> > 
> > FS is xfs, and disk is LVM over AHCI SATA with NCQ(depth 32), because the
> > machine is picked up from RH beaker, and it is the only disk in the box.
> > 
> > #lsblk
> > NAME                            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
> > sda                               8:0    0 931.5G  0 disk
> > =E2=94=9C=E2=94=80sda1                            8:1    0     1G  0 part /boot
> > =E2=94=94=E2=94=80sda2                            8:2    0 930.5G  0 part
> >   =E2=94=9C=E2=94=80rhel_hpe--ml10gen9--01-root 253:0    0    50G  0 lvm  /
> >   =E2=94=9C=E2=94=80rhel_hpe--ml10gen9--01-swap 253:1    0   3.9G  0 lvm  [SWAP]
> >   =E2=94=94=E2=94=80rhel_hpe--ml10gen9--01-home 253:2    0 876.6G  0 lvm  /home
> > 
> > 
> > kernel: 3a7ea2c483a53fc("scsi: provide mq_ops->busy() hook") which is
> > the previous commit of f664a3cc17b7 ("scsi: kill off the legacy IO path").
> > 
> >             |scsi_mod.use_blk_mq=N |scsi_mod.use_blk_mq=Y |
> > -----------------------------------------------------------
> > throughput: |244MB/s               |169MB/s               |
> > -----------------------------------------------------------
> > 
> > Similar result can be observed on v5.4 kernel(184MB/s) with same test
> > steps.
> 
> 
> The simple diff makes direct issue of requests take pending requests
> also into account and goes the nornal enqueue-and-dequeue path if any
> pending requests exist.
> 
> Then it sorts requests regardless of the number of hard queues in a
> bid to make requests as sequencial as they are. Let's see if the
> added sorting cost can make any sense.
> 
> --->8---
> 
> --- a/block/blk-mq-sched.c
> +++ b/block/blk-mq-sched.c
> @@ -410,6 +410,11 @@ run:
>  		blk_mq_run_hw_queue(hctx, async);
>  }
>  
> +static inline bool blk_mq_sched_hctx_has_pending_rq(struct blk_mq_hw_ctx *hctx)
> +{
> +	return sbitmap_any_bit_set(&hctx->ctx_map);
> +}
> +
>  void blk_mq_sched_insert_requests(struct blk_mq_hw_ctx *hctx,
>  				  struct blk_mq_ctx *ctx,
>  				  struct list_head *list, bool run_queue_async)
> @@ -433,7 +438,8 @@ void blk_mq_sched_insert_requests(struct
>  		 * busy in case of 'none' scheduler, and this way may save
>  		 * us one extra enqueue & dequeue to sw queue.
>  		 */
> -		if (!hctx->dispatch_busy && !e && !run_queue_async) {
> +		if (!hctx->dispatch_busy && !e && !run_queue_async &&
> +		    !blk_mq_sched_hctx_has_pending_rq(hctx)) {
>  			blk_mq_try_issue_list_directly(hctx, list);
>  			if (list_empty(list))
>  				goto out;
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -1692,7 +1692,7 @@ void blk_mq_flush_plug_list(struct blk_p
>  
>  	list_splice_init(&plug->mq_list, &list);
>  
> -	if (plug->rq_count > 2 && plug->multiple_queues)
> +	if (plug->rq_count > 1)
>  		list_sort(NULL, &list, plug_rq_cmp);
>  
>  	plug->rq_count = 0;

I guess you may not understand the reason, and the issue is related
with neither MQ nor plug.

AHCI/SATA is single queue drive, and for HDD. IO throughput is very
sensitive with IO order in case of sequential IO.

Legacy IO path supports ioc batching and BDI queue congestion. When
there are more than one writeback IO paths, there may be only one
active IO submission path, meantime others are blocked attributed to
ioc batching, so writeback IO is still dispatched to disk in strict
IO order.

But ioc batching and BDI queue congestion is killed when converting to
blk-mq.

Please see the following IO trace with legacy IO request path:

https://lore.kernel.org/linux-scsi/f82fd5129e3dcacae703a689be60b20a7fedadf6.camel@unipv.it/2-log_ming_20191128_182751.zip


Thanks,
Ming


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-12-26  8:37                                                                                                                                 ` Ming Lei
@ 2020-01-07  7:51                                                                                                                                   ` Andrea Vai
  0 siblings, 0 replies; 104+ messages in thread
From: Andrea Vai @ 2020-01-07  7:51 UTC (permalink / raw)
  To: Ming Lei, Theodore Y. Ts'o
  Cc: Schmid, Carsten, Finn Thain, Damien Le Moal, Alan Stern,
	Jens Axboe, Johannes Thumshirn, USB list, SCSI development list,
	Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
	Martin K. Petersen, Greg KH, Hans Holmberg,
	Kernel development list, linux-ext4, linux-fsdevel

Il giorno gio, 26/12/2019 alle 16.37 +0800, Ming Lei ha scritto:
> On Wed, Dec 25, 2019 at 10:30:57PM -0500, Theodore Y. Ts'o wrote:
> > On Thu, Dec 26, 2019 at 10:27:02AM +0800, Ming Lei wrote:
> > > Maybe we need to be careful for HDD., since the request count in
> scheduler
> > > queue is double of in-flight request count, and in theory NCQ
> should only
> > > cover all in-flight 32 requests. I will find a sata HDD., and
> see if
> > > performance drop can be observed in the similar 'cp' test.
> > 
> > Please try to measure it, but I'd be really surprised if it's
> > significant with with modern HDD's.
> 
> Just find one machine with AHCI SATA, and run the following xfs
> overwrite test:
> 
> #!/bin/bash
> DIR=$1
> echo 3 > /proc/sys/vm/drop_caches
> fio --readwrite=write --filesize=5g --overwrite=1 --
> filename=$DIR/fiofile \
>         --runtime=60s --time_based --ioengine=psync --direct=0 --
> bs=4k
> 		--iodepth=128 --numjobs=2 --group_reporting=1 --
> name=overwrite
> 
> FS is xfs, and disk is LVM over AHCI SATA with NCQ(depth 32),
> because the
> machine is picked up from RH beaker, and it is the only disk in the
> box.
> 
> #lsblk
> NAME                            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
> sda                               8:0    0 931.5G  0 disk 
> ├─sda1                            8:1    0     1G  0 part /boot
> └─sda2                            8:2    0 930.5G  0 part 
>   ├─rhel_hpe--ml10gen9--01-root 253:0    0    50G  0 lvm  /
>   ├─rhel_hpe--ml10gen9--01-swap 253:1    0   3.9G  0 lvm  [SWAP]
>   └─rhel_hpe--ml10gen9--01-home 253:2    0 876.6G  0 lvm  /home
> 
> 
> kernel: 3a7ea2c483a53fc("scsi: provide mq_ops->busy() hook") which
> is
> the previous commit of f664a3cc17b7 ("scsi: kill off the legacy IO
> path").
> 
>             |scsi_mod.use_blk_mq=N |scsi_mod.use_blk_mq=Y |
> -----------------------------------------------------------
> throughput: |244MB/s               |169MB/s               |
> -----------------------------------------------------------
> 
> Similar result can be observed on v5.4 kernel(184MB/s) with same
> test
> steps.
> 
> 
> > That because they typically have
> > a queue depth of 16, and a max_sectors_kb of 32767 (e.g., just
> under
> > 32 MiB).  Sort seeks are typically 1-2 ms, with full stroke seeks
> > 8-10ms.  Typical sequential write speeds on a 7200 RPM drive is
> > 125-150 MiB/s.  So suppose every other request sent to the HDD is
> from
> > the other request stream.  The disk will chose the 8 requests from
> its
> > queue that are contiguous, and so it will be writing around 256
> MiB,
> > which will take 2-3 seconds.  If it then needs to spend between 1
> and
> > 10 ms seeking to another location of the disk, before it writes
> the
> > next 256 MiB, the worst case overhead of that seek is 10ms / 2s,
> or
> > 0.5%.  That may very well be within your measurements' error bars.
> 
> Looks you assume that disk seeking just happens once when writing
> around
> 256MB. This assumption may not be true, given all data can be in
> page
> cache before writing. So when two tasks are submitting IOs
> concurrently,
> IOs from each single task is sequential, and NCQ may order the
> current batch
> submitted from the two streams. However disk seeking may still be
> needed
> for the next batch handled by NCQ.
> 
> > And of course, note that in real life, we are very *often* writing
> to
> > multiple files in parallel, for example, during a "make -j16"
> while
> > building the kernel.  Writing a single large file is certainly
> > something people do (but even there people who are burning a 4G
> DVD
> > rip are often browsing the web while they are waiting for it to
> > complete, and the browser will be writing cache files, etc.).  So
> > whether or not this is something where we should be stressing over
> > this specific workload is going to be quite debateable.
> 

Hi,
  is there any update on this? Sorry if I am making noise, but I would
like to help to improve the kernel (or fix it) if I can help.
Otherwise, please let me know how to consider this case,

Thanks, and bye
Andrea


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
  2019-08-09 12:07 ` Andrea Vai
@ 2019-08-13 19:52   ` Alan Stern
  0 siblings, 0 replies; 104+ messages in thread
From: Alan Stern @ 2019-08-13 19:52 UTC (permalink / raw)
  To: Andrea Vai
  Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH

On Fri, 9 Aug 2019, Andrea Vai wrote:

> Il giorno lun, 08/07/2019 alle 11.38 -0400, Alan Stern ha scritto:
> > 
> > [...]
> >  Andrea, another thing you could try is to collect a usbmon trace
> > under 
> > > > one of the "slow" kernels.  Follow the instructions in 
> > > > Documentation/usb/usbmon.txt.  I think you could kill the file-
> > copy 
> > > > operation after just a couple of seconds; that should provide
> > enough 
> > > > trace information to help see what causes the slowdown.
> > > > 
> > > > (If you want, do the same test with a "fast" kernel and then
> > we'll 
> > > > compare the results.)
> > > > 
> > > > Alan Stern
> > > > 
> > > 
> > > Thanks Alan,
> > >   so I attach two ouputs, one for a "good" and one for a "bad"
> > kernel.
> > 
> > [...]
> > I don't know what the results mean, but I _can_ tell you what's
> > happening.  Every so often (at intervals of about a second) the pen
> > drive completely stops communicating with the "bad" kernel for about
> > one second and then starts up again.
> > 
> > Here's a short example from the "bad" trace:
> > 
> > ffff9169f0d399c0 513072808 S Bo:6:008:2 -115 122880 = 00000000
> > 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> > ffff9169f0d399c0 514262176 C Bo:6:008:2 0 122880 >
> > 
> > The second column is a timestamp (in microseconds).  This little
> > extract shows a 120-KB write starting at time 513.072 and ending at
> > 514.262, more than a second later.  Normally such a write would
> > complete in about 0.06 s.
> > 
> > The cumulative effect of all these delays is to slow the transfer 
> > drastically.  The "good" kernel trace shows a few delays like this,
> > but 
> > only one or two.
> > 
> > I have no idea how commit f664a3cc17b7, getting rid of the legacy
> > IO 
> > path, could have caused these delays.  It seems more likely that
> > the 
> > pen drive itself is the cause, perhaps because it is flushing
> > buffers 
> > more often under the "bad" kernel.
> > 
> > I'd like you to try doing another pair of usbmon tests.  This time,
> > start collecting the usbmon trace _before_ you plug in the pen
> > drive,
> > and stop the trace shortly after the pen drive has been mounted.  
> > Don't try to transfer any data.  Perhaps the two kernels are
> > initializing the pen drive with different settings and that accounts
> > for the different behaviors.
> 
> Thanks Alan, sorry for the delay (I haven't had physical access to the
> device for a while). I attach the two new usbmon logs.
> 
> Meanwhile, I can try with other pendrives, although I already tried
> two more and both behave "bad" (but one is identical to the first, and
> the other one is similar).

I looked through your two traces.  Unfortunately they don't help --
they are essentially identical.  Exactly the same sequence of commands
was sent to the device in both traces, except for a couple of
inconsequential TEST UNIT READY commands.

There are some timing differences, but they occur in between commands,
not during a command.  They probably are the times you spent typing
stuff into the terminal.

Can you run another test?  Similar to the last one, but this time start
collecting the usbmon trace at least 10 seconds after the drive is
plugged in.  Then mount the drive, copy a small file (say less than 500
bytes) to it, unmount it, and then stop the trace.

Alan Stern


^ permalink raw reply	[flat|nested] 104+ messages in thread

* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
       [not found] <Pine.LNX.4.44L0.1907081101260.1424-100000@iolanthe.rowland.org>
@ 2019-08-09 12:07 ` Andrea Vai
  2019-08-13 19:52   ` Alan Stern
  0 siblings, 1 reply; 104+ messages in thread
From: Andrea Vai @ 2019-08-09 12:07 UTC (permalink / raw)
  To: Alan Stern
  Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
	Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
	Martin K. Petersen, Greg KH

[-- Attachment #1: Type: text/plain, Size: 2533 bytes --]

Il giorno lun, 08/07/2019 alle 11.38 -0400, Alan Stern ha scritto:
> 
> [...]
>  Andrea, another thing you could try is to collect a usbmon trace
> under 
> > > one of the "slow" kernels.  Follow the instructions in 
> > > Documentation/usb/usbmon.txt.  I think you could kill the file-
> copy 
> > > operation after just a couple of seconds; that should provide
> enough 
> > > trace information to help see what causes the slowdown.
> > > 
> > > (If you want, do the same test with a "fast" kernel and then
> we'll 
> > > compare the results.)
> > > 
> > > Alan Stern
> > > 
> > 
> > Thanks Alan,
> >   so I attach two ouputs, one for a "good" and one for a "bad"
> kernel.
> 
> [...]
> I don't know what the results mean, but I _can_ tell you what's
> happening.  Every so often (at intervals of about a second) the pen
> drive completely stops communicating with the "bad" kernel for about
> one second and then starts up again.
> 
> Here's a short example from the "bad" trace:
> 
> ffff9169f0d399c0 513072808 S Bo:6:008:2 -115 122880 = 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> ffff9169f0d399c0 514262176 C Bo:6:008:2 0 122880 >
> 
> The second column is a timestamp (in microseconds).  This little
> extract shows a 120-KB write starting at time 513.072 and ending at
> 514.262, more than a second later.  Normally such a write would
> complete in about 0.06 s.
> 
> The cumulative effect of all these delays is to slow the transfer 
> drastically.  The "good" kernel trace shows a few delays like this,
> but 
> only one or two.
> 
> I have no idea how commit f664a3cc17b7, getting rid of the legacy
> IO 
> path, could have caused these delays.  It seems more likely that
> the 
> pen drive itself is the cause, perhaps because it is flushing
> buffers 
> more often under the "bad" kernel.
> 
> I'd like you to try doing another pair of usbmon tests.  This time,
> start collecting the usbmon trace _before_ you plug in the pen
> drive,
> and stop the trace shortly after the pen drive has been mounted.  
> Don't try to transfer any data.  Perhaps the two kernels are
> initializing the pen drive with different settings and that accounts
> for the different behaviors.

Thanks Alan, sorry for the delay (I haven't had physical access to the
device for a while). I attach the two new usbmon logs.

Meanwhile, I can try with other pendrives, although I already tried
two more and both behave "bad" (but one is identical to the first, and
the other one is similar).

Thanks, and bye
Andrea

[-- Attachment #2: bad.mon.out --]
[-- Type: text/plain, Size: 142928 bytes --]

ffff8a35f0776d80 664543488 S Bo:6:003:1 -115 31 = 55534243 e10a0000 00000000 00030600 00000000 00000000 00000000 000000
ffff8a35f0776d80 664543546 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 664543555 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 664543923 C Bi:6:003:2 0 13 = 55534253 e10a0000 00000000 01
ffff8a35f0776d80 664543930 S Bo:6:003:1 -115 31 = 55534243 e20a0000 12000000 80030603 00000012 00000000 00000000 000000
ffff8a35f0776d80 664543974 C Bo:6:003:1 0 31 >
ffff8a3527547180 664543981 S Bi:6:003:2 -115 18 <
ffff8a3527547180 664544510 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 664544521 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 664544656 C Bi:6:003:2 0 13 = 55534253 e20a0000 00000000 00
ffff8a35e91f69c0 664584688 S Ci:6:002:0 s a3 00 0000 0001 0004 4 <
ffff8a35e91f69c0 664584742 C Ci:6:002:0 0 4 = 07050000
ffff8a3527547c00 664584764 S Co:6:002:0 s 23 01 0002 0001 0000 0
ffff8a3527547c00 664584788 C Co:6:002:0 0 0
ffff8a3527547c00 664626531 S Ci:6:002:0 s a3 00 0000 0001 0004 4 <
ffff8a3527547c00 664626577 C Ci:6:002:0 0 4 = 03050400
ffff8a3527547c00 664638574 S Co:6:002:0 s 23 01 0012 0001 0000 0
ffff8a3527547c00 664638629 C Co:6:002:0 0 0
ffff8a3527547c00 664638642 S Ci:6:005:0 s 80 00 0000 0000 0002 2 <
ffff8a3527547c00 664638672 C Ci:6:005:0 0 2 = 0300
ffff8a3527547c00 664638696 S Co:6:005:0 s 00 01 0001 0000 0000 0
ffff8a3527547c00 664638714 C Co:6:005:0 0 0
ffff8a3527547c00 664638731 S Ci:6:005:0 s a3 00 0000 0001 0004 4 <
ffff8a3527547c00 664638755 C Ci:6:005:0 0 4 = 00010000
ffff8a3527547c00 664638786 S Ci:6:005:0 s a3 00 0000 0002 0004 4 <
ffff8a3527547c00 664638805 C Ci:6:005:0 0 4 = 00010000
ffff8a3527547c00 664638821 S Ci:6:005:0 s a3 00 0000 0003 0004 4 <
ffff8a3527547c00 664638885 C Ci:6:005:0 0 4 = 00010000
ffff8a3527547c00 664638914 S Ci:6:005:0 s a3 00 0000 0004 0004 4 <
ffff8a3527547c00 664638955 C Ci:6:005:0 0 4 = 00010000
ffff8a35ef1d5780 664638968 S Ii:6:005:1 -115:2048 1 <
ffff8a35ef1d5780 664639102 C Ii:6:005:1 -2:2048 0
ffff8a3527547f00 664639120 S Co:6:005:0 s 00 03 0001 0000 0000 0
ffff8a3527547f00 664639138 C Co:6:005:0 0 0
ffff8a3527547f00 664639152 S Co:6:002:0 s 23 03 0002 0001 0000 0
ffff8a3527547f00 664639180 C Co:6:002:0 0 0
ffff8a35f0776d80 666527511 S Bo:6:003:1 -115 31 = 55534243 e30a0000 00000000 00020600 00000000 00000000 00000000 000000
ffff8a35f0776d80 666527572 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 666527584 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 666527989 C Bi:6:003:2 0 13 = 55534253 e30a0000 00000000 01
ffff8a35f0776d80 666528071 S Bo:6:003:1 -115 31 = 55534243 e40a0000 12000000 80020603 00000012 00000000 00000000 000000
ffff8a35f0776d80 666528149 C Bo:6:003:1 0 31 >
ffff8a3527547840 666528159 S Bi:6:003:2 -115 18 <
ffff8a3527547840 666528709 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 666528731 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 666528854 C Bi:6:003:2 0 13 = 55534253 e40a0000 00000000 00
ffff8a35f0776d80 666528932 S Bo:6:003:1 -115 31 = 55534243 e50a0000 00000000 00010600 00000000 00000000 00000000 000000
ffff8a35f0776d80 666529007 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 666529016 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 666529386 C Bi:6:003:2 0 13 = 55534253 e50a0000 00000000 01
ffff8a35f0776d80 666529409 S Bo:6:003:1 -115 31 = 55534243 e60a0000 12000000 80010603 00000012 00000000 00000000 000000
ffff8a35f0776d80 666529470 C Bo:6:003:1 0 31 >
ffff8a3527547840 666529479 S Bi:6:003:2 -115 18 <
ffff8a3527547840 666529996 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 666530026 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 666530130 C Bi:6:003:2 0 13 = 55534253 e60a0000 00000000 00
ffff8a35f0776d80 666530178 S Bo:6:003:1 -115 31 = 55534243 e70a0000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35f0776d80 666530216 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 666530225 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 666530612 C Bi:6:003:2 0 13 = 55534253 e70a0000 00000000 01
ffff8a35f0776d80 666530624 S Bo:6:003:1 -115 31 = 55534243 e80a0000 12000000 80000603 00000012 00000000 00000000 000000
ffff8a35f0776d80 666530673 C Bo:6:003:1 0 31 >
ffff8a3527547840 666530682 S Bi:6:003:2 -115 18 <
ffff8a3527547840 666531203 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 666531225 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 666531351 C Bi:6:003:2 0 13 = 55534253 e80a0000 00000000 00
ffff8a35f0776d80 666591580 S Bo:6:003:1 -115 31 = 55534243 e90a0000 00000000 00030600 00000000 00000000 00000000 000000
ffff8a35f0776d80 666591682 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 666591698 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 666592048 C Bi:6:003:2 0 13 = 55534253 e90a0000 00000000 01
ffff8a35f0776d80 666592071 S Bo:6:003:1 -115 31 = 55534243 ea0a0000 12000000 80030603 00000012 00000000 00000000 000000
ffff8a35f0776d80 666592135 C Bo:6:003:1 0 31 >
ffff8a3527547840 666592144 S Bi:6:003:2 -115 18 <
ffff8a3527547840 666592725 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 666592763 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 666592791 C Bi:6:003:2 0 13 = 55534253 ea0a0000 00000000 00
ffff8a35f16de540 668181769 C Ii:6:002:1 0:2048 1 = 08
ffff8a35f16de540 668181799 S Ii:6:002:1 -115:2048 1 <
ffff8a3527547840 668181826 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff8a3527547840 668181855 C Ci:6:002:0 0 4 = 01010100
ffff8a3527547840 668181867 S Co:6:002:0 s 23 01 0010 0003 0000 0
ffff8a3527547840 668181907 C Co:6:002:0 0 0
ffff8a3527547840 668181919 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff8a3527547840 668181959 C Ci:6:002:0 0 4 = 01010000
ffff8a3527547840 668208532 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff8a3527547840 668208592 C Ci:6:002:0 0 4 = 01010000
ffff8a3527547840 668235529 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff8a3527547840 668235581 C Ci:6:002:0 0 4 = 01010000
ffff8a3527547840 668262547 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff8a3527547840 668262597 C Ci:6:002:0 0 4 = 01010000
ffff8a3527547840 668289493 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff8a3527547840 668289522 C Ci:6:002:0 0 4 = 01010000
ffff8a3527547840 668289728 S Co:6:002:0 s 23 03 0004 0003 0000 0
ffff8a3527547840 668289762 C Co:6:002:0 0 0
ffff8a3527547840 668301463 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff8a3527547840 668301498 C Ci:6:002:0 0 4 = 11050000
ffff8a3527547840 668313450 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff8a3527547840 668313495 C Ci:6:002:0 0 4 = 03051000
ffff8a3527547840 668313517 S Co:6:002:0 s 23 01 0014 0003 0000 0
ffff8a3527547840 668313532 C Co:6:002:0 0 0
ffff8a3527547840 668365596 S Ci:6:000:0 s 80 06 0100 0000 0040 64 <
ffff8a3527547840 668367909 C Ci:6:000:0 0 18 = 12011002 00000040 51096616 01000102 0301
ffff8a3527547840 668367955 S Co:6:002:0 s 23 03 0004 0003 0000 0
ffff8a3527547840 668367979 C Co:6:002:0 0 0
ffff8a3527547840 668379534 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff8a3527547840 668379594 C Ci:6:002:0 0 4 = 11050000
ffff8a3527547840 668391539 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff8a3527547840 668391589 C Ci:6:002:0 0 4 = 03051000
ffff8a3527547840 668391603 S Co:6:002:0 s 23 01 0014 0003 0000 0
ffff8a3527547840 668391621 C Co:6:002:0 0 0
ffff8a3527547840 668456517 S Ci:6:006:0 s 80 06 0100 0000 0012 18 <
ffff8a3527547840 668458848 C Ci:6:006:0 0 18 = 12011002 00000040 51096616 01000102 0301
ffff8a3527547840 668458879 S Ci:6:006:0 s 80 06 0f00 0000 0005 5 <
ffff8a3527547840 668459386 C Ci:6:006:0 0 5 = 050f1600 02
ffff8a3527547840 668459415 S Ci:6:006:0 s 80 06 0f00 0000 0016 22 <
ffff8a3527547840 668459940 C Ci:6:006:0 0 22 = 050f1600 02071002 06000000 0a100300 0e00020a ff07
ffff8a3527547840 668459973 S Ci:6:006:0 s 80 06 0200 0000 0009 9 <
ffff8a3527547840 668462415 C Ci:6:006:0 0 9 = 09022000 01010080 fa
ffff8a3527547840 668462456 S Ci:6:006:0 s 80 06 0200 0000 0020 32 <
ffff8a3527547840 668464904 C Ci:6:006:0 0 32 = 09022000 01010080 fa090400 00020806 50000705 81020002 ff070502 020002ff
ffff8a3527547000 668464942 S Ci:6:006:0 s 80 06 0300 0000 00ff 255 <
ffff8a3527547000 668467348 C Ci:6:006:0 0 4 = 04030904
ffff8a3527547000 668467376 S Ci:6:006:0 s 80 06 0302 0409 00ff 255 <
ffff8a3527547000 668469894 C Ci:6:006:0 0 34 = 22034400 61007400 61005400 72006100 76006500 6c006500 72002000 33002e00
ffff8a3527547000 668469931 S Ci:6:006:0 s 80 06 0301 0409 00ff 255 <
ffff8a3527547000 668472323 C Ci:6:006:0 0 18 = 12034b00 69006e00 67007300 74006f00 6e00
ffff8a3527547000 668472391 S Ci:6:006:0 s 80 06 0303 0409 00ff 255 <
ffff8a3527547000 668474814 C Ci:6:006:0 0 50 = 32033600 30004100 34003400 43003400 32003500 36003800 43004600 46003700
ffff8a3527547000 668476354 S Co:6:006:0 s 00 09 0001 0000 0000 0
ffff8a3527547000 668476684 C Co:6:006:0 0 0
ffff8a35f0776d80 668575544 S Bo:6:003:1 -115 31 = 55534243 eb0a0000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35f0776d80 668575605 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 668575617 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 668576022 C Bi:6:003:2 0 13 = 55534253 eb0a0000 00000000 01
ffff8a35f0776d80 668576035 S Bo:6:003:1 -115 31 = 55534243 ec0a0000 12000000 80000603 00000012 00000000 00000000 000000
ffff8a35f0776d80 668576082 C Bo:6:003:1 0 31 >
ffff8a3527547d80 668576090 S Bi:6:003:2 -115 18 <
ffff8a3527547d80 668576648 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 668576661 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 668576781 C Bi:6:003:2 0 13 = 55534253 ec0a0000 00000000 00
ffff8a35f0776d80 668576845 S Bo:6:003:1 -115 31 = 55534243 ed0a0000 00000000 00020600 00000000 00000000 00000000 000000
ffff8a35f0776d80 668576911 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 668576928 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 668577298 C Bi:6:003:2 0 13 = 55534253 ed0a0000 00000000 01
ffff8a35f0776d80 668577316 S Bo:6:003:1 -115 31 = 55534243 ee0a0000 12000000 80020603 00000012 00000000 00000000 000000
ffff8a35f0776d80 668577351 C Bo:6:003:1 0 31 >
ffff8a3527547d80 668577355 S Bi:6:003:2 -115 18 <
ffff8a3527547d80 668577909 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 668577927 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 668578043 C Bi:6:003:2 0 13 = 55534253 ee0a0000 00000000 00
ffff8a35f0776d80 668578111 S Bo:6:003:1 -115 31 = 55534243 ef0a0000 00000000 00010600 00000000 00000000 00000000 000000
ffff8a35f0776d80 668578150 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 668578154 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 668578539 C Bi:6:003:2 0 13 = 55534253 ef0a0000 00000000 01
ffff8a35f0776d80 668578557 S Bo:6:003:1 -115 31 = 55534243 f00a0000 12000000 80010603 00000012 00000000 00000000 000000
ffff8a35f0776d80 668578611 C Bo:6:003:1 0 31 >
ffff8a3527547d80 668578616 S Bi:6:003:2 -115 18 <
ffff8a3527547d80 668579161 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 668579179 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 668579274 C Bi:6:003:2 0 13 = 55534253 f00a0000 00000000 00
ffff8a35f0d2d840 668629883 C Ii:6:004:1 0:2 8 = 00000300 0c000000
ffff8a35f0d2d840 668629939 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668637887 C Ii:6:004:1 0:2 8 = 00000200 0d000000
ffff8a35f0d2d840 668637946 S Ii:6:004:1 -115:2 8 <
ffff8a35f0776d80 668639564 S Bo:6:003:1 -115 31 = 55534243 f10a0000 00000000 00030600 00000000 00000000 00000000 000000
ffff8a35f0776d80 668639639 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 668639658 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 668640060 C Bi:6:003:2 0 13 = 55534253 f10a0000 00000000 01
ffff8a35f0776d80 668640093 S Bo:6:003:1 -115 31 = 55534243 f20a0000 12000000 80030603 00000012 00000000 00000000 000000
ffff8a35f0776d80 668640174 C Bo:6:003:1 0 31 >
ffff8a3527547d80 668640206 S Bi:6:003:2 -115 18 <
ffff8a3527547d80 668640779 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 668640811 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 668640893 C Bi:6:003:2 0 13 = 55534253 f20a0000 00000000 00
ffff8a35f0d2d840 668645876 C Ii:6:004:1 0:2 8 = 00000300 0d000000
ffff8a35f0d2d840 668645929 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668653888 C Ii:6:004:1 0:2 8 = 00000100 0c000000
ffff8a35f0d2d840 668653952 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668661843 C Ii:6:004:1 0:2 8 = 00000300 0a000000
ffff8a35f0d2d840 668661905 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668669890 C Ii:6:004:1 0:2 8 = 00000300 09000000
ffff8a35f0d2d840 668669962 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668675840 C Ii:6:004:1 0:2 8 = 00000400 0b000000
ffff8a35f0d2d840 668675910 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668683895 C Ii:6:004:1 0:2 8 = 00000200 0e000000
ffff8a35f0d2d840 668683975 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668691897 C Ii:6:004:1 0:2 8 = 00000400 0e000000
ffff8a35f0d2d840 668691980 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668699884 C Ii:6:004:1 0:2 8 = 00000200 0e000000
ffff8a35f0d2d840 668699972 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668707869 C Ii:6:004:1 0:2 8 = 00000500 0f000000
ffff8a35f0d2d840 668707957 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668715831 C Ii:6:004:1 0:2 8 = 00000200 0f000000
ffff8a35f0d2d840 668715921 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668723887 C Ii:6:004:1 0:2 8 = 00000200 0e000000
ffff8a35f0d2d840 668723978 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668731823 C Ii:6:004:1 0:2 8 = 00000300 0b000000
ffff8a35f0d2d840 668731893 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668739888 C Ii:6:004:1 0:2 8 = 00000100 0b000000
ffff8a35f0d2d840 668739961 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668747889 C Ii:6:004:1 0:2 8 = 00000100 0a000000
ffff8a35f0d2d840 668747961 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668755888 C Ii:6:004:1 0:2 8 = 00000000 07000000
ffff8a35f0d2d840 668755961 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668761846 C Ii:6:004:1 0:2 8 = 00000000 07000000
ffff8a35f0d2d840 668761907 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668769883 C Ii:6:004:1 0:2 8 = 00000000 06000000
ffff8a35f0d2d840 668769969 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668777849 C Ii:6:004:1 0:2 8 = 00000000 05000000
ffff8a35f0d2d840 668777921 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668785887 C Ii:6:004:1 0:2 8 = 0000fdff 05000000
ffff8a35f0d2d840 668785967 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668793851 C Ii:6:004:1 0:2 8 = 0000feff 04000000
ffff8a35f0d2d840 668793927 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668801886 C Ii:6:004:1 0:2 8 = 0000ffff 05000000
ffff8a35f0d2d840 668801975 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668809886 C Ii:6:004:1 0:2 8 = 0000ffff 05000000
ffff8a35f0d2d840 668809976 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668817886 C Ii:6:004:1 0:2 8 = 0000ffff 06000000
ffff8a35f0d2d840 668817973 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668825885 C Ii:6:004:1 0:2 8 = 00000000 05000000
ffff8a35f0d2d840 668825973 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668833886 C Ii:6:004:1 0:2 8 = 0000ffff 04000000
ffff8a35f0d2d840 668833965 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668841881 C Ii:6:004:1 0:2 8 = 00000000 04000000
ffff8a35f0d2d840 668841959 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668847852 C Ii:6:004:1 0:2 8 = 00000000 03000000
ffff8a35f0d2d840 668847920 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668855877 C Ii:6:004:1 0:2 8 = 00000000 02000000
ffff8a35f0d2d840 668855928 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668863817 C Ii:6:004:1 0:2 8 = 00000200 01000000
ffff8a35f0d2d840 668863859 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668871883 C Ii:6:004:1 0:2 8 = 00000100 01000000
ffff8a35f0d2d840 668871944 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668879862 C Ii:6:004:1 0:2 8 = 00000100 00000000
ffff8a35f0d2d840 668879912 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668927892 C Ii:6:004:1 0:2 8 = 00000000 feff0000
ffff8a35f0d2d840 668927972 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668981871 C Ii:6:004:1 0:2 8 = 0000feff 00000000
ffff8a35f0d2d840 668981909 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668989873 C Ii:6:004:1 0:2 8 = 00000000 03000000
ffff8a35f0d2d840 668989918 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 668997850 C Ii:6:004:1 0:2 8 = 00000200 05000000
ffff8a35f0d2d840 668997888 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 669005872 C Ii:6:004:1 0:2 8 = 00000200 05000000
ffff8a35f0d2d840 669005925 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 669011876 C Ii:6:004:1 0:2 8 = 00000400 05000000
ffff8a35f0d2d840 669011926 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 669019873 C Ii:6:004:1 0:2 8 = 00000400 06000000
ffff8a35f0d2d840 669019924 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 669027883 C Ii:6:004:1 0:2 8 = 00000400 06000000
ffff8a35f0d2d840 669027947 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 669035882 C Ii:6:004:1 0:2 8 = 00000600 06000000
ffff8a35f0d2d840 669035942 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 669043882 C Ii:6:004:1 0:2 8 = 00000600 06000000
ffff8a35f0d2d840 669043940 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 669051878 C Ii:6:004:1 0:2 8 = 00000600 04000000
ffff8a35f0d2d840 669051943 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 669059812 C Ii:6:004:1 0:2 8 = 00000400 02000000
ffff8a35f0d2d840 669059872 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 669067882 C Ii:6:004:1 0:2 8 = 00000300 01000000
ffff8a35f0d2d840 669067949 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 669075882 C Ii:6:004:1 0:2 8 = 00000200 00000000
ffff8a35f0d2d840 669075954 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 669089841 C Ii:6:004:1 0:2 8 = 00000100 00000000
ffff8a35f0d2d840 669089926 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 669097861 C Ii:6:004:1 0:2 8 = 00000000 01000000
ffff8a35f0d2d840 669097922 S Ii:6:004:1 -115:2 8 <
ffff8a35275470c0 669535521 S Ci:6:006:0 s a1 fe 0000 0000 0001 1 <
ffff8a35275470c0 669535921 C Ci:6:006:0 0 1 = 00
ffff8a35275470c0 669536160 S Bo:6:006:2 -115 31 = 55534243 01000000 24000000 80000612 00000024 00000000 00000000 000000
ffff8a35275470c0 669536217 C Bo:6:006:2 0 31 >
ffff8a352b385d80 669536283 S Bi:6:006:1 -115 36 <
ffff8a352b385d80 669650003 C Bi:6:006:1 0 36 = 00800602 f8000000 4b696e67 73746f6e 44617461 54726176 656c6572 20332e30
ffff8a35275470c0 669650061 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669650220 C Bi:6:006:1 0 13 = 55534253 01000000 00000000 00
ffff8a35275470c0 669650863 S Bo:6:006:2 -115 31 = 55534243 02000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669650897 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669650920 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669650932 C Bi:6:006:1 0 13 = 55534253 02000000 00000000 00
ffff8a35275470c0 669651013 S Bo:6:006:2 -115 31 = 55534243 03000000 08000000 80000a25 00000000 00000000 00000000 000000
ffff8a35275470c0 669651037 C Bo:6:006:2 0 31 >
ffff8a35ff68a000 669651064 S Bi:6:006:1 -115 8 <
ffff8a35ff68a000 669651299 C Bi:6:006:1 0 8 = 039a33f3 00000200
ffff8a35275470c0 669651357 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669651528 C Bi:6:006:1 0 13 = 55534253 03000000 00000000 00
ffff8a35275470c0 669652397 S Bo:6:006:2 -115 31 = 55534243 04000000 c0000000 8000061a 003f00c0 00000000 00000000 000000
ffff8a35275470c0 669652446 C Bo:6:006:2 0 31 >
ffff8a3527547d80 669652481 S Bi:6:006:1 -115 192 <
ffff8a3527547d80 669652719 C Bi:6:006:1 -121 80 = 4f000000 051ef000 00000200 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669652739 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669652954 C Bi:6:006:1 0 13 = 55534253 04000000 70000000 00
ffff8a35275470c0 669653026 S Bo:6:006:2 -115 31 = 55534243 05000000 c0000000 8000061a 003f00c0 00000000 00000000 000000
ffff8a35275470c0 669653063 C Bo:6:006:2 0 31 >
ffff8a3527547d80 669653077 S Bi:6:006:1 -115 192 <
ffff8a3527547d80 669653331 C Bi:6:006:1 -121 80 = 4f000000 051ef000 00000200 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669653344 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669653559 C Bi:6:006:1 0 13 = 55534253 05000000 70000000 00
ffff8a35275470c0 669653926 S Bo:6:006:2 -115 31 = 55534243 06000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669653947 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669653968 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669653984 C Bi:6:006:1 0 13 = 55534253 06000000 00000000 00
ffff8a35275470c0 669654022 S Bo:6:006:2 -115 31 = 55534243 07000000 00000000 0000061e 00000001 00000000 00000000 000000
ffff8a35275470c0 669654037 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669654043 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669654527 C Bi:6:006:1 0 13 = 55534253 07000000 00000000 01
ffff8a35275470c0 669654566 S Bo:6:006:2 -115 31 = 55534243 08000000 60000000 80000603 00000060 00000000 00000000 000000
ffff8a35275470c0 669654594 C Bo:6:006:2 0 31 >
ffff8a35f006e240 669654612 S Bi:6:006:1 -115 96 <
ffff8a35f006e240 669654861 C Bi:6:006:1 0 96 = 70000500 0000000a 00000000 20000000 00000a7f 00000000 10000ae0 00400000
ffff8a35275470c0 669654871 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669655101 C Bi:6:006:1 0 13 = 55534253 08000000 00000000 00
ffff8a35275470c0 669655137 S Bo:6:006:2 -115 31 = 55534243 09000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669655150 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669655154 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669655190 C Bi:6:006:1 0 13 = 55534253 09000000 00000000 00
ffff8a35275470c0 669655211 S Bo:6:006:2 -115 31 = 55534243 0a000000 08000000 80000a25 00000000 00000000 00000000 000000
ffff8a35275470c0 669655230 C Bo:6:006:2 0 31 >
ffff8a3527547d80 669655236 S Bi:6:006:1 -115 8 <
ffff8a3527547d80 669655483 C Bi:6:006:1 0 8 = 039a33f3 00000200
ffff8a35275470c0 669655489 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669655713 C Bi:6:006:1 0 13 = 55534253 0a000000 00000000 00
ffff8a35275470c0 669655736 S Bo:6:006:2 -115 31 = 55534243 0b000000 c0000000 8000061a 003f00c0 00000000 00000000 000000
ffff8a35275470c0 669655753 C Bo:6:006:2 0 31 >
ffff8a3527547540 669655834 S Bi:6:006:1 -115 192 <
ffff8a3527547540 669656025 C Bi:6:006:1 -121 80 = 4f000000 051ef000 00000200 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669656031 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669656253 C Bi:6:006:1 0 13 = 55534253 0b000000 70000000 00
ffff8a35275470c0 669656275 S Bo:6:006:2 -115 31 = 55534243 0c000000 c0000000 8000061a 003f00c0 00000000 00000000 000000
ffff8a35275470c0 669656294 C Bo:6:006:2 0 31 >
ffff8a35ff68a840 669656365 S Bi:6:006:1 -115 192 <
ffff8a35ff68a840 669656569 C Bi:6:006:1 -121 80 = 4f000000 051ef000 00000200 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669656588 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669656794 C Bi:6:006:1 0 13 = 55534253 0c000000 70000000 00
ffff8a35275470c0 669656868 S Bo:6:006:2 -115 31 = 55534243 0d000000 00080000 80000a28 00000000 00000004 00000000 000000
ffff8a35275470c0 669656882 C Bo:6:006:2 0 31 >
ffff8a35ff68a840 669656885 S Bi:6:006:1 -115 2048 <
ffff8a35ff68a840 669658335 C Bi:6:006:1 0 2048 = fab80000 8ed0bc00 7c8bf450 07501ffb fcbf0006 b90001f3 a5ea1e06 0000bebe
ffff8a35275470c0 669658361 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669658374 C Bi:6:006:1 0 13 = 55534253 0d000000 00000000 00
ffff8a35275470c0 669658415 S Bo:6:006:2 -115 31 = 55534243 0e000000 00080000 80000a28 00000000 04000004 00000000 000000
ffff8a35275470c0 669658436 C Bo:6:006:2 0 31 >
ffff8a352b385840 669658453 S Bi:6:006:1 -115 2048 <
ffff8a352b385840 669658525 C Bi:6:006:1 0 2048 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669658530 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669658565 C Bi:6:006:1 0 13 = 55534253 0e000000 00000000 00
ffff8a35275470c0 669658895 S Bo:6:006:2 -115 31 = 55534243 0f000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669658910 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669658915 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669658950 C Bi:6:006:1 0 13 = 55534253 0f000000 00000000 00
ffff8a35275470c0 669658971 S Bo:6:006:2 -115 31 = 55534243 10000000 08000000 80000a25 00000000 00000000 00000000 000000
ffff8a35275470c0 669658990 C Bo:6:006:2 0 31 >
ffff8a352b385840 669658996 S Bi:6:006:1 -115 8 <
ffff8a352b385840 669659257 C Bi:6:006:1 0 8 = 039a33f3 00000200
ffff8a35275470c0 669659269 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669659504 C Bi:6:006:1 0 13 = 55534253 10000000 00000000 00
ffff8a35275470c0 669659539 S Bo:6:006:2 -115 31 = 55534243 11000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669659553 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669659560 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669659592 C Bi:6:006:1 0 13 = 55534253 11000000 00000000 00
ffff8a35275470c0 669659621 S Bo:6:006:2 -115 31 = 55534243 12000000 c0000000 8000061a 003f00c0 00000000 00000000 000000
ffff8a35275470c0 669659632 C Bo:6:006:2 0 31 >
ffff8a3527547780 669659637 S Bi:6:006:1 -115 192 <
ffff8a3527547780 669659903 C Bi:6:006:1 -121 80 = 4f000000 051ef000 00000200 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669659912 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669660151 C Bi:6:006:1 0 13 = 55534253 12000000 70000000 00
ffff8a35275470c0 669660177 S Bo:6:006:2 -115 31 = 55534243 13000000 c0000000 8000061a 003f00c0 00000000 00000000 000000
ffff8a35275470c0 669660191 C Bo:6:006:2 0 31 >
ffff8a3527547780 669660211 S Bi:6:006:1 -115 192 <
ffff8a3527547780 669660463 C Bi:6:006:1 -121 80 = 4f000000 051ef000 00000200 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669660483 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669660690 C Bi:6:006:1 0 13 = 55534253 13000000 70000000 00
ffff8a35275470c0 669660725 S Bo:6:006:2 -115 31 = 55534243 14000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669660739 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669660749 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669660793 C Bi:6:006:1 0 13 = 55534253 14000000 00000000 00
ffff8a35275470c0 669662979 S Bo:6:006:2 -115 31 = 55534243 15000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669662998 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669663004 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669663052 C Bi:6:006:1 0 13 = 55534253 15000000 00000000 00
ffff8a35275470c0 669663139 S Bo:6:006:2 -115 31 = 55534243 16000000 00100000 80000a28 00039a33 00000008 00000000 000000
ffff8a35275470c0 669663151 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669663156 S Bi:6:006:1 -115 4096 <
ffff8a35f006e6c0 669664709 C Bi:6:006:1 0 4096 = 4c776f62 69e41245 64d22d08 ae9db7b8 01c4e6fb 8b029833 8e4a9f4c 30a097df
ffff8a35275470c0 669664745 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669664794 C Bi:6:006:1 0 13 = 55534253 16000000 00000000 00
ffff8a35275470c0 669664854 S Bo:6:006:2 -115 31 = 55534243 17000000 00100000 80000a28 00039a33 e0000008 00000000 000000
ffff8a35275470c0 669664877 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669664891 S Bi:6:006:1 -115 4096 <
ffff8a35f006e6c0 669665536 C Bi:6:006:1 0 4096 = 85bfff3d 1c873fc8 99903146 ddceb7e5 10fe7570 715c5ff7 9343c2c9 c6ff195c
ffff8a35275470c0 669665582 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669665613 C Bi:6:006:1 0 13 = 55534253 17000000 00000000 00
ffff8a35275470c0 669665658 S Bo:6:006:2 -115 31 = 55534243 18000000 00100000 80000a28 00000000 00000008 00000000 000000
ffff8a35275470c0 669665687 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669665702 S Bi:6:006:1 -115 4096 <
ffff8a35f006e6c0 669667190 C Bi:6:006:1 0 4096 = fab80000 8ed0bc00 7c8bf450 07501ffb fcbf0006 b90001f3 a5ea1e06 0000bebe
ffff8a35275470c0 669667209 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669667218 C Bi:6:006:1 0 13 = 55534253 18000000 00000000 00
ffff8a35275470c0 669667264 S Bo:6:006:2 -115 31 = 55534243 19000000 00100000 80000a28 00000000 08000008 00000000 000000
ffff8a35275470c0 669667275 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669667278 S Bi:6:006:1 -115 4096 <
ffff8a35f006e6c0 669667963 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669668008 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669668061 C Bi:6:006:1 0 13 = 55534253 19000000 00000000 00
ffff8a35275470c0 669668113 S Bo:6:006:2 -115 31 = 55534243 1a000000 00020000 80000a28 00039a33 f0000001 00000000 000000
ffff8a35275470c0 669668135 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669668139 S Bi:6:006:1 -115 512 <
ffff8a35f006e6c0 669669591 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff8a35275470c0 669669643 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669669664 C Bi:6:006:1 0 13 = 55534253 1a000000 00000000 00
ffff8a35275470c0 669669750 S Bo:6:006:2 -115 31 = 55534243 1b000000 00020000 80000a28 00039a33 f1000001 00000000 000000
ffff8a35275470c0 669669778 C Bo:6:006:2 0 31 >
ffff8a352b385540 669669783 S Bi:6:006:1 -115 512 <
ffff8a352b385540 669670381 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff8a35275470c0 669670387 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669670423 C Bi:6:006:1 0 13 = 55534253 1b000000 00000000 00
ffff8a35275470c0 669670445 S Bo:6:006:2 -115 31 = 55534243 1c000000 00020000 80000a28 00039a33 f2000001 00000000 000000
ffff8a35275470c0 669670461 C Bo:6:006:2 0 31 >
ffff8a3527547540 669670467 S Bi:6:006:1 -115 512 <
ffff8a3527547540 669671080 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff8a35275470c0 669671096 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669671102 C Bi:6:006:1 0 13 = 55534253 1c000000 00000000 00
ffff8a35275470c0 669671125 S Bo:6:006:2 -115 31 = 55534243 1d000000 00020000 80000a28 00039a33 f3000001 00000000 000000
ffff8a35275470c0 669671143 C Bo:6:006:2 0 31 >
ffff8a352b385540 669671146 S Bi:6:006:1 -115 512 <
ffff8a352b385540 669671746 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff8a35275470c0 669671771 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669671816 C Bi:6:006:1 0 13 = 55534253 1d000000 00000000 00
ffff8a35275470c0 669671860 S Bo:6:006:2 -115 31 = 55534243 1e000000 00100000 80000a28 00039a32 f0000008 00000000 000000
ffff8a35275470c0 669671873 C Bo:6:006:2 0 31 >
ffff8a352b385540 669671889 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669672523 C Bi:6:006:1 0 4096 = 8add6e3c 2432a9cf 357c4940 a360e0ab a20f802f 1d321990 325a7363 73ab2a85
ffff8a35275470c0 669672548 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669672563 C Bi:6:006:1 0 13 = 55534253 1e000000 00000000 00
ffff8a35275470c0 669672614 S Bo:6:006:2 -115 31 = 55534243 1f000000 00100000 80000a28 00039a33 b0000008 00000000 000000
ffff8a35275470c0 669672627 C Bo:6:006:2 0 31 >
ffff8a352b385540 669672630 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669673301 C Bi:6:006:1 0 4096 = fd53643f 276dee14 bca4e177 d85ab32c 5a52be3a 75d14c46 8b06f53f 39e85942
ffff8a35275470c0 669673305 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669673341 C Bi:6:006:1 0 13 = 55534253 1f000000 00000000 00
ffff8a35275470c0 669673369 S Bo:6:006:2 -115 31 = 55534243 20000000 00100000 80000a28 00039a32 60000008 00000000 000000
ffff8a35275470c0 669673382 C Bo:6:006:2 0 31 >
ffff8a352b385540 669673384 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669674051 C Bi:6:006:1 0 4096 = 8841fcc0 f141f63c 74b09b13 582943df 1f84ee19 ca7625f1 1e71698a d8e9723f
ffff8a35275470c0 669674055 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669674091 C Bi:6:006:1 0 13 = 55534253 20000000 00000000 00
ffff8a35275470c0 669674109 S Bo:6:006:2 -115 31 = 55534243 21000000 00100000 80000a28 00039a31 a0000008 00000000 000000
ffff8a35275470c0 669674131 C Bo:6:006:2 0 31 >
ffff8a352b385540 669674134 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669674772 C Bi:6:006:1 0 4096 = e83e664b f50389eb 1b408ee4 65d0216d 44c3e132 eb58ebd2 3b8f99e3 f0cf5765
ffff8a35275470c0 669674776 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669674812 C Bi:6:006:1 0 13 = 55534253 21000000 00000000 00
ffff8a35275470c0 669674856 S Bo:6:006:2 -115 31 = 55534243 22000000 00100000 80000a28 00039a31 50000008 00000000 000000
ffff8a35275470c0 669674867 C Bo:6:006:2 0 31 >
ffff8a352b385540 669674883 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669675569 C Bi:6:006:1 0 4096 = f92826b9 0f9982fd 742b040f f35d44f5 d590c27e 669f6b9f c7c8491f c9a8cfa7
ffff8a35275470c0 669675599 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669675606 C Bi:6:006:1 0 13 = 55534253 22000000 00000000 00
ffff8a35275470c0 669675658 S Bo:6:006:2 -115 31 = 55534243 23000000 00100000 80000a28 00039a31 10000008 00000000 000000
ffff8a35275470c0 669675672 C Bo:6:006:2 0 31 >
ffff8a352b385540 669675679 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669676342 C Bi:6:006:1 0 4096 = 2dbbe381 8f0e583d 890d82b4 4d73de7e 7b0a3609 2658d90d ba959458 1aed842c
ffff8a35275470c0 669676347 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669676382 C Bi:6:006:1 0 13 = 55534253 23000000 00000000 00
ffff8a35275470c0 669676406 S Bo:6:006:2 -115 31 = 55534243 24000000 00100000 80000a28 00039a30 60000008 00000000 000000
ffff8a35275470c0 669676421 C Bo:6:006:2 0 31 >
ffff8a352b385540 669676425 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669677102 C Bi:6:006:1 0 4096 = 2ec65134 a721e3ca 2a1f8931 2f630dea 7a552159 fab8e3da 7d6c7ef9 2c814b72
ffff8a35275470c0 669677109 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669677132 C Bi:6:006:1 0 13 = 55534253 24000000 00000000 00
ffff8a35275470c0 669677159 S Bo:6:006:2 -115 31 = 55534243 25000000 00100000 80000a28 00039a30 20000008 00000000 000000
ffff8a35275470c0 669677172 C Bo:6:006:2 0 31 >
ffff8a352b385540 669677176 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669677855 C Bi:6:006:1 0 4096 = fa0cab42 80b853f2 b7ad415c d4c255af 9c7a1f34 b93c10a9 f257457b 2cc3df29
ffff8a35275470c0 669677861 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669677884 C Bi:6:006:1 0 13 = 55534253 25000000 00000000 00
ffff8a35275470c0 669677948 S Bo:6:006:2 -115 31 = 55534243 26000000 00100000 80000a28 00039a30 10000008 00000000 000000
ffff8a35275470c0 669677964 C Bo:6:006:2 0 31 >
ffff8a352b385540 669677975 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669678665 C Bi:6:006:1 0 4096 = 560e4b62 84950a4a a3c3ec6c 4ac0e404 a0220047 a97c1a1c 00758e94 58b20f2b
ffff8a35275470c0 669678702 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669678712 C Bi:6:006:1 0 13 = 55534253 26000000 00000000 00
ffff8a35275470c0 669678752 S Bo:6:006:2 -115 31 = 55534243 27000000 00100000 80000a28 00039a30 38000008 00000000 000000
ffff8a35275470c0 669678767 C Bo:6:006:2 0 31 >
ffff8a352b385540 669678779 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669679446 C Bi:6:006:1 0 4096 = c87d4091 ebf570d5 61fb8e44 21a5418f a14adebf f11a4627 b196f961 216317b1
ffff8a35275470c0 669679482 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669679491 C Bi:6:006:1 0 13 = 55534253 27000000 00000000 00
ffff8a35275470c0 669679546 S Bo:6:006:2 -115 31 = 55534243 28000000 00100000 80000a28 00039a27 e0000008 00000000 000000
ffff8a35275470c0 669679561 C Bo:6:006:2 0 31 >
ffff8a352b385540 669679565 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669680227 C Bi:6:006:1 0 4096 = 4d758507 4874e7ab 733c4eaf a1b9c050 e0c0867e 61b66631 c080b58e 18107d17
ffff8a35275470c0 669680233 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669680266 C Bi:6:006:1 0 13 = 55534253 28000000 00000000 00
ffff8a35275470c0 669680317 S Bo:6:006:2 -115 31 = 55534243 29000000 00080000 80000a28 00039a33 e8000004 00000000 000000
ffff8a35275470c0 669680329 C Bo:6:006:2 0 31 >
ffff8a352b385540 669680333 S Bi:6:006:1 -115 2048 <
ffff8a352b385540 669680973 C Bi:6:006:1 0 2048 = c71672c3 62f9e441 93556627 49f7d2f6 c1556d39 f35c478c b7303075 c5c622c8
ffff8a35275470c0 669680979 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669681036 C Bi:6:006:1 0 13 = 55534253 29000000 00000000 00
ffff8a35275470c0 669681080 S Bo:6:006:2 -115 31 = 55534243 2a000000 00020000 80000a28 00039a33 ec000001 00000000 000000
ffff8a35275470c0 669681095 C Bo:6:006:2 0 31 >
ffff8a3527547540 669681100 S Bi:6:006:1 -115 512 <
ffff8a3527547540 669681703 C Bi:6:006:1 0 512 = 809f4d0a bc5667de 57467a8e 646e8913 8d0bae35 d0f85a27 97065766 1ac31874
ffff8a35275470c0 669681721 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669681750 C Bi:6:006:1 0 13 = 55534253 2a000000 00000000 00
ffff8a35275470c0 669681794 S Bo:6:006:2 -115 31 = 55534243 2b000000 00020000 80000a28 00039a33 ed000001 00000000 000000
ffff8a35275470c0 669681854 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669681871 S Bi:6:006:1 -115 512 <
ffff8a35f006e6c0 669682435 C Bi:6:006:1 0 512 = 97bc843d bb40e8dd 37106a17 8b97f602 c439d3b5 e4dfd162 bd8e6a42 09f83484
ffff8a35275470c0 669682453 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669682465 C Bi:6:006:1 0 13 = 55534253 2b000000 00000000 00
ffff8a35275470c0 669682512 S Bo:6:006:2 -115 31 = 55534243 2c000000 00020000 80000a28 00039a33 ee000001 00000000 000000
ffff8a35275470c0 669682530 C Bo:6:006:2 0 31 >
ffff8a35ff68a9c0 669682555 S Bi:6:006:1 -115 512 <
ffff8a35ff68a9c0 669683141 C Bi:6:006:1 0 512 = 66f05e02 eb030290 ceab76c6 e426f4b0 6ed9b467 31e125f2 28e6c092 135d8134
ffff8a35275470c0 669683178 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669683191 C Bi:6:006:1 0 13 = 55534253 2c000000 00000000 00
ffff8a35275470c0 669683236 S Bo:6:006:2 -115 31 = 55534243 2d000000 00020000 80000a28 00039a33 ef000001 00000000 000000
ffff8a35275470c0 669683255 C Bo:6:006:2 0 31 >
ffff8a352b385540 669683270 S Bi:6:006:1 -115 512 <
ffff8a352b385540 669683868 C Bi:6:006:1 0 512 = 3a43294b 77481b39 4cec9d97 da766318 358af9ff e8c5dfce 6bbfff83 e78d3f1c
ffff8a35275470c0 669683905 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669683927 C Bi:6:006:1 0 13 = 55534253 2d000000 00000000 00
ffff8a35275470c0 669684047 S Bo:6:006:2 -115 31 = 55534243 2e000000 00100000 80000a28 00000000 20000008 00000000 000000
ffff8a35275470c0 669684068 C Bo:6:006:2 0 31 >
ffff8a352b385540 669684091 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669685576 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669685614 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669685636 C Bi:6:006:1 0 13 = 55534253 2e000000 00000000 00
ffff8a35275470c0 669685746 S Bo:6:006:2 -115 31 = 55534243 2f000000 00100000 80000a28 00000000 40000008 00000000 000000
ffff8a35275470c0 669685776 C Bo:6:006:2 0 31 >
ffff8a352b385540 669685781 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669686447 C Bi:6:006:1 0 4096 = eb589054 48524545 53202000 02204000 02000000 00f80000 40002000 40000000
ffff8a35275470c0 669686455 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669686486 C Bi:6:006:1 0 13 = 55534253 2f000000 00000000 00
ffff8a35275470c0 669686574 S Bo:6:006:2 -115 31 = 55534243 30000000 00100000 80000a28 00000000 80000008 00000000 000000
ffff8a35275470c0 669686591 C Bo:6:006:2 0 31 >
ffff8a352b385540 669686596 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669687252 C Bi:6:006:1 0 4096 = f8ffffff ffffff0f ffffff0f 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669687276 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669687294 C Bi:6:006:1 0 13 = 55534253 30000000 00000000 00
ffff8a35275470c0 669687364 S Bo:6:006:2 -115 31 = 55534243 31000000 00100000 80000a28 00000001 00000008 00000000 000000
ffff8a35275470c0 669687380 C Bo:6:006:2 0 31 >
ffff8a352b385540 669687396 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669688065 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669688089 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669688102 C Bi:6:006:1 0 13 = 55534253 31000000 00000000 00
ffff8a35275470c0 669688186 S Bo:6:006:2 -115 31 = 55534243 32000000 00100000 80000a28 00000002 00000008 00000000 000000
ffff8a35275470c0 669688203 C Bo:6:006:2 0 31 >
ffff8a352b385540 669688219 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669688910 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669688939 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669688954 C Bi:6:006:1 0 13 = 55534253 32000000 00000000 00
ffff8a35275470c0 669689061 S Bo:6:006:2 -115 31 = 55534243 33000000 00100000 80000a28 00000004 00000008 00000000 000000
ffff8a35275470c0 669689111 C Bo:6:006:2 0 31 >
ffff8a352b385540 669689149 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669689806 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669689835 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669689850 C Bi:6:006:1 0 13 = 55534253 33000000 00000000 00
ffff8a35275470c0 669689956 S Bo:6:006:2 -115 31 = 55534243 34000000 00100000 80000a28 00000008 00000008 00000000 000000
ffff8a35275470c0 669690005 C Bo:6:006:2 0 31 >
ffff8a352b385540 669690042 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669690719 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669690749 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669690763 C Bi:6:006:1 0 13 = 55534253 34000000 00000000 00
ffff8a35275470c0 669690852 S Bo:6:006:2 -115 31 = 55534243 35000000 00100000 80000a28 00000010 00000008 00000000 000000
ffff8a35275470c0 669690878 C Bo:6:006:2 0 31 >
ffff8a352b385540 669690906 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669691608 C Bi:6:006:1 0 4096 = 00810000 00810100 00810200 00810300 00810400 00810c00 00810d00 00811800
ffff8a35275470c0 669691642 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669691690 C Bi:6:006:1 0 13 = 55534253 35000000 00000000 00
ffff8a35275470c0 669691873 S Bo:6:006:2 -115 31 = 55534243 36000000 00100000 80000a28 00000020 00000008 00000000 000000
ffff8a35275470c0 669691917 C Bo:6:006:2 0 31 >
ffff8a352b385540 669691957 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669692630 C Bi:6:006:1 0 4096 = 00830000 00830100 00830200 00830300 00830400 00830c00 00830d00 00831800
ffff8a35275470c0 669692668 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669692678 C Bi:6:006:1 0 13 = 55534253 36000000 00000000 00
ffff8a35275470c0 669692854 S Bo:6:006:2 -115 31 = 55534243 37000000 00100000 80000a28 00000000 18000008 00000000 000000
ffff8a35275470c0 669692900 C Bo:6:006:2 0 31 >
ffff8a352b385540 669692940 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669693556 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669693589 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669693606 C Bi:6:006:1 0 13 = 55534253 37000000 00000000 00
ffff8a35275470c0 669693731 S Bo:6:006:2 -115 31 = 55534243 38000000 00100000 80000a28 00000000 38000008 00000000 000000
ffff8a35275470c0 669693759 C Bo:6:006:2 0 31 >
ffff8a352b385540 669693791 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669694459 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669694490 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669694506 C Bi:6:006:1 0 13 = 55534253 38000000 00000000 00
ffff8a35275470c0 669694654 S Bo:6:006:2 -115 31 = 55534243 39000000 00100000 80000a28 00000000 78000008 00000000 000000
ffff8a35275470c0 669694676 C Bo:6:006:2 0 31 >
ffff8a352b385540 669694683 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669695367 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669695377 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669695385 C Bi:6:006:1 0 13 = 55534253 39000000 00000000 00
ffff8a35275470c0 669695673 S Bo:6:006:2 -115 31 = 55534243 3a000000 00100000 80000a28 00000000 10000008 00000000 000000
ffff8a35275470c0 669695696 C Bo:6:006:2 0 31 >
ffff8a352b385540 669695718 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669696378 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669696421 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669696451 C Bi:6:006:1 0 13 = 55534253 3a000000 00000000 00
ffff8a35275470c0 669696526 S Bo:6:006:2 -115 31 = 55534243 3b000000 00200000 80000a28 00000000 28000010 00000000 000000
ffff8a35275470c0 669696558 C Bo:6:006:2 0 31 >
ffff8a3527547540 669696565 S Bi:6:006:1 -115 8192 <
ffff8a3527547540 669697335 C Bi:6:006:1 0 8192 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669697363 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669697375 C Bi:6:006:1 0 13 = 55534253 3b000000 00000000 00
ffff8a35275470c0 669697420 S Bo:6:006:2 -115 31 = 55534243 3c000000 00600000 80000a28 00000000 48000030 00000000 000000
ffff8a35275470c0 669697451 C Bo:6:006:2 0 31 >
ffff8a352b385540 669697471 S Bi:6:006:1 -115 24576 <
ffff8a352b385540 669698599 C Bi:6:006:1 0 24576 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669698664 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669698686 C Bi:6:006:1 0 13 = 55534253 3c000000 00000000 00
ffff8a35275470c0 669698744 S Bo:6:006:2 -115 31 = 55534243 3d000000 00f00000 80000a28 00000000 88000078 00000000 000000
ffff8a35275470c0 669698762 C Bo:6:006:2 0 31 >
ffff8a3527547780 669698768 S Bi:6:006:1 -115 61440 <
ffff8a3527547780 669700775 C Bi:6:006:1 0 61440 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669700795 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669700812 C Bi:6:006:1 0 13 = 55534253 3d000000 00000000 00
ffff8a35275470c0 669700879 S Bo:6:006:2 -115 31 = 55534243 3e000000 00100000 80000a28 00000001 08000008 00000000 000000
ffff8a35275470c0 669700954 C Bo:6:006:2 0 31 >
ffff8a352b385540 669700991 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669701595 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669701634 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669701651 C Bi:6:006:1 0 13 = 55534253 3e000000 00000000 00
ffff8a35275470c0 669701795 S Bo:6:006:2 -115 31 = 55534243 3f000000 00e00100 80000a28 00000001 100000f0 00000000 000000
ffff8a35275470c0 669701817 C Bo:6:006:2 0 31 >
ffff8a352b385540 669701837 S Bi:6:006:1 -115 122880 <
ffff8a352b385540 669705508 C Bi:6:006:1 0 122880 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669705561 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669705575 C Bi:6:006:1 0 13 = 55534253 3f000000 00000000 00
ffff8a35275470c0 669706061 S Bo:6:006:2 -115 31 = 55534243 40000000 00e00100 80000a28 00000002 080000f0 00000000 000000
ffff8a35275470c0 669706096 C Bo:6:006:2 0 31 >
ffff8a352b385540 669706136 S Bi:6:006:1 -115 122880 <
ffff8a352b385540 669709783 C Bi:6:006:1 0 122880 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669709838 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669709852 C Bi:6:006:1 0 13 = 55534253 40000000 00000000 00
ffff8a35275470c0 669709943 S Bo:6:006:2 -115 31 = 55534243 41000000 00200000 80000a28 00000002 f8000010 00000000 000000
ffff8a35275470c0 669709984 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669709997 S Bi:6:006:1 -115 8192 <
ffff8a35f006e6c0 669710850 C Bi:6:006:1 0 8192 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669710904 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669710920 C Bi:6:006:1 0 13 = 55534253 41000000 00000000 00
ffff8a35275470c0 669711178 S Bo:6:006:2 -115 31 = 55534243 42000000 00e00100 80000a28 00000003 080000f0 00000000 000000
ffff8a35275470c0 669711213 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669711254 S Bi:6:006:1 -115 122880 <
ffff8a35f006e6c0 669714882 C Bi:6:006:1 0 122880 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669714945 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669714961 C Bi:6:006:1 0 13 = 55534253 42000000 00000000 00
ffff8a35275470c0 669715041 S Bo:6:006:2 -115 31 = 55534243 43000000 00100000 80000a28 00000003 f8000008 00000000 000000
ffff8a35275470c0 669715069 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669715093 S Bi:6:006:1 -115 4096 <
ffff8a35f006e6c0 669715225 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669715249 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669715261 C Bi:6:006:1 0 13 = 55534253 43000000 00000000 00
ffff8a35275470c0 669715840 S Bo:6:006:2 -115 31 = 55534243 44000000 00e00100 80000a28 00039a2e 000000f0 00000000 000000
ffff8a35275470c0 669715916 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669715955 S Bi:6:006:1 -115 122880 <
ffff8a35f006e6c0 669720426 C Bi:6:006:1 0 122880 = c77ccf3d b18bbc20 49a7c771 ab78fd88 353bdfd1 c2f59b72 ee1c7082 2c62c6f6
ffff8a35275470c0 669720491 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669720507 C Bi:6:006:1 0 13 = 55534253 44000000 00000000 00
ffff8a35275470c0 669720588 S Bo:6:006:2 -115 31 = 55534243 45000000 00200000 80000a28 00039a2e f0000010 00000000 000000
ffff8a35275470c0 669720622 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669720644 S Bi:6:006:1 -115 8192 <
ffff8a35f006e6c0 669721405 C Bi:6:006:1 0 8192 = 2d1e8c4f 0d6d6356 384be8bc b5cfb50e cdb3025f bcfecddd 4f80d2e5 eb6b3284
ffff8a35275470c0 669721453 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669721465 C Bi:6:006:1 0 13 = 55534253 45000000 00000000 00
ffff8a35275470c0 669721714 S Bo:6:006:2 -115 31 = 55534243 46000000 00e00100 80000a28 00039a2f 000000f0 00000000 000000
ffff8a35275470c0 669721791 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669721833 S Bi:6:006:1 -115 122880 <
ffff8a35f006e6c0 669725505 C Bi:6:006:1 0 122880 = aa1c9eea 8ab1cada d7bd4902 c36a01ea 724ec451 e5509315 ca093a94 9dcab358
ffff8a35275470c0 669725562 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669725578 C Bi:6:006:1 0 13 = 55534253 46000000 00000000 00
ffff8a35275470c0 669725662 S Bo:6:006:2 -115 31 = 55534243 47000000 00200000 80000a28 00039a2f f0000010 00000000 000000
ffff8a35275470c0 669725689 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669725714 S Bi:6:006:1 -115 8192 <
ffff8a35f006e6c0 669725923 C Bi:6:006:1 0 8192 = 6ddcf87d 817edc15 a527992a 77e6c620 1adc6410 77ef845a d44f3327 5ac54da3
ffff8a35275470c0 669725964 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669725986 C Bi:6:006:1 0 13 = 55534253 47000000 00000000 00
ffff8a35275470c0 669726552 S Bo:6:006:2 -115 31 = 55534243 48000000 00200000 80000a28 00039a30 00000010 00000000 000000
ffff8a35275470c0 669726582 C Bo:6:006:2 0 31 >
ffff8a35ff68a9c0 669726595 S Bi:6:006:1 -115 8192 <
ffff8a35ff68a9c0 669726838 C Bi:6:006:1 0 8192 = c2bb8f02 e6227d01 1316d5d4 f8589196 3738834e 7c5c90cb 17868ef4 9eb65148
ffff8a35275470c0 669727341 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669727364 C Bi:6:006:1 0 13 = 55534253 48000000 00000000 00
ffff8a35275470c0 669727443 S Bo:6:006:2 -115 31 = 55534243 49000000 00100000 80000a28 00039a30 18000008 00000000 000000
ffff8a35275470c0 669727463 C Bo:6:006:2 0 31 >
ffff8a35ff68a9c0 669727472 S Bi:6:006:1 -115 4096 <
ffff8a35ff68a9c0 669728193 C Bi:6:006:1 0 4096 = 3fa743bc cc91e902 d72dfafc 5ceb71ec 3a4d1ec0 d33baba5 e7323bf0 e6eec907
ffff8a35275470c0 669728273 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669728306 C Bi:6:006:1 0 13 = 55534253 49000000 00000000 00
ffff8a35275470c0 669728377 S Bo:6:006:2 -115 31 = 55534243 4a000000 00200000 80000a28 00039a30 28000010 00000000 000000
ffff8a35275470c0 669728397 C Bo:6:006:2 0 31 >
ffff8a35ff68a9c0 669728428 S Bi:6:006:1 -115 8192 <
ffff8a35ff68a9c0 669729180 C Bi:6:006:1 0 8192 = 04529ef3 d43afe6d 30080475 edaff8de 8de2c099 9f5f242b 139da8b7 840879d0
ffff8a35275470c0 669729252 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669729279 C Bi:6:006:1 0 13 = 55534253 4a000000 00000000 00
ffff8a35275470c0 669729356 S Bo:6:006:2 -115 31 = 55534243 4b000000 00400000 80000a28 00039a30 40000020 00000000 000000
ffff8a35275470c0 669729379 C Bo:6:006:2 0 31 >
ffff8a35ff68a9c0 669729407 S Bi:6:006:1 -115 16384 <
ffff8a35ff68a9c0 669730351 C Bi:6:006:1 0 16384 = 877035b3 af82887f 9c7bab44 76656b91 64401358 4211b9e0 165863b9 e453e6f3
ffff8a35275470c0 669730401 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669730420 C Bi:6:006:1 0 13 = 55534253 4b000000 00000000 00
ffff8a35275470c0 669730486 S Bo:6:006:2 -115 31 = 55534243 4c000000 00300100 80000a28 00039a30 68000098 00000000 000000
ffff8a35275470c0 669730507 C Bo:6:006:2 0 31 >
ffff8a35ff68a9c0 669730538 S Bi:6:006:1 -115 77824 <
ffff8a35ff68a9c0 669733097 C Bi:6:006:1 0 77824 = 8a0e21ae 43b2cb2b 22eab472 19a923c0 5c814e49 7a105fc9 79938e83 79833c64
ffff8a35275470c0 669733163 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669733208 C Bi:6:006:1 0 13 = 55534253 4c000000 00000000 00
ffff8a35275470c0 669733449 S Bo:6:006:2 -115 31 = 55534243 4d000000 00200000 80000a28 00039a31 00000010 00000000 000000
ffff8a35275470c0 669733479 C Bo:6:006:2 0 31 >
ffff8a35ff68a9c0 669733516 S Bi:6:006:1 -115 8192 <
ffff8a35ff68a9c0 669733761 C Bi:6:006:1 0 8192 = 315bbba0 afae98ec 6e222b9d ba2bc3bd 1a3f0717 1688e8e5 c30b4584 6f4f58ae
ffff8a35275470c0 669733848 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669733892 C Bi:6:006:1 0 13 = 55534253 4d000000 00000000 00
ffff8a35275470c0 669733994 S Bo:6:006:2 -115 31 = 55534243 4e000000 00700000 80000a28 00039a31 18000038 00000000 000000
ffff8a35275470c0 669734019 C Bo:6:006:2 0 31 >
ffff8a35ff68a9c0 669734053 S Bi:6:006:1 -115 28672 <
ffff8a35ff68a9c0 669735465 C Bi:6:006:1 0 28672 = 3f13f01d 8a7e9065 563a44f7 0ef01f79 42c5d9cf 91d9e2cd 0f878026 67c1a426
ffff8a35275470c0 669735530 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669735575 C Bi:6:006:1 0 13 = 55534253 4e000000 00000000 00
ffff8a35275470c0 669735704 S Bo:6:006:2 -115 31 = 55534243 4f000000 00900000 80000a28 00039a31 58000048 00000000 000000
ffff8a35275470c0 669735732 C Bo:6:006:2 0 31 >
ffff8a35ff68a9c0 669735748 S Bi:6:006:1 -115 36864 <
ffff8a35ff68a9c0 669737337 C Bi:6:006:1 0 36864 = 063e5961 7fe48390 3d427f2a 2dfd5e3a b589c6d3 1c72b289 dc3b9458 a96cf1e0
ffff8a35275470c0 669737403 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669737451 C Bi:6:006:1 0 13 = 55534253 4f000000 00000000 00
ffff8a35275470c0 669737573 S Bo:6:006:2 -115 31 = 55534243 50000000 00b00000 80000a28 00039a31 a8000058 00000000 000000
ffff8a35275470c0 669737600 C Bo:6:006:2 0 31 >
ffff8a35ff68a9c0 669737638 S Bi:6:006:1 -115 45056 <
ffff8a35ff68a9c0 669739350 C Bi:6:006:1 0 45056 = 98fc9677 d852f513 9fbf7375 d7fdf2d4 5d650447 eb96f5b2 8292a863 52207607
ffff8a35275470c0 669739415 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669739460 C Bi:6:006:1 0 13 = 55534253 50000000 00000000 00
ffff8a35275470c0 669739848 S Bo:6:006:2 -115 31 = 55534243 51000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669739881 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669739914 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669739927 C Bi:6:006:1 0 13 = 55534253 51000000 00000000 00
ffff8a35275470c0 669745673 S Bo:6:006:2 -115 31 = 55534243 52000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669745698 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669745712 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669745734 C Bi:6:006:1 0 13 = 55534253 52000000 00000000 00
ffff8a35275470c0 669746771 S Bo:6:006:2 -115 31 = 55534243 53000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669746797 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669746810 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669746841 C Bi:6:006:1 0 13 = 55534253 53000000 00000000 00
ffff8a35275470c0 669747468 S Bo:6:006:2 -115 31 = 55534243 54000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669747490 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669747496 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669747566 C Bi:6:006:1 0 13 = 55534253 54000000 00000000 00
ffff8a35275470c0 669749880 S Bo:6:006:2 -115 31 = 55534243 55000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669749903 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669749911 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669749940 C Bi:6:006:1 0 13 = 55534253 55000000 00000000 00
ffff8a35275470c0 669750696 S Bo:6:006:2 -115 31 = 55534243 56000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669750716 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669750736 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669750759 C Bi:6:006:1 0 13 = 55534253 56000000 00000000 00
ffff8a35275470c0 669752370 S Bo:6:006:2 -115 31 = 55534243 57000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669752397 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669752402 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669752423 C Bi:6:006:1 0 13 = 55534253 57000000 00000000 00
ffff8a35275470c0 669752573 S Bo:6:006:2 -115 31 = 55534243 58000000 00100000 80000a28 00039a33 00000008 00000000 000000
ffff8a35275470c0 669752585 C Bo:6:006:2 0 31 >
ffff8a3527547780 669752590 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669753270 C Bi:6:006:1 0 4096 = 4c776f62 69e41245 64d22d08 ae9db7b8 01c4e6fb 8b029833 8e4a9f4c 30a097df
ffff8a35275470c0 669753274 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669753310 C Bi:6:006:1 0 13 = 55534253 58000000 00000000 00
ffff8a35275470c0 669753331 S Bo:6:006:2 -115 31 = 55534243 59000000 00100000 80000a28 00039a33 e0000008 00000000 000000
ffff8a35275470c0 669753351 C Bo:6:006:2 0 31 >
ffff8a3527547780 669753355 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669754010 C Bi:6:006:1 0 4096 = 85bfff3d 1c873fc8 99903146 ddceb7e5 10fe7570 715c5ff7 9343c2c9 c6ff195c
ffff8a35275470c0 669754016 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669754050 C Bi:6:006:1 0 13 = 55534253 59000000 00000000 00
ffff8a35275470c0 669754084 S Bo:6:006:2 -115 31 = 55534243 5a000000 00100000 80000a28 00000008 00000008 00000000 000000
ffff8a35275470c0 669754096 C Bo:6:006:2 0 31 >
ffff8a3527547780 669754100 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669755620 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669755625 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669755660 C Bi:6:006:1 0 13 = 55534253 5a000000 00000000 00
ffff8a35275470c0 669755737 S Bo:6:006:2 -115 31 = 55534243 5b000000 00100000 80000a28 00000008 08000008 00000000 000000
ffff8a35275470c0 669755751 C Bo:6:006:2 0 31 >
ffff8a3527547780 669755756 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669756450 C Bi:6:006:1 0 4096 = 01040000 11040000 21040000 e85bdf1f 02000400 00000000 00000000 dc1f8466
ffff8a35275470c0 669756465 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669756522 C Bi:6:006:1 0 13 = 55534253 5b000000 00000000 00
ffff8a35275470c0 669756586 S Bo:6:006:2 -115 31 = 55534243 5c000000 00020000 80000a28 00039a33 f0000001 00000000 000000
ffff8a35275470c0 669756643 C Bo:6:006:2 0 31 >
ffff8a3527547780 669756647 S Bi:6:006:1 -115 512 <
ffff8a3527547780 669758281 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff8a35275470c0 669758285 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669758321 C Bi:6:006:1 0 13 = 55534253 5c000000 00000000 00
ffff8a35275470c0 669758348 S Bo:6:006:2 -115 31 = 55534243 5d000000 00020000 80000a28 00039a33 f1000001 00000000 000000
ffff8a35275470c0 669758362 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669758366 S Bi:6:006:1 -115 512 <
ffff8a35f006e6c0 669758966 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff8a35275470c0 669758985 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669759010 C Bi:6:006:1 0 13 = 55534253 5d000000 00000000 00
ffff8a35275470c0 669759036 S Bo:6:006:2 -115 31 = 55534243 5e000000 00020000 80000a28 00039a33 f2000001 00000000 000000
ffff8a35275470c0 669759049 C Bo:6:006:2 0 31 >
ffff8a35ff68a9c0 669759051 S Bi:6:006:1 -115 512 <
ffff8a35ff68a9c0 669759638 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff8a35275470c0 669759673 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669759702 C Bi:6:006:1 0 13 = 55534253 5e000000 00000000 00
ffff8a35275470c0 669759792 S Bo:6:006:2 -115 31 = 55534243 5f000000 00020000 80000a28 00039a33 f3000001 00000000 000000
ffff8a35275470c0 669759806 C Bo:6:006:2 0 31 >
ffff8a3527547780 669759809 S Bi:6:006:1 -115 512 <
ffff8a3527547780 669760387 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff8a35275470c0 669760392 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669760427 C Bi:6:006:1 0 13 = 55534253 5f000000 00000000 00
ffff8a35275470c0 669760480 S Bo:6:006:2 -115 31 = 55534243 60000000 00100000 80000a28 00039a32 f0000008 00000000 000000
ffff8a35275470c0 669760497 C Bo:6:006:2 0 31 >
ffff8a3527547780 669760503 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669761207 C Bi:6:006:1 0 4096 = 8add6e3c 2432a9cf 357c4940 a360e0ab a20f802f 1d321990 325a7363 73ab2a85
ffff8a35275470c0 669761226 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669761238 C Bi:6:006:1 0 13 = 55534253 60000000 00000000 00
ffff8a35275470c0 669761331 S Bo:6:006:2 -115 31 = 55534243 61000000 00080000 80000a28 00039a33 e8000004 00000000 000000
ffff8a35275470c0 669761344 C Bo:6:006:2 0 31 >
ffff8a3527547780 669761348 S Bi:6:006:1 -115 2048 <
ffff8a3527547780 669761990 C Bi:6:006:1 0 2048 = c71672c3 62f9e441 93556627 49f7d2f6 c1556d39 f35c478c b7303075 c5c622c8
ffff8a35275470c0 669761998 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669762024 C Bi:6:006:1 0 13 = 55534253 61000000 00000000 00
ffff8a35275470c0 669762066 S Bo:6:006:2 -115 31 = 55534243 62000000 00020000 80000a28 00039a33 ec000001 00000000 000000
ffff8a35275470c0 669762107 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669762155 S Bi:6:006:1 -115 512 <
ffff8a35f006e6c0 669762698 C Bi:6:006:1 0 512 = 809f4d0a bc5667de 57467a8e 646e8913 8d0bae35 d0f85a27 97065766 1ac31874
ffff8a35275470c0 669762759 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669762777 C Bi:6:006:1 0 13 = 55534253 62000000 00000000 00
ffff8a35275470c0 669762858 S Bo:6:006:2 -115 31 = 55534243 63000000 00020000 80000a28 00039a33 ed000001 00000000 000000
ffff8a35275470c0 669762900 C Bo:6:006:2 0 31 >
ffff8a35ff68a9c0 669762955 S Bi:6:006:1 -115 512 <
ffff8a35ff68a9c0 669763471 C Bi:6:006:1 0 512 = 97bc843d bb40e8dd 37106a17 8b97f602 c439d3b5 e4dfd162 bd8e6a42 09f83484
ffff8a35275470c0 669763497 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669763520 C Bi:6:006:1 0 13 = 55534253 63000000 00000000 00
ffff8a35275470c0 669763552 S Bo:6:006:2 -115 31 = 55534243 64000000 00020000 80000a28 00039a33 ee000001 00000000 000000
ffff8a35275470c0 669763578 C Bo:6:006:2 0 31 >
ffff8a352b385540 669763581 S Bi:6:006:1 -115 512 <
ffff8a352b385540 669764178 C Bi:6:006:1 0 512 = 66f05e02 eb030290 ceab76c6 e426f4b0 6ed9b467 31e125f2 28e6c092 135d8134
ffff8a35275470c0 669764182 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669764224 C Bi:6:006:1 0 13 = 55534253 64000000 00000000 00
ffff8a35275470c0 669764241 S Bo:6:006:2 -115 31 = 55534243 65000000 00020000 80000a28 00039a33 ef000001 00000000 000000
ffff8a35275470c0 669764254 C Bo:6:006:2 0 31 >
ffff8a3527547780 669764258 S Bi:6:006:1 -115 512 <
ffff8a3527547780 669764845 C Bi:6:006:1 0 512 = 3a43294b 77481b39 4cec9d97 da766318 358af9ff e8c5dfce 6bbfff83 e78d3f1c
ffff8a35275470c0 669764848 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669764895 C Bi:6:006:1 0 13 = 55534253 65000000 00000000 00
ffff8a35275470c0 669764961 S Bo:6:006:2 -115 31 = 55534243 66000000 00100000 80000a28 00000008 20000008 00000000 000000
ffff8a35275470c0 669764980 C Bo:6:006:2 0 31 >
ffff8a3527547780 669764984 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669766502 C Bi:6:006:1 0 4096 = 04800000 04800100 04800200 04800300 04800400 04800c00 04800d00 04801800
ffff8a35275470c0 669766518 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669766570 C Bi:6:006:1 0 13 = 55534253 66000000 00000000 00
ffff8a35275470c0 669766623 S Bo:6:006:2 -115 31 = 55534243 67000000 00100000 80000a28 00000008 40000008 00000000 000000
ffff8a35275470c0 669766635 C Bo:6:006:2 0 31 >
ffff8a3527547780 669766638 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669767321 C Bi:6:006:1 0 4096 = 08800000 08800100 08800200 08800300 08800400 08800c00 08800d00 08801800
ffff8a35275470c0 669767336 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669767359 C Bi:6:006:1 0 13 = 55534253 67000000 00000000 00
ffff8a35275470c0 669767407 S Bo:6:006:2 -115 31 = 55534243 68000000 00100000 80000a28 00000008 80000008 00000000 000000
ffff8a35275470c0 669767421 C Bo:6:006:2 0 31 >
ffff8a3527547780 669767424 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669768098 C Bi:6:006:1 0 4096 = 10800000 10800100 10800200 10800300 10800400 10800c00 10800d00 10801800
ffff8a35275470c0 669768102 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669768138 C Bi:6:006:1 0 13 = 55534253 68000000 00000000 00
ffff8a35275470c0 669768185 S Bo:6:006:2 -115 31 = 55534243 69000000 00100000 80000a28 00000009 00000008 00000000 000000
ffff8a35275470c0 669768194 C Bo:6:006:2 0 31 >
ffff8a3527547780 669768197 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669768874 C Bi:6:006:1 0 4096 = 20800000 20800100 20800200 20800300 20800400 20800c00 20800d00 20801800
ffff8a35275470c0 669768878 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669768935 C Bi:6:006:1 0 13 = 55534253 69000000 00000000 00
ffff8a35275470c0 669769024 S Bo:6:006:2 -115 31 = 55534243 6a000000 00100000 80000a28 0000000a 00000008 00000000 000000
ffff8a35275470c0 669769061 C Bo:6:006:2 0 31 >
ffff8a3527547780 669769076 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669769807 C Bi:6:006:1 0 4096 = 40800000 40800100 40800200 40800300 40800400 40800c00 40800d00 40801800
ffff8a35275470c0 669769827 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669769840 C Bi:6:006:1 0 13 = 55534253 6a000000 00000000 00
ffff8a35275470c0 669769955 S Bo:6:006:2 -115 31 = 55534243 6b000000 00100000 80000a28 0000000c 00000008 00000000 000000
ffff8a35275470c0 669769993 C Bo:6:006:2 0 31 >
ffff8a3527547780 669770008 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669770707 C Bi:6:006:1 0 4096 = 80800000 80800100 80800200 80800300 80800400 80800c00 80800d00 80801800
ffff8a35275470c0 669770728 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669770740 C Bi:6:006:1 0 13 = 55534253 6b000000 00000000 00
ffff8a35275470c0 669770855 S Bo:6:006:2 -115 31 = 55534243 6c000000 00100000 80000a28 00000010 00000008 00000000 000000
ffff8a35275470c0 669770891 C Bo:6:006:2 0 31 >
ffff8a3527547780 669770906 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669771633 C Bi:6:006:1 0 4096 = 00810000 00810100 00810200 00810300 00810400 00810c00 00810d00 00811800
ffff8a35275470c0 669771654 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669771667 C Bi:6:006:1 0 13 = 55534253 6c000000 00000000 00
ffff8a35275470c0 669771779 S Bo:6:006:2 -115 31 = 55534243 6d000000 00100000 80000a28 00000018 00000008 00000000 000000
ffff8a35275470c0 669771816 C Bo:6:006:2 0 31 >
ffff8a3527547780 669771832 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669772518 C Bi:6:006:1 0 4096 = 00820000 00820100 00820200 00820300 00820400 00820c00 00820d00 00821800
ffff8a35275470c0 669772528 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669772544 C Bi:6:006:1 0 13 = 55534253 6d000000 00000000 00
ffff8a35275470c0 669772625 S Bo:6:006:2 -115 31 = 55534243 6e000000 00100000 80000a28 00000028 00000008 00000000 000000
ffff8a35275470c0 669772639 C Bo:6:006:2 0 31 >
ffff8a3527547780 669772643 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669773338 C Bi:6:006:1 0 4096 = 00840000 00840100 00840200 00840300 00840400 00840c00 00840d00 00841800
ffff8a35275470c0 669773343 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669773378 C Bi:6:006:1 0 13 = 55534253 6e000000 00000000 00
ffff8a35275470c0 669773465 S Bo:6:006:2 -115 31 = 55534243 6f000000 00100000 80000a28 00000008 18000008 00000000 000000
ffff8a35275470c0 669773479 C Bo:6:006:2 0 31 >
ffff8a3527547780 669773483 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669774174 C Bi:6:006:1 0 4096 = 03800000 03800100 03800200 03800300 03800400 03800c00 03800d00 03801800
ffff8a35275470c0 669774178 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669774235 C Bi:6:006:1 0 13 = 55534253 6f000000 00000000 00
ffff8a35275470c0 669774316 S Bo:6:006:2 -115 31 = 55534243 70000000 00100000 80000a28 00000008 38000008 00000000 000000
ffff8a35275470c0 669774349 C Bo:6:006:2 0 31 >
ffff8a3527547780 669774364 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669775051 C Bi:6:006:1 0 4096 = 07800000 07800100 07800200 07800300 07800400 07800c00 07800d00 07801800
ffff8a35275470c0 669775060 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669775068 C Bi:6:006:1 0 13 = 55534253 70000000 00000000 00
ffff8a35275470c0 669775145 S Bo:6:006:2 -115 31 = 55534243 71000000 00100000 80000a28 00000008 78000008 00000000 000000
ffff8a35275470c0 669775178 C Bo:6:006:2 0 31 >
ffff8a3527547780 669775197 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669775903 C Bi:6:006:1 0 4096 = 0f800000 0f800100 0f800200 0f800300 0f800400 0f800c00 0f800d00 0f801800
ffff8a35275470c0 669775927 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669775939 C Bi:6:006:1 0 13 = 55534253 71000000 00000000 00
ffff8a35275470c0 669776134 S Bo:6:006:2 -115 31 = 55534243 72000000 00100000 80000a28 00000008 10000008 00000000 000000
ffff8a35275470c0 669776149 C Bo:6:006:2 0 31 >
ffff8a3527547780 669776153 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669776878 C Bi:6:006:1 0 4096 = 00004000 10004000 20004000 f05ff01f 00000500 00000000 00000000 f01f7270
ffff8a35275470c0 669776886 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669776893 C Bi:6:006:1 0 13 = 55534253 72000000 00000000 00
ffff8a35275470c0 669776920 S Bo:6:006:2 -115 31 = 55534243 73000000 00200000 80000a28 00000008 28000010 00000000 000000
ffff8a35275470c0 669776958 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669776977 S Bi:6:006:1 -115 8192 <
ffff8a35f006e6c0 669777743 C Bi:6:006:1 0 8192 = 05800000 05800100 05800200 05800300 05800400 05800c00 05800d00 05801800
ffff8a35275470c0 669777781 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669777813 C Bi:6:006:1 0 13 = 55534253 73000000 00000000 00
ffff8a35275470c0 669777915 S Bo:6:006:2 -115 31 = 55534243 74000000 00600000 80000a28 00000008 48000030 00000000 000000
ffff8a35275470c0 669777953 C Bo:6:006:2 0 31 >
ffff8a3527547780 669777968 S Bi:6:006:1 -115 24576 <
ffff8a3527547780 669779053 C Bi:6:006:1 0 24576 = 09800000 09800100 09800200 09800300 09800400 09800c00 09800d00 09801800
ffff8a35275470c0 669779061 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669779087 C Bi:6:006:1 0 13 = 55534253 74000000 00000000 00
ffff8a35275470c0 669779134 S Bo:6:006:2 -115 31 = 55534243 75000000 00f00000 80000a28 00000008 88000078 00000000 000000
ffff8a35275470c0 669779157 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669779176 S Bi:6:006:1 -115 61440 <
ffff8a35f006e6c0 669781177 C Bi:6:006:1 0 61440 = 11800000 11800100 11800200 11800300 11800400 11800c00 11800d00 11801800
ffff8a35275470c0 669781230 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669781254 C Bi:6:006:1 0 13 = 55534253 75000000 00000000 00
ffff8a35275470c0 669781355 S Bo:6:006:2 -115 31 = 55534243 76000000 00100000 80000a28 00000009 08000008 00000000 000000
ffff8a35275470c0 669781373 C Bo:6:006:2 0 31 >
ffff8a3527547780 669781379 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669782085 C Bi:6:006:1 0 4096 = 21800000 21800100 21800200 21800300 21800400 21800c00 21800d00 21801800
ffff8a35275470c0 669782102 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669782129 C Bi:6:006:1 0 13 = 55534253 76000000 00000000 00
ffff8a35275470c0 669782246 S Bo:6:006:2 -115 31 = 55534243 77000000 00e00100 80000a28 00000009 100000f0 00000000 000000
ffff8a35275470c0 669782299 C Bo:6:006:2 0 31 >
ffff8a3527547780 669782317 S Bi:6:006:1 -115 122880 <
ffff8a3527547780 669785963 C Bi:6:006:1 0 122880 = 22800000 22800100 22800200 22800300 22800400 22800c00 22800d00 22801800
ffff8a35275470c0 669785994 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669786012 C Bi:6:006:1 0 13 = 55534253 77000000 00000000 00
ffff8a35275470c0 669786372 S Bo:6:006:2 -115 31 = 55534243 78000000 00e00100 80000a28 0000000a 080000f0 00000000 000000
ffff8a35275470c0 669786388 C Bo:6:006:2 0 31 >
ffff8a3527547780 669786396 S Bi:6:006:1 -115 122880 <
ffff8a3527547780 669790028 C Bi:6:006:1 0 122880 = 41800000 41800100 41800200 41800300 41800400 41800c00 41800d00 41801800
ffff8a35275470c0 669790043 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669790064 C Bi:6:006:1 0 13 = 55534253 78000000 00000000 00
ffff8a35275470c0 669790124 S Bo:6:006:2 -115 31 = 55534243 79000000 00200000 80000a28 0000000a f8000010 00000000 000000
ffff8a35275470c0 669790142 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669790159 S Bi:6:006:1 -115 8192 <
ffff8a35f006e6c0 669790945 C Bi:6:006:1 0 8192 = 5f800000 5f800100 5f800200 5f800300 5f800400 5f800c00 5f800d00 5f801800
ffff8a35275470c0 669790979 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669790988 C Bi:6:006:1 0 13 = 55534253 79000000 00000000 00
ffff8a35275470c0 669791136 S Bo:6:006:2 -115 31 = 55534243 7a000000 00e00100 80000a28 0000000b 080000f0 00000000 000000
ffff8a35275470c0 669791153 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669791169 S Bi:6:006:1 -115 122880 <
ffff8a35f006e6c0 669794771 C Bi:6:006:1 0 122880 = 61800000 61800100 61800200 61800300 61800400 61800c00 61800d00 61801800
ffff8a35275470c0 669794839 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669794859 C Bi:6:006:1 0 13 = 55534253 7a000000 00000000 00
ffff8a35275470c0 669794940 S Bo:6:006:2 -115 31 = 55534243 7b000000 00100000 80000a28 0000000b f8000008 00000000 000000
ffff8a35275470c0 669794963 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669794970 S Bi:6:006:1 -115 4096 <
ffff8a35f006e6c0 669795089 C Bi:6:006:1 0 4096 = 7f800000 7f800100 7f800200 7f800300 7f800400 7f800c00 7f800d00 7f801800
ffff8a35275470c0 669795095 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669795129 C Bi:6:006:1 0 13 = 55534253 7b000000 00000000 00
ffff8a35275470c0 669795424 S Bo:6:006:2 -115 31 = 55534243 7c000000 00e00100 80000a28 00039a2e 000000f0 00000000 000000
ffff8a35275470c0 669795439 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669795444 S Bi:6:006:1 -115 122880 <
ffff8a35f006e6c0 669800092 C Bi:6:006:1 0 122880 = c77ccf3d b18bbc20 49a7c771 ab78fd88 353bdfd1 c2f59b72 ee1c7082 2c62c6f6
ffff8a35275470c0 669800152 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669800177 C Bi:6:006:1 0 13 = 55534253 7c000000 00000000 00
ffff8a35275470c0 669800218 S Bo:6:006:2 -115 31 = 55534243 7d000000 00200000 80000a28 00039a2e f0000010 00000000 000000
ffff8a35275470c0 669800234 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669800245 S Bi:6:006:1 -115 8192 <
ffff8a35f006e6c0 669801003 C Bi:6:006:1 0 8192 = 2d1e8c4f 0d6d6356 384be8bc b5cfb50e cdb3025f bcfecddd 4f80d2e5 eb6b3284
ffff8a35275470c0 669801070 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669801090 C Bi:6:006:1 0 13 = 55534253 7d000000 00000000 00
ffff8a35275470c0 669801241 S Bo:6:006:2 -115 31 = 55534243 7e000000 00e00100 80000a28 00039a2f 000000f0 00000000 000000
ffff8a35275470c0 669801271 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669801275 S Bi:6:006:1 -115 122880 <
ffff8a35f006e6c0 669804990 C Bi:6:006:1 0 122880 = aa1c9eea 8ab1cada d7bd4902 c36a01ea 724ec451 e5509315 ca093a94 9dcab358
ffff8a35275470c0 669805033 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669805056 C Bi:6:006:1 0 13 = 55534253 7e000000 00000000 00
ffff8a35275470c0 669805117 S Bo:6:006:2 -115 31 = 55534243 7f000000 00200000 80000a28 00039a2f f0000010 00000000 000000
ffff8a35275470c0 669805151 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669805161 S Bi:6:006:1 -115 8192 <
ffff8a35f006e6c0 669805350 C Bi:6:006:1 0 8192 = 6ddcf87d 817edc15 a527992a 77e6c620 1adc6410 77ef845a d44f3327 5ac54da3
ffff8a35275470c0 669805357 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669805379 C Bi:6:006:1 0 13 = 55534253 7f000000 00000000 00
ffff8a35275470c0 669805730 S Bo:6:006:2 -115 31 = 55534243 80000000 00e00100 80000a28 00039a30 000000f0 00000000 000000
ffff8a35275470c0 669805768 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669805796 S Bi:6:006:1 -115 122880 <
ffff8a35f006e6c0 669808955 C Bi:6:006:1 0 122880 = c2bb8f02 e6227d01 1316d5d4 f8589196 3738834e 7c5c90cb 17868ef4 9eb65148
ffff8a35275470c0 669809001 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669809023 C Bi:6:006:1 0 13 = 55534253 80000000 00000000 00
ffff8a35275470c0 669809085 S Bo:6:006:2 -115 31 = 55534243 81000000 00200000 80000a28 00039a30 f0000010 00000000 000000
ffff8a35275470c0 669809116 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669809131 S Bi:6:006:1 -115 8192 <
ffff8a35f006e6c0 669809340 C Bi:6:006:1 0 8192 = 2378ad20 bc2a7d26 58ef9fc8 48429609 72a99082 f13ed042 ba93cad1 de25670f
ffff8a35275470c0 669809346 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669809369 C Bi:6:006:1 0 13 = 55534253 81000000 00000000 00
ffff8a35275470c0 669809542 S Bo:6:006:2 -115 31 = 55534243 82000000 00e00100 80000a28 00039a31 000000f0 00000000 000000
ffff8a35275470c0 669809588 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669809614 S Bi:6:006:1 -115 122880 <
ffff8a35f006e6c0 669812737 C Bi:6:006:1 0 122880 = 315bbba0 afae98ec 6e222b9d ba2bc3bd 1a3f0717 1688e8e5 c30b4584 6f4f58ae
ffff8a35275470c0 669812767 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669812813 C Bi:6:006:1 0 13 = 55534253 82000000 00000000 00
ffff8a35275470c0 669812869 S Bo:6:006:2 -115 31 = 55534243 83000000 00200000 80000a28 00039a31 f0000010 00000000 000000
ffff8a35275470c0 669812899 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669812922 S Bi:6:006:1 -115 8192 <
ffff8a35f006e6c0 669813127 C Bi:6:006:1 0 8192 = 15c5f9a5 4fdcf7b5 d4013ce6 82011093 87fb73d9 5b8553c4 c7965bfb e4ae9642
ffff8a35275470c0 669813178 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669813212 C Bi:6:006:1 0 13 = 55534253 83000000 00000000 00
ffff8a35275470c0 669813569 S Bo:6:006:2 -115 31 = 55534243 84000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669813590 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669813613 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669813628 C Bi:6:006:1 0 13 = 55534253 84000000 00000000 00
ffff8a35275470c0 669813802 S Bo:6:006:2 -115 31 = 55534243 85000000 00100000 80000a28 00000000 00000008 00000000 000000
ffff8a35275470c0 669813866 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669813899 S Bi:6:006:1 -115 4096 <
ffff8a35f006e6c0 669815357 C Bi:6:006:1 0 4096 = fab80000 8ed0bc00 7c8bf450 07501ffb fcbf0006 b90001f3 a5ea1e06 0000bebe
ffff8a35275470c0 669815423 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669815463 C Bi:6:006:1 0 13 = 55534253 85000000 00000000 00
ffff8a35275470c0 669815828 S Bo:6:006:2 -115 31 = 55534243 86000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669815870 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669815906 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669815947 C Bi:6:006:1 0 13 = 55534253 86000000 00000000 00
ffff8a35275470c0 669816054 S Bo:6:006:2 -115 31 = 55534243 87000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669816096 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669816116 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669816162 C Bi:6:006:1 0 13 = 55534253 87000000 00000000 00
ffff8a35275470c0 669821120 S Bo:6:006:2 -115 31 = 55534243 88000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669821151 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669821373 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669821386 C Bi:6:006:1 0 13 = 55534253 88000000 00000000 00
ffff8a35275470c0 669826786 S Bo:6:006:2 -115 31 = 55534243 89000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669826807 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669826829 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669826844 C Bi:6:006:1 0 13 = 55534253 89000000 00000000 00
ffff8a35275470c0 669826945 S Bo:6:006:2 -115 31 = 55534243 8a000000 00400000 80000a28 00000008 00000020 00000000 000000
ffff8a35275470c0 669826959 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669826966 S Bi:6:006:1 -115 16384 <
ffff8a35f006e6c0 669827954 C Bi:6:006:1 0 16384 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669827961 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669827984 C Bi:6:006:1 0 13 = 55534253 8a000000 00000000 00
ffff8a35275470c0 669828044 S Bo:6:006:2 -115 31 = 55534243 8b000000 00800000 80000a28 00000008 20000040 00000000 000000
ffff8a35275470c0 669828059 C Bo:6:006:2 0 31 >
ffff8a35f006e6c0 669828071 S Bi:6:006:1 -115 32768 <
ffff8a35f006e6c0 669829340 C Bi:6:006:1 0 32768 = 04800000 04800100 04800200 04800300 04800400 04800c00 04800d00 04801800
ffff8a35275470c0 669829352 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669829378 C Bi:6:006:1 0 13 = 55534253 8b000000 00000000 00
ffff8a35275470c0 669829402 S Bo:6:006:2 -115 31 = 55534243 8c000000 00100000 80000a28 00000029 08000008 00000000 000000
ffff8a35275470c0 669829417 C Bo:6:006:2 0 31 >
ffff8a3527547780 669829423 S Bi:6:006:1 -115 4096 <
ffff8a3527547780 669830498 C Bi:6:006:1 0 4096 = 00000000 00000000 33f72257 33f72257 33f72257 00000000 00000000 00000000
ffff8a35275470c0 669830505 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669830537 C Bi:6:006:1 0 13 = 55534253 8c000000 00000000 00
ffff8a35275470c0 669830594 S Bo:6:006:2 -115 31 = 55534243 8d000000 00100000 80000a28 0001c408 00000008 00000000 000000
ffff8a35275470c0 669830612 C Bo:6:006:2 0 31 >
ffff8a35f006e300 669830619 S Bi:6:006:1 -115 4096 <
ffff8a35f006e300 669833756 C Bi:6:006:1 0 4096 = c03b3998 00000004 00000000 00001000 00008000 00000001 0003101a 00000000
ffff8a35275470c0 669833779 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669833791 C Bi:6:006:1 0 13 = 55534253 8d000000 00000000 00
ffff8a35275470c0 669834049 S Bo:6:006:2 -115 31 = 55534243 8e000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669834064 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669834073 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669834101 C Bi:6:006:1 0 13 = 55534253 8e000000 00000000 00
ffff8a35275470c0 669835867 S Bo:6:006:2 -115 31 = 55534243 8f000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669835892 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669835925 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669835933 C Bi:6:006:1 0 13 = 55534253 8f000000 00000000 00
ffff8a35275470c0 669835997 S Bo:6:006:2 -115 31 = 55534243 90000000 00400000 80000a28 00000008 00000020 00000000 000000
ffff8a35275470c0 669836007 C Bo:6:006:2 0 31 >
ffff8a352b385540 669836040 S Bi:6:006:1 -115 16384 <
ffff8a352b385540 669837826 C Bi:6:006:1 0 16384 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 669837855 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669837865 C Bi:6:006:1 0 13 = 55534253 90000000 00000000 00
ffff8a35275470c0 669837923 S Bo:6:006:2 -115 31 = 55534243 91000000 00800000 80000a28 00000008 20000040 00000000 000000
ffff8a35275470c0 669837936 C Bo:6:006:2 0 31 >
ffff8a352b385540 669837952 S Bi:6:006:1 -115 32768 <
ffff8a352b385540 669839227 C Bi:6:006:1 0 32768 = 04800000 04800100 04800200 04800300 04800400 04800c00 04800d00 04801800
ffff8a35275470c0 669839247 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669839254 C Bi:6:006:1 0 13 = 55534253 91000000 00000000 00
ffff8a35275470c0 669839267 S Bo:6:006:2 -115 31 = 55534243 92000000 00100000 80000a28 00000029 08000008 00000000 000000
ffff8a35275470c0 669839294 C Bo:6:006:2 0 31 >
ffff8a352b385540 669839298 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669840373 C Bi:6:006:1 0 4096 = 00000000 00000000 33f72257 33f72257 33f72257 00000000 00000000 00000000
ffff8a35275470c0 669840408 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669840415 C Bi:6:006:1 0 13 = 55534253 92000000 00000000 00
ffff8a35275470c0 669840485 S Bo:6:006:2 -115 31 = 55534243 93000000 00100000 80000a28 0001c408 00000008 00000000 000000
ffff8a35275470c0 669840520 C Bo:6:006:2 0 31 >
ffff8a352b385540 669840548 S Bi:6:006:1 -115 4096 <
ffff8a352b385540 669843657 C Bi:6:006:1 0 4096 = c03b3998 00000004 00000000 00001000 00008000 00000001 0003101a 00000000
ffff8a35275470c0 669843718 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669843735 C Bi:6:006:1 0 13 = 55534253 93000000 00000000 00
ffff8a35275470c0 669843824 S Bo:6:006:2 -115 31 = 55534243 94000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 669843835 C Bo:6:006:2 0 31 >
ffff8a35275470c0 669843856 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 669843887 C Bi:6:006:1 0 13 = 55534253 94000000 00000000 00
ffff8a35f0d2d840 670449838 C Ii:6:004:1 0:2 8 = 01000000 00000000
ffff8a35f0d2d840 670449879 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 670511808 C Ii:6:004:1 0:2 8 = 00000000 00000000
ffff8a35f0d2d840 670511890 S Ii:6:004:1 -115:2 8 <
ffff8a35f0776d80 670623502 S Bo:6:003:1 -115 31 = 55534243 f30a0000 00000000 00010600 00000000 00000000 00000000 000000
ffff8a35f0776d80 670623548 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 670623556 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 670623931 C Bi:6:003:2 0 13 = 55534253 f30a0000 00000000 01
ffff8a35f0776d80 670623938 S Bo:6:003:1 -115 31 = 55534243 f40a0000 12000000 80010603 00000012 00000000 00000000 000000
ffff8a35f0776d80 670623983 C Bo:6:003:1 0 31 >
ffff8a352b385840 670623988 S Bi:6:003:2 -115 18 <
ffff8a352b385840 670624502 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 670624508 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 670624675 C Bi:6:003:2 0 13 = 55534253 f40a0000 00000000 00
ffff8a35f0776d80 670624708 S Bo:6:003:1 -115 31 = 55534243 f50a0000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35f0776d80 670624748 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 670624752 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 670625146 C Bi:6:003:2 0 13 = 55534253 f50a0000 00000000 01
ffff8a35f0776d80 670625175 S Bo:6:003:1 -115 31 = 55534243 f60a0000 12000000 80000603 00000012 00000000 00000000 000000
ffff8a35f0776d80 670625213 C Bo:6:003:1 0 31 >
ffff8a352b385840 670625219 S Bi:6:003:2 -115 18 <
ffff8a352b385840 670625744 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 670625760 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 670625911 C Bi:6:003:2 0 13 = 55534253 f60a0000 00000000 00
ffff8a35f0776d80 670625961 S Bo:6:003:1 -115 31 = 55534243 f70a0000 00000000 00020600 00000000 00000000 00000000 000000
ffff8a35f0776d80 670626016 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 670626020 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 670626376 C Bi:6:003:2 0 13 = 55534253 f70a0000 00000000 01
ffff8a35f0776d80 670626393 S Bo:6:003:1 -115 31 = 55534243 f80a0000 12000000 80020603 00000012 00000000 00000000 000000
ffff8a35f0776d80 670626428 C Bo:6:003:1 0 31 >
ffff8a35f5f68c00 670626439 S Bi:6:003:2 -115 18 <
ffff8a35f5f68c00 670626952 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 670626982 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 670627105 C Bi:6:003:2 0 13 = 55534253 f80a0000 00000000 00
ffff8a35f0776d80 670687501 S Bo:6:003:1 -115 31 = 55534243 f90a0000 00000000 00030600 00000000 00000000 00000000 000000
ffff8a35f0776d80 670687549 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 670687571 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 670687962 C Bi:6:003:2 0 13 = 55534253 f90a0000 00000000 01
ffff8a35f0776d80 670687995 S Bo:6:003:1 -115 31 = 55534243 fa0a0000 12000000 80030603 00000012 00000000 00000000 000000
ffff8a35f0776d80 670688057 C Bo:6:003:1 0 31 >
ffff8a35ff68a3c0 670688101 S Bi:6:003:2 -115 18 <
ffff8a35ff68a3c0 670688603 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 670688635 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 670688753 C Bi:6:003:2 0 13 = 55534253 fa0a0000 00000000 00
ffff8a35275470c0 671903568 S Bo:6:006:2 -115 31 = 55534243 95000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 671903607 C Bo:6:006:2 0 31 >
ffff8a35275470c0 671903633 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 671903651 C Bi:6:006:1 0 13 = 55534253 95000000 00000000 00
ffff8a35f0776d80 672671618 S Bo:6:003:1 -115 31 = 55534243 fb0a0000 00000000 00020600 00000000 00000000 00000000 000000
ffff8a35f0776d80 672671714 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 672671766 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 672672106 C Bi:6:003:2 0 13 = 55534253 fb0a0000 00000000 01
ffff8a35f0776d80 672672164 S Bo:6:003:1 -115 31 = 55534243 fc0a0000 12000000 80020603 00000012 00000000 00000000 000000
ffff8a35f0776d80 672672238 C Bo:6:003:1 0 31 >
ffff8a35ff68a3c0 672672295 S Bi:6:003:2 -115 18 <
ffff8a35ff68a3c0 672672800 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 672672870 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 672672933 C Bi:6:003:2 0 13 = 55534253 fc0a0000 00000000 00
ffff8a35f0776d80 672673142 S Bo:6:003:1 -115 31 = 55534243 fd0a0000 00000000 00010600 00000000 00000000 00000000 000000
ffff8a35f0776d80 672673228 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 672673288 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 672673699 C Bi:6:003:2 0 13 = 55534253 fd0a0000 00000000 01
ffff8a35f0776d80 672673767 S Bo:6:003:1 -115 31 = 55534243 fe0a0000 12000000 80010603 00000012 00000000 00000000 000000
ffff8a35f0776d80 672673844 C Bo:6:003:1 0 31 >
ffff8a35ff68a3c0 672673901 S Bi:6:003:2 -115 18 <
ffff8a35ff68a3c0 672674416 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 672674487 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 672674550 C Bi:6:003:2 0 13 = 55534253 fe0a0000 00000000 00
ffff8a35f0776d80 672674765 S Bo:6:003:1 -115 31 = 55534243 ff0a0000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35f0776d80 672674853 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 672674883 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 672675224 C Bi:6:003:2 0 13 = 55534253 ff0a0000 00000000 01
ffff8a35f0776d80 672675243 S Bo:6:003:1 -115 31 = 55534243 000b0000 12000000 80000603 00000012 00000000 00000000 000000
ffff8a35f0776d80 672675322 C Bo:6:003:1 0 31 >
ffff8a35ff68a3c0 672675375 S Bi:6:003:2 -115 18 <
ffff8a35ff68a3c0 672675875 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 672675946 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 672675992 C Bi:6:003:2 0 13 = 55534253 000b0000 00000000 00
ffff8a35f0776d80 672735615 S Bo:6:003:1 -115 31 = 55534243 010b0000 00000000 00030600 00000000 00000000 00000000 000000
ffff8a35f0776d80 672735712 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 672735762 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 672736122 C Bi:6:003:2 0 13 = 55534253 010b0000 00000000 01
ffff8a35f0776d80 672736191 S Bo:6:003:1 -115 31 = 55534243 020b0000 12000000 80030603 00000012 00000000 00000000 000000
ffff8a35f0776d80 672736268 C Bo:6:003:1 0 31 >
ffff8a35ff68a3c0 672736326 S Bi:6:003:2 -115 18 <
ffff8a35ff68a3c0 672736858 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 672736926 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 672736988 C Bi:6:003:2 0 13 = 55534253 020b0000 00000000 00
ffff8a35f0d2d840 673169796 C Ii:6:004:1 0:2 8 = 0000feff 00000000
ffff8a35f0d2d840 673169889 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 673175794 C Ii:6:004:1 0:2 8 = 0000feff fcff0000
ffff8a35f0d2d840 673175883 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 673183797 C Ii:6:004:1 0:2 8 = 0000fdff feff0000
ffff8a35f0d2d840 673183881 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 673191755 C Ii:6:004:1 0:2 8 = 0000fbff fdff0000
ffff8a35f0d2d840 673191820 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 673199801 C Ii:6:004:1 0:2 8 = 0000fbff fdff0000
ffff8a35f0d2d840 673199875 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 673207760 C Ii:6:004:1 0:2 8 = 0000faff fdff0000
ffff8a35f0d2d840 673207812 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 673215802 C Ii:6:004:1 0:2 8 = 0000fbff fdff0000
ffff8a35f0d2d840 673215882 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 673223801 C Ii:6:004:1 0:2 8 = 0000fbff feff0000
ffff8a35f0d2d840 673223882 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 673231742 C Ii:6:004:1 0:2 8 = 0000feff ffff0000
ffff8a35f0d2d840 673231794 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 673239792 C Ii:6:004:1 0:2 8 = 0000feff ffff0000
ffff8a35f0d2d840 673239878 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 673247792 C Ii:6:004:1 0:2 8 = 0000ffff ffff0000
ffff8a35f0d2d840 673247878 S Ii:6:004:1 -115:2 8 <
ffff8a35275470c0 673951611 S Bo:6:006:2 -115 31 = 55534243 96000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 673951679 C Bo:6:006:2 0 31 >
ffff8a35275470c0 673951731 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 673951779 C Bi:6:006:1 0 13 = 55534253 96000000 00000000 00
ffff8a35f0776d80 674719586 S Bo:6:003:1 -115 31 = 55534243 030b0000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35f0776d80 674719682 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 674719732 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 674720063 C Bi:6:003:2 0 13 = 55534253 030b0000 00000000 01
ffff8a35f0776d80 674720117 S Bo:6:003:1 -115 31 = 55534243 040b0000 12000000 80000603 00000012 00000000 00000000 000000
ffff8a35f0776d80 674720172 C Bo:6:003:1 0 31 >
ffff8a35ff68ae40 674720200 S Bi:6:003:2 -115 18 <
ffff8a35ff68ae40 674720720 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 674720773 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 674720841 C Bi:6:003:2 0 13 = 55534253 040b0000 00000000 00
ffff8a35f0776d80 674720966 S Bo:6:003:1 -115 31 = 55534243 050b0000 00000000 00020600 00000000 00000000 00000000 000000
ffff8a35f0776d80 674721016 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 674721027 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 674721404 C Bi:6:003:2 0 13 = 55534253 050b0000 00000000 01
ffff8a35f0776d80 674721417 S Bo:6:003:1 -115 31 = 55534243 060b0000 12000000 80020603 00000012 00000000 00000000 000000
ffff8a35f0776d80 674721454 C Bo:6:003:1 0 31 >
ffff8a35ff68ae40 674721482 S Bi:6:003:2 -115 18 <
ffff8a35ff68ae40 674722000 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 674722053 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 674722127 C Bi:6:003:2 0 13 = 55534253 060b0000 00000000 00
ffff8a35f0776d80 674722309 S Bo:6:003:1 -115 31 = 55534243 070b0000 00000000 00010600 00000000 00000000 00000000 000000
ffff8a35f0776d80 674722363 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 674722411 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 674722766 C Bi:6:003:2 0 13 = 55534253 070b0000 00000000 01
ffff8a35f0776d80 674722819 S Bo:6:003:1 -115 31 = 55534243 080b0000 12000000 80010603 00000012 00000000 00000000 000000
ffff8a35f0776d80 674722901 C Bo:6:003:1 0 31 >
ffff8a35ff68ae40 674722950 S Bi:6:003:2 -115 18 <
ffff8a35ff68ae40 674723480 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 674723551 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 674723617 C Bi:6:003:2 0 13 = 55534253 080b0000 00000000 00
ffff8a35f0776d80 674783612 S Bo:6:003:1 -115 31 = 55534243 090b0000 00000000 00030600 00000000 00000000 00000000 000000
ffff8a35f0776d80 674783706 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 674783756 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 674784114 C Bi:6:003:2 0 13 = 55534253 090b0000 00000000 01
ffff8a35f0776d80 674784183 S Bo:6:003:1 -115 31 = 55534243 0a0b0000 12000000 80030603 00000012 00000000 00000000 000000
ffff8a35f0776d80 674784259 C Bo:6:003:1 0 31 >
ffff8a35ff68ae40 674784317 S Bi:6:003:2 -115 18 <
ffff8a35ff68ae40 674784843 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 674784913 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 674784975 C Bi:6:003:2 0 13 = 55534253 0a0b0000 00000000 00
ffff8a35275470c0 675999619 S Bo:6:006:2 -115 31 = 55534243 97000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 675999685 C Bo:6:006:2 0 31 >
ffff8a35275470c0 675999710 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 675999778 C Bi:6:006:1 0 13 = 55534253 97000000 00000000 00
ffff8a35f0776d80 676767598 S Bo:6:003:1 -115 31 = 55534243 0b0b0000 00000000 00010600 00000000 00000000 00000000 000000
ffff8a35f0776d80 676767694 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 676767745 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 676768066 C Bi:6:003:2 0 13 = 55534253 0b0b0000 00000000 01
ffff8a35f0776d80 676768125 S Bo:6:003:1 -115 31 = 55534243 0c0b0000 12000000 80010603 00000012 00000000 00000000 000000
ffff8a35f0776d80 676768180 C Bo:6:003:1 0 31 >
ffff8a35ff68ad80 676768231 S Bi:6:003:2 -115 18 <
ffff8a35ff68ad80 676768734 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 676768791 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 676768860 C Bi:6:003:2 0 13 = 55534253 0c0b0000 00000000 00
ffff8a35f0776d80 676768983 S Bo:6:003:1 -115 31 = 55534243 0d0b0000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35f0776d80 676769048 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 676769063 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 676769430 C Bi:6:003:2 0 13 = 55534253 0d0b0000 00000000 01
ffff8a35f0776d80 676769459 S Bo:6:003:1 -115 31 = 55534243 0e0b0000 12000000 80000603 00000012 00000000 00000000 000000
ffff8a35f0776d80 676769511 C Bo:6:003:1 0 31 >
ffff8a35f5f68240 676769527 S Bi:6:003:2 -115 18 <
ffff8a35f5f68240 676770051 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 676770065 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 676770205 C Bi:6:003:2 0 13 = 55534253 0e0b0000 00000000 00
ffff8a35f0776d80 676770292 S Bo:6:003:1 -115 31 = 55534243 0f0b0000 00000000 00020600 00000000 00000000 00000000 000000
ffff8a35f0776d80 676770344 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 676770364 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 676770761 C Bi:6:003:2 0 13 = 55534253 0f0b0000 00000000 01
ffff8a35f0776d80 676770813 S Bo:6:003:1 -115 31 = 55534243 100b0000 12000000 80020603 00000012 00000000 00000000 000000
ffff8a35f0776d80 676770887 C Bo:6:003:1 0 31 >
ffff8a35ff68ad80 676770919 S Bi:6:003:2 -115 18 <
ffff8a35ff68ad80 676771432 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 676771459 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 676771575 C Bi:6:003:2 0 13 = 55534253 100b0000 00000000 00
ffff8a35f0776d80 676831587 S Bo:6:003:1 -115 31 = 55534243 110b0000 00000000 00030600 00000000 00000000 00000000 000000
ffff8a35f0776d80 676831696 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 676831738 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 676832116 C Bi:6:003:2 0 13 = 55534253 110b0000 00000000 01
ffff8a35f0776d80 676832175 S Bo:6:003:1 -115 31 = 55534243 120b0000 12000000 80030603 00000012 00000000 00000000 000000
ffff8a35f0776d80 676832248 C Bo:6:003:1 0 31 >
ffff8a35ff68ad80 676832280 S Bi:6:003:2 -115 18 <
ffff8a35ff68ad80 676832799 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 676832834 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 676832914 C Bi:6:003:2 0 13 = 55534253 120b0000 00000000 00
ffff8a35275470c0 677996555 S Bo:6:006:2 -115 31 = 55534243 98000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 677996600 C Bo:6:006:2 0 31 >
ffff8a35275470c0 677996605 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 677996690 C Bi:6:006:1 0 13 = 55534253 98000000 00000000 00
ffff8a35275470c0 677996871 S Bo:6:006:2 -115 31 = 55534243 99000000 00100000 80000a28 00039a33 00000008 00000000 000000
ffff8a35275470c0 677996894 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 677996898 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 677998467 C Bi:6:006:1 0 4096 = 4c776f62 69e41245 64d22d08 ae9db7b8 01c4e6fb 8b029833 8e4a9f4c 30a097df
ffff8a35275470c0 677998487 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 677998500 C Bi:6:006:1 0 13 = 55534253 99000000 00000000 00
ffff8a35275470c0 677998537 S Bo:6:006:2 -115 31 = 55534243 9a000000 00100000 80000a28 00039a33 e0000008 00000000 000000
ffff8a35275470c0 677998548 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 677998551 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 677999211 C Bi:6:006:1 0 4096 = 85bfff3d 1c873fc8 99903146 ddceb7e5 10fe7570 715c5ff7 9343c2c9 c6ff195c
ffff8a35275470c0 677999218 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 677999250 C Bi:6:006:1 0 13 = 55534253 9a000000 00000000 00
ffff8a35275470c0 677999280 S Bo:6:006:2 -115 31 = 55534243 9b000000 00100000 80000a28 00000008 00000008 00000000 000000
ffff8a35275470c0 677999291 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 677999296 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678000828 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 678000836 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678000877 C Bi:6:006:1 0 13 = 55534253 9b000000 00000000 00
ffff8a35275470c0 678000936 S Bo:6:006:2 -115 31 = 55534243 9c000000 00100000 80000a28 00000008 08000008 00000000 000000
ffff8a35275470c0 678000953 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678000957 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678001688 C Bi:6:006:1 0 4096 = 01040000 11040000 21040000 e85bdf1f 02000400 00000000 00000000 dc1f8466
ffff8a35275470c0 678001699 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678001710 C Bi:6:006:1 0 13 = 55534253 9c000000 00000000 00
ffff8a35275470c0 678001764 S Bo:6:006:2 -115 31 = 55534243 9d000000 00020000 80000a28 00039a33 f0000001 00000000 000000
ffff8a35275470c0 678001781 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678001786 S Bi:6:006:1 -115 512 <
ffff8a35f5f680c0 678003507 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff8a35275470c0 678003518 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678003529 C Bi:6:006:1 0 13 = 55534253 9d000000 00000000 00
ffff8a35275470c0 678003578 S Bo:6:006:2 -115 31 = 55534243 9e000000 00020000 80000a28 00039a33 f1000001 00000000 000000
ffff8a35275470c0 678003620 C Bo:6:006:2 0 31 >
ffff8a35ff68a000 678003651 S Bi:6:006:1 -115 512 <
ffff8a35ff68a000 678004241 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff8a35275470c0 678004263 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678004286 C Bi:6:006:1 0 13 = 55534253 9e000000 00000000 00
ffff8a35275470c0 678004325 S Bo:6:006:2 -115 31 = 55534243 9f000000 00020000 80000a28 00039a33 f2000001 00000000 000000
ffff8a35275470c0 678004356 C Bo:6:006:2 0 31 >
ffff8a352b385780 678004361 S Bi:6:006:1 -115 512 <
ffff8a352b385780 678004970 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff8a35275470c0 678004993 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678005017 C Bi:6:006:1 0 13 = 55534253 9f000000 00000000 00
ffff8a35275470c0 678005062 S Bo:6:006:2 -115 31 = 55534243 a0000000 00020000 80000a28 00039a33 f3000001 00000000 000000
ffff8a35275470c0 678005077 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678005081 S Bi:6:006:1 -115 512 <
ffff8a35f5f680c0 678005700 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff8a35275470c0 678005711 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678005722 C Bi:6:006:1 0 13 = 55534253 a0000000 00000000 00
ffff8a35275470c0 678005775 S Bo:6:006:2 -115 31 = 55534243 a1000000 00100000 80000a28 00039a32 f0000008 00000000 000000
ffff8a35275470c0 678005791 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678005822 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678006465 C Bi:6:006:1 0 4096 = 8add6e3c 2432a9cf 357c4940 a360e0ab a20f802f 1d321990 325a7363 73ab2a85
ffff8a35275470c0 678006485 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678006542 C Bi:6:006:1 0 13 = 55534253 a1000000 00000000 00
ffff8a35275470c0 678006589 S Bo:6:006:2 -115 31 = 55534243 a2000000 00080000 80000a28 00039a33 e8000004 00000000 000000
ffff8a35275470c0 678006601 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678006605 S Bi:6:006:1 -115 2048 <
ffff8a35f5f680c0 678007232 C Bi:6:006:1 0 2048 = c71672c3 62f9e441 93556627 49f7d2f6 c1556d39 f35c478c b7303075 c5c622c8
ffff8a35275470c0 678007237 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678007272 C Bi:6:006:1 0 13 = 55534253 a2000000 00000000 00
ffff8a35275470c0 678007299 S Bo:6:006:2 -115 31 = 55534243 a3000000 00020000 80000a28 00039a33 ec000001 00000000 000000
ffff8a35275470c0 678007316 C Bo:6:006:2 0 31 >
ffff8a35f006e000 678007335 S Bi:6:006:1 -115 512 <
ffff8a35f006e000 678007905 C Bi:6:006:1 0 512 = 809f4d0a bc5667de 57467a8e 646e8913 8d0bae35 d0f85a27 97065766 1ac31874
ffff8a35275470c0 678007957 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678007991 C Bi:6:006:1 0 13 = 55534253 a3000000 00000000 00
ffff8a35275470c0 678008067 S Bo:6:006:2 -115 31 = 55534243 a4000000 00020000 80000a28 00039a33 ed000001 00000000 000000
ffff8a35275470c0 678008123 C Bo:6:006:2 0 31 >
ffff8a35ff68a000 678008149 S Bi:6:006:1 -115 512 <
ffff8a35ff68a000 678008743 C Bi:6:006:1 0 512 = 97bc843d bb40e8dd 37106a17 8b97f602 c439d3b5 e4dfd162 bd8e6a42 09f83484
ffff8a35275470c0 678008784 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678008811 C Bi:6:006:1 0 13 = 55534253 a4000000 00000000 00
ffff8a35275470c0 678008895 S Bo:6:006:2 -115 31 = 55534243 a5000000 00020000 80000a28 00039a33 ee000001 00000000 000000
ffff8a35275470c0 678008939 C Bo:6:006:2 0 31 >
ffff8a352b385780 678008946 S Bi:6:006:1 -115 512 <
ffff8a352b385780 678009545 C Bi:6:006:1 0 512 = 66f05e02 eb030290 ceab76c6 e426f4b0 6ed9b467 31e125f2 28e6c092 135d8134
ffff8a35275470c0 678009576 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678009609 C Bi:6:006:1 0 13 = 55534253 a5000000 00000000 00
ffff8a35275470c0 678009702 S Bo:6:006:2 -115 31 = 55534243 a6000000 00020000 80000a28 00039a33 ef000001 00000000 000000
ffff8a35275470c0 678009717 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678009722 S Bi:6:006:1 -115 512 <
ffff8a35f5f680c0 678010320 C Bi:6:006:1 0 512 = 3a43294b 77481b39 4cec9d97 da766318 358af9ff e8c5dfce 6bbfff83 e78d3f1c
ffff8a35275470c0 678010325 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678010378 C Bi:6:006:1 0 13 = 55534253 a6000000 00000000 00
ffff8a35275470c0 678010490 S Bo:6:006:2 -115 31 = 55534243 a7000000 00100000 80000a28 00000008 20000008 00000000 000000
ffff8a35275470c0 678010519 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678010524 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678012080 C Bi:6:006:1 0 4096 = 04800000 04800100 04800200 04800300 04800400 04800c00 04800d00 04801800
ffff8a35275470c0 678012093 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678012104 C Bi:6:006:1 0 13 = 55534253 a7000000 00000000 00
ffff8a35275470c0 678012148 S Bo:6:006:2 -115 31 = 55534243 a8000000 00100000 80000a28 00000008 40000008 00000000 000000
ffff8a35275470c0 678012184 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678012200 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678012868 C Bi:6:006:1 0 4096 = 08800000 08800100 08800200 08800300 08800400 08800c00 08800d00 08801800
ffff8a35275470c0 678012879 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678012889 C Bi:6:006:1 0 13 = 55534253 a8000000 00000000 00
ffff8a35275470c0 678012961 S Bo:6:006:2 -115 31 = 55534243 a9000000 00100000 80000a28 00000008 80000008 00000000 000000
ffff8a35275470c0 678013005 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678013011 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678013687 C Bi:6:006:1 0 4096 = 10800000 10800100 10800200 10800300 10800400 10800c00 10800d00 10801800
ffff8a35275470c0 678013699 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678013710 C Bi:6:006:1 0 13 = 55534253 a9000000 00000000 00
ffff8a35275470c0 678013763 S Bo:6:006:2 -115 31 = 55534243 aa000000 00100000 80000a28 00000009 00000008 00000000 000000
ffff8a35275470c0 678013779 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678013784 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678014490 C Bi:6:006:1 0 4096 = 20800000 20800100 20800200 20800300 20800400 20800c00 20800d00 20801800
ffff8a35275470c0 678014502 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678014513 C Bi:6:006:1 0 13 = 55534253 aa000000 00000000 00
ffff8a35275470c0 678014564 S Bo:6:006:2 -115 31 = 55534243 ab000000 00100000 80000a28 0000000a 00000008 00000000 000000
ffff8a35275470c0 678014580 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678014585 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678015278 C Bi:6:006:1 0 4096 = 40800000 40800100 40800200 40800300 40800400 40800c00 40800d00 40801800
ffff8a35275470c0 678015284 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678015319 C Bi:6:006:1 0 13 = 55534253 ab000000 00000000 00
ffff8a35275470c0 678015346 S Bo:6:006:2 -115 31 = 55534243 ac000000 00100000 80000a28 0000000c 00000008 00000000 000000
ffff8a35275470c0 678015358 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678015363 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678016048 C Bi:6:006:1 0 4096 = 80800000 80800100 80800200 80800300 80800400 80800c00 80800d00 80801800
ffff8a35275470c0 678016052 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678016112 C Bi:6:006:1 0 13 = 55534253 ac000000 00000000 00
ffff8a35275470c0 678016186 S Bo:6:006:2 -115 31 = 55534243 ad000000 00100000 80000a28 00000010 00000008 00000000 000000
ffff8a35275470c0 678016204 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678016209 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678016922 C Bi:6:006:1 0 4096 = 00810000 00810100 00810200 00810300 00810400 00810c00 00810d00 00811800
ffff8a35275470c0 678016934 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678016944 C Bi:6:006:1 0 13 = 55534253 ad000000 00000000 00
ffff8a35275470c0 678017004 S Bo:6:006:2 -115 31 = 55534243 ae000000 00100000 80000a28 00000018 00000008 00000000 000000
ffff8a35275470c0 678017020 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678017025 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678017731 C Bi:6:006:1 0 4096 = 00820000 00820100 00820200 00820300 00820400 00820c00 00820d00 00821800
ffff8a35275470c0 678017741 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678017762 C Bi:6:006:1 0 13 = 55534253 ae000000 00000000 00
ffff8a35275470c0 678017799 S Bo:6:006:2 -115 31 = 55534243 af000000 00100000 80000a28 00000028 00000008 00000000 000000
ffff8a35275470c0 678017811 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678017815 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678018549 C Bi:6:006:1 0 4096 = 00840000 00840100 00840200 00840300 00840400 00840c00 00840d00 00841800
ffff8a35275470c0 678018559 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678018567 C Bi:6:006:1 0 13 = 55534253 af000000 00000000 00
ffff8a35275470c0 678018613 S Bo:6:006:2 -115 31 = 55534243 b0000000 00100000 80000a28 00000008 18000008 00000000 000000
ffff8a35275470c0 678018628 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678018632 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678019318 C Bi:6:006:1 0 4096 = 03800000 03800100 03800200 03800300 03800400 03800c00 03800d00 03801800
ffff8a35275470c0 678019322 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678019358 C Bi:6:006:1 0 13 = 55534253 b0000000 00000000 00
ffff8a35275470c0 678019389 S Bo:6:006:2 -115 31 = 55534243 b1000000 00100000 80000a28 00000008 38000008 00000000 000000
ffff8a35275470c0 678019400 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678019414 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678020079 C Bi:6:006:1 0 4096 = 07800000 07800100 07800200 07800300 07800400 07800c00 07800d00 07801800
ffff8a35275470c0 678020085 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678020140 C Bi:6:006:1 0 13 = 55534253 b1000000 00000000 00
ffff8a35275470c0 678020215 S Bo:6:006:2 -115 31 = 55534243 b2000000 00100000 80000a28 00000008 78000008 00000000 000000
ffff8a35275470c0 678020235 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678020240 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678020920 C Bi:6:006:1 0 4096 = 0f800000 0f800100 0f800200 0f800300 0f800400 0f800c00 0f800d00 0f801800
ffff8a35275470c0 678020938 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678020955 C Bi:6:006:1 0 13 = 55534253 b2000000 00000000 00
ffff8a35275470c0 678021015 S Bo:6:006:2 -115 31 = 55534243 b3000000 00100000 80000a28 00000008 10000008 00000000 000000
ffff8a35275470c0 678021026 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678021030 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678021775 C Bi:6:006:1 0 4096 = 00004000 10004000 20004000 f05ff01f 00000500 00000000 00000000 f01f7270
ffff8a35275470c0 678021803 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678021823 C Bi:6:006:1 0 13 = 55534253 b3000000 00000000 00
ffff8a35275470c0 678021876 S Bo:6:006:2 -115 31 = 55534243 b4000000 00200000 80000a28 00000008 28000010 00000000 000000
ffff8a35275470c0 678021917 C Bo:6:006:2 0 31 >
ffff8a35f006e000 678021943 S Bi:6:006:1 -115 8192 <
ffff8a35f006e000 678022739 C Bi:6:006:1 0 8192 = 05800000 05800100 05800200 05800300 05800400 05800c00 05800d00 05801800
ffff8a35275470c0 678022782 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678022811 C Bi:6:006:1 0 13 = 55534253 b4000000 00000000 00
ffff8a35275470c0 678022927 S Bo:6:006:2 -115 31 = 55534243 b5000000 00600000 80000a28 00000008 48000030 00000000 000000
ffff8a35275470c0 678022968 C Bo:6:006:2 0 31 >
ffff8a35ff68a000 678022992 S Bi:6:006:1 -115 24576 <
ffff8a35ff68a000 678024129 C Bi:6:006:1 0 24576 = 09800000 09800100 09800200 09800300 09800400 09800c00 09800d00 09801800
ffff8a35275470c0 678024177 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678024209 C Bi:6:006:1 0 13 = 55534253 b5000000 00000000 00
ffff8a35275470c0 678024350 S Bo:6:006:2 -115 31 = 55534243 b6000000 00f00000 80000a28 00000008 88000078 00000000 000000
ffff8a35275470c0 678024372 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678024378 S Bi:6:006:1 -115 61440 <
ffff8a35f5f680c0 678026480 C Bi:6:006:1 0 61440 = 11800000 11800100 11800200 11800300 11800400 11800c00 11800d00 11801800
ffff8a35275470c0 678026509 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678026525 C Bi:6:006:1 0 13 = 55534253 b6000000 00000000 00
ffff8a35275470c0 678026611 S Bo:6:006:2 -115 31 = 55534243 b7000000 00100000 80000a28 00000009 08000008 00000000 000000
ffff8a35275470c0 678026637 C Bo:6:006:2 0 31 >
ffff8a35f5f680c0 678026647 S Bi:6:006:1 -115 4096 <
ffff8a35f5f680c0 678027332 C Bi:6:006:1 0 4096 = 21800000 21800100 21800200 21800300 21800400 21800c00 21800d00 21801800
ffff8a35275470c0 678027362 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678027388 C Bi:6:006:1 0 13 = 55534253 b7000000 00000000 00
ffff8a35275470c0 678027530 S Bo:6:006:2 -115 31 = 55534243 b8000000 00e00100 80000a28 00000009 100000f0 00000000 000000
ffff8a35275470c0 678027579 C Bo:6:006:2 0 31 >
ffff8a35ff68a000 678027608 S Bi:6:006:1 -115 122880 <
ffff8a35ff68a000 678031234 C Bi:6:006:1 0 122880 = 22800000 22800100 22800200 22800300 22800400 22800c00 22800d00 22801800
ffff8a35275470c0 678031294 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678031324 C Bi:6:006:1 0 13 = 55534253 b8000000 00000000 00
ffff8a35275470c0 678031660 S Bo:6:006:2 -115 31 = 55534243 b9000000 00e00100 80000a28 0000000a 080000f0 00000000 000000
ffff8a35275470c0 678031697 C Bo:6:006:2 0 31 >
ffff8a35ff68a000 678031727 S Bi:6:006:1 -115 122880 <
ffff8a35ff68a000 678035402 C Bi:6:006:1 0 122880 = 41800000 41800100 41800200 41800300 41800400 41800c00 41800d00 41801800
ffff8a35275470c0 678035451 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678035484 C Bi:6:006:1 0 13 = 55534253 b9000000 00000000 00
ffff8a35275470c0 678035589 S Bo:6:006:2 -115 31 = 55534243 ba000000 00200000 80000a28 0000000a f8000010 00000000 000000
ffff8a35275470c0 678035629 C Bo:6:006:2 0 31 >
ffff8a352b385780 678035660 S Bi:6:006:1 -115 8192 <
ffff8a352b385780 678036475 C Bi:6:006:1 0 8192 = 5f800000 5f800100 5f800200 5f800300 5f800400 5f800c00 5f800d00 5f801800
ffff8a35275470c0 678036524 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678036556 C Bi:6:006:1 0 13 = 55534253 ba000000 00000000 00
ffff8a35275470c0 678036855 S Bo:6:006:2 -115 31 = 55534243 bb000000 00e00100 80000a28 0000000b 080000f0 00000000 000000
ffff8a35275470c0 678036882 C Bo:6:006:2 0 31 >
ffff8a352b385780 678036896 S Bi:6:006:1 -115 122880 <
ffff8a352b385780 678040555 C Bi:6:006:1 0 122880 = 61800000 61800100 61800200 61800300 61800400 61800c00 61800d00 61801800
ffff8a35275470c0 678040633 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678040669 C Bi:6:006:1 0 13 = 55534253 bb000000 00000000 00
ffff8a35275470c0 678040812 S Bo:6:006:2 -115 31 = 55534243 bc000000 00100000 80000a28 0000000b f8000008 00000000 000000
ffff8a35275470c0 678040881 C Bo:6:006:2 0 31 >
ffff8a35f006e000 678040920 S Bi:6:006:1 -115 4096 <
ffff8a35f006e000 678041030 C Bi:6:006:1 0 4096 = 7f800000 7f800100 7f800200 7f800300 7f800400 7f800c00 7f800d00 7f801800
ffff8a35275470c0 678041099 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678041137 C Bi:6:006:1 0 13 = 55534253 bc000000 00000000 00
ffff8a35275470c0 678041488 S Bo:6:006:2 -115 31 = 55534243 bd000000 00e00100 80000a28 00039a2e 000000f0 00000000 000000
ffff8a35275470c0 678041509 C Bo:6:006:2 0 31 >
ffff8a35f006e000 678041516 S Bi:6:006:1 -115 122880 <
ffff8a35f006e000 678046043 C Bi:6:006:1 0 122880 = c77ccf3d b18bbc20 49a7c771 ab78fd88 353bdfd1 c2f59b72 ee1c7082 2c62c6f6
ffff8a35275470c0 678046088 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678046100 C Bi:6:006:1 0 13 = 55534253 bd000000 00000000 00
ffff8a35275470c0 678046179 S Bo:6:006:2 -115 31 = 55534243 be000000 00200000 80000a28 00039a2e f0000010 00000000 000000
ffff8a35275470c0 678046202 C Bo:6:006:2 0 31 >
ffff8a352b385780 678046225 S Bi:6:006:1 -115 8192 <
ffff8a352b385780 678046958 C Bi:6:006:1 0 8192 = 2d1e8c4f 0d6d6356 384be8bc b5cfb50e cdb3025f bcfecddd 4f80d2e5 eb6b3284
ffff8a35275470c0 678046994 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678047005 C Bi:6:006:1 0 13 = 55534253 be000000 00000000 00
ffff8a35275470c0 678047234 S Bo:6:006:2 -115 31 = 55534243 bf000000 00e00100 80000a28 00039a2f 000000f0 00000000 000000
ffff8a35275470c0 678047258 C Bo:6:006:2 0 31 >
ffff8a352b385780 678047264 S Bi:6:006:1 -115 122880 <
ffff8a352b385780 678050994 C Bi:6:006:1 0 122880 = aa1c9eea 8ab1cada d7bd4902 c36a01ea 724ec451 e5509315 ca093a94 9dcab358
ffff8a35275470c0 678051052 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678051090 C Bi:6:006:1 0 13 = 55534253 bf000000 00000000 00
ffff8a35275470c0 678051234 S Bo:6:006:2 -115 31 = 55534243 c0000000 00200000 80000a28 00039a2f f0000010 00000000 000000
ffff8a35275470c0 678051272 C Bo:6:006:2 0 31 >
ffff8a35f006e000 678051291 S Bi:6:006:1 -115 8192 <
ffff8a35f006e000 678051497 C Bi:6:006:1 0 8192 = 6ddcf87d 817edc15 a527992a 77e6c620 1adc6410 77ef845a d44f3327 5ac54da3
ffff8a35275470c0 678051574 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678051611 C Bi:6:006:1 0 13 = 55534253 c0000000 00000000 00
ffff8a35275470c0 678052005 S Bo:6:006:2 -115 31 = 55534243 c1000000 00e00100 80000a28 00039a30 000000f0 00000000 000000
ffff8a35275470c0 678052031 C Bo:6:006:2 0 31 >
ffff8a35f006e000 678052058 S Bi:6:006:1 -115 122880 <
ffff8a35f006e000 678055233 C Bi:6:006:1 0 122880 = c2bb8f02 e6227d01 1316d5d4 f8589196 3738834e 7c5c90cb 17868ef4 9eb65148
ffff8a35275470c0 678055290 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678055329 C Bi:6:006:1 0 13 = 55534253 c1000000 00000000 00
ffff8a35275470c0 678055475 S Bo:6:006:2 -115 31 = 55534243 c2000000 00200000 80000a28 00039a30 f0000010 00000000 000000
ffff8a35275470c0 678055512 C Bo:6:006:2 0 31 >
ffff8a35ff68a000 678055545 S Bi:6:006:1 -115 8192 <
ffff8a35ff68a000 678055821 C Bi:6:006:1 0 8192 = 2378ad20 bc2a7d26 58ef9fc8 48429609 72a99082 f13ed042 ba93cad1 de25670f
ffff8a35275470c0 678055887 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678055932 C Bi:6:006:1 0 13 = 55534253 c2000000 00000000 00
ffff8a35275470c0 678056390 S Bo:6:006:2 -115 31 = 55534243 c3000000 00e00100 80000a28 00039a31 000000f0 00000000 000000
ffff8a35275470c0 678056452 C Bo:6:006:2 0 31 >
ffff8a35ff68a000 678056498 S Bi:6:006:1 -115 122880 <
ffff8a35ff68a000 678059683 C Bi:6:006:1 0 122880 = 315bbba0 afae98ec 6e222b9d ba2bc3bd 1a3f0717 1688e8e5 c30b4584 6f4f58ae
ffff8a35275470c0 678059749 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678059795 C Bi:6:006:1 0 13 = 55534253 c3000000 00000000 00
ffff8a35275470c0 678059945 S Bo:6:006:2 -115 31 = 55534243 c4000000 00200000 80000a28 00039a31 f0000010 00000000 000000
ffff8a35275470c0 678059982 C Bo:6:006:2 0 31 >
ffff8a352b385780 678060016 S Bi:6:006:1 -115 8192 <
ffff8a352b385780 678060195 C Bi:6:006:1 0 8192 = 15c5f9a5 4fdcf7b5 d4013ce6 82011093 87fb73d9 5b8553c4 c7965bfb e4ae9642
ffff8a35275470c0 678060244 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678060260 C Bi:6:006:1 0 13 = 55534253 c4000000 00000000 00
ffff8a35275470c0 678060759 S Bo:6:006:2 -115 31 = 55534243 c5000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 678060787 C Bo:6:006:2 0 31 >
ffff8a35275470c0 678060822 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678060833 C Bi:6:006:1 0 13 = 55534253 c5000000 00000000 00
ffff8a35275470c0 678061075 S Bo:6:006:2 -115 31 = 55534243 c6000000 00100000 80000a28 00000000 00000008 00000000 000000
ffff8a35275470c0 678061101 C Bo:6:006:2 0 31 >
ffff8a352b385780 678061137 S Bi:6:006:1 -115 4096 <
ffff8a352b385780 678062606 C Bi:6:006:1 0 4096 = fab80000 8ed0bc00 7c8bf450 07501ffb fcbf0006 b90001f3 a5ea1e06 0000bebe
ffff8a35275470c0 678062655 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678062669 C Bi:6:006:1 0 13 = 55534253 c6000000 00000000 00
ffff8a35275470c0 678063575 S Bo:6:006:2 -115 31 = 55534243 c7000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 678063604 C Bo:6:006:2 0 31 >
ffff8a35275470c0 678063634 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678063645 C Bi:6:006:1 0 13 = 55534253 c7000000 00000000 00
ffff8a35275470c0 678063812 S Bo:6:006:2 -115 31 = 55534243 c8000000 00040000 80000a28 00000008 02000002 00000000 000000
ffff8a35275470c0 678063837 C Bo:6:006:2 0 31 >
ffff8a352b385780 678063847 S Bi:6:006:1 -115 1024 <
ffff8a352b385780 678064482 C Bi:6:006:1 0 1024 = 90d11c00 7e457300 79c30500 2fa65700 7fd11c00 00000000 02000000 02000000
ffff8a35275470c0 678064515 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678064529 C Bi:6:006:1 0 13 = 55534253 c8000000 00000000 00
ffff8a35275470c0 678064618 S Bo:6:006:2 -115 31 = 55534243 c9000000 00100000 80000a28 00000008 00000008 00000000 000000
ffff8a35275470c0 678064638 C Bo:6:006:2 0 31 >
ffff8a352b385780 678064646 S Bi:6:006:1 -115 4096 <
ffff8a352b385780 678065335 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a35275470c0 678065344 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678065375 C Bi:6:006:1 0 13 = 55534253 c9000000 00000000 00
ffff8a35275470c0 678065421 S Bo:6:006:2 -115 31 = 55534243 ca000000 00100000 80000a28 00000008 08000008 00000000 000000
ffff8a35275470c0 678065468 C Bo:6:006:2 0 31 >
ffff8a352b385780 678065475 S Bi:6:006:1 -115 4096 <
ffff8a352b385780 678066167 C Bi:6:006:1 0 4096 = 01040000 11040000 21040000 e85bdf1f 02000400 00000000 00000000 dc1f8466
ffff8a35275470c0 678066179 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678066195 C Bi:6:006:1 0 13 = 55534253 ca000000 00000000 00
ffff8a35275470c0 678066233 S Bo:6:006:2 -115 31 = 55534243 cb000000 00100000 80000a28 00000008 10000008 00000000 000000
ffff8a35275470c0 678066252 C Bo:6:006:2 0 31 >
ffff8a352b385780 678066260 S Bi:6:006:1 -115 4096 <
ffff8a352b385780 678066385 C Bi:6:006:1 0 4096 = 00004000 10004000 20004000 f05ff01f 00000500 00000000 00000000 f01f7270
ffff8a35275470c0 678066394 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678066425 C Bi:6:006:1 0 13 = 55534253 cb000000 00000000 00
ffff8a35275470c0 678066656 S Bo:6:006:2 -115 31 = 55534243 cc000000 00100000 80000a28 00000029 10000008 00000000 000000
ffff8a35275470c0 678066678 C Bo:6:006:2 0 31 >
ffff8a352b385780 678066699 S Bi:6:006:1 -115 4096 <
ffff8a352b385780 678067398 C Bi:6:006:1 0 4096 = a4810000 0080841e 2a0d4d5d 92124d5d 92124d5d 00000000 00000100 40420f00
ffff8a35275470c0 678067449 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678067473 C Bi:6:006:1 0 13 = 55534253 cc000000 00000000 00
ffff8a35275470c0 678067536 S Bo:6:006:2 -115 31 = 55534243 cd000000 00100000 80000a28 00000029 08000008 00000000 000000
ffff8a35275470c0 678067589 C Bo:6:006:2 0 31 >
ffff8a352b385780 678067639 S Bi:6:006:1 -115 4096 <
ffff8a352b385780 678068229 C Bi:6:006:1 0 4096 = 00000000 00000000 33f72257 33f72257 33f72257 00000000 00000000 00000000
ffff8a35275470c0 678068273 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678068286 C Bi:6:006:1 0 13 = 55534253 cd000000 00000000 00
ffff8a35275470c0 678068403 S Bo:6:006:2 -115 31 = 55534243 ce000000 00100000 80000a28 0001c408 00000008 00000000 000000
ffff8a35275470c0 678068427 C Bo:6:006:2 0 31 >
ffff8a352b385780 678068460 S Bi:6:006:1 -115 4096 <
ffff8a352b385780 678069983 C Bi:6:006:1 0 4096 = c03b3998 00000004 00000000 00001000 00008000 00000001 0003101a 00000000
ffff8a35275470c0 678070030 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678070044 C Bi:6:006:1 0 13 = 55534253 ce000000 00000000 00
ffff8a35275470c0 678071257 S Bo:6:006:2 -115 31 = 55534243 cf000000 00100000 00000a2a 00000008 00000008 00000000 000000
ffff8a35275470c0 678071283 C Bo:6:006:2 0 31 >
ffff8a352b385780 678071314 S Bo:6:006:2 -115 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff8a352b385780 678071415 C Bo:6:006:2 0 4096 >
ffff8a35275470c0 678071459 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678071483 C Bi:6:006:1 0 13 = 55534253 cf000000 00000000 00
ffff8a35275470c0 678080003 S Bo:6:006:2 -115 31 = 55534243 d0000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 678080022 C Bo:6:006:2 0 31 >
ffff8a35275470c0 678080027 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 678080058 C Bi:6:006:1 0 13 = 55534253 d0000000 00000000 00
ffff8a35f0776d80 678815582 S Bo:6:003:1 -115 31 = 55534243 130b0000 00000000 00020600 00000000 00000000 00000000 000000
ffff8a35f0776d80 678815678 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 678815728 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 678816060 C Bi:6:003:2 0 13 = 55534253 130b0000 00000000 01
ffff8a35f0776d80 678816117 S Bo:6:003:1 -115 31 = 55534243 140b0000 12000000 80020603 00000012 00000000 00000000 000000
ffff8a35f0776d80 678816191 C Bo:6:003:1 0 31 >
ffff8a35ff68a000 678816254 S Bi:6:003:2 -115 18 <
ffff8a35ff68a000 678816725 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 678816782 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 678816870 C Bi:6:003:2 0 13 = 55534253 140b0000 00000000 00
ffff8a35f0776d80 678816995 S Bo:6:003:1 -115 31 = 55534243 150b0000 00000000 00010600 00000000 00000000 00000000 000000
ffff8a35f0776d80 678817076 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 678817130 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 678817458 C Bi:6:003:2 0 13 = 55534253 150b0000 00000000 01
ffff8a35f0776d80 678817519 S Bo:6:003:1 -115 31 = 55534243 160b0000 12000000 80010603 00000012 00000000 00000000 000000
ffff8a35f0776d80 678817593 C Bo:6:003:1 0 31 >
ffff8a35ff68a000 678817647 S Bi:6:003:2 -115 18 <
ffff8a35ff68a000 678818121 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 678818176 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 678818274 C Bi:6:003:2 0 13 = 55534253 160b0000 00000000 00
ffff8a35f0776d80 678818455 S Bo:6:003:1 -115 31 = 55534243 170b0000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35f0776d80 678818549 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 678818568 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 678818937 C Bi:6:003:2 0 13 = 55534253 170b0000 00000000 01
ffff8a35f0776d80 678818990 S Bo:6:003:1 -115 31 = 55534243 180b0000 12000000 80000603 00000012 00000000 00000000 000000
ffff8a35f0776d80 678819071 C Bo:6:003:1 0 31 >
ffff8a35ff68a000 678819121 S Bi:6:003:2 -115 18 <
ffff8a35ff68a000 678819628 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 678819702 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 678819764 C Bi:6:003:2 0 13 = 55534253 180b0000 00000000 00
ffff8a35f0776d80 678879616 S Bo:6:003:1 -115 31 = 55534243 190b0000 00000000 00030600 00000000 00000000 00000000 000000
ffff8a35f0776d80 678879710 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 678879761 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 678880099 C Bi:6:003:2 0 13 = 55534253 190b0000 00000000 01
ffff8a35f0776d80 678880156 S Bo:6:003:1 -115 31 = 55534243 1a0b0000 12000000 80030603 00000012 00000000 00000000 000000
ffff8a35f0776d80 678880212 C Bo:6:003:1 0 31 >
ffff8a35ff68a000 678880264 S Bi:6:003:2 -115 18 <
ffff8a35ff68a000 678880757 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 678880812 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 678880869 C Bi:6:003:2 0 13 = 55534253 1a0b0000 00000000 00
ffff8a35f0d2d840 678943645 C Ii:6:004:1 0:2 8 = 0000fdff 00000000
ffff8a35f0d2d840 678943718 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 678951621 C Ii:6:004:1 0:2 8 = 0000ffff 00000000
ffff8a35f0d2d840 678951638 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 678959635 C Ii:6:004:1 0:2 8 = 0000ffff 00000000
ffff8a35f0d2d840 678959668 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 678975672 C Ii:6:004:1 0:2 8 = 0000ffff 00000000
ffff8a35f0d2d840 678975722 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679037680 C Ii:6:004:1 0:2 8 = 0000ffff 00000000
ffff8a35f0d2d840 679037744 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679045686 C Ii:6:004:1 0:2 8 = 0000ffff 00000000
ffff8a35f0d2d840 679045756 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679403668 C Ii:6:004:1 0:2 8 = 00000600 02000000
ffff8a35f0d2d840 679403760 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679411636 C Ii:6:004:1 0:2 8 = 00000600 01000000
ffff8a35f0d2d840 679411692 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679419669 C Ii:6:004:1 0:2 8 = 00000500 02000000
ffff8a35f0d2d840 679419755 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679427669 C Ii:6:004:1 0:2 8 = 00000500 02000000
ffff8a35f0d2d840 679427755 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679435668 C Ii:6:004:1 0:2 8 = 00000400 01000000
ffff8a35f0d2d840 679435755 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679443635 C Ii:6:004:1 0:2 8 = 00000300 01000000
ffff8a35f0d2d840 679443712 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679451635 C Ii:6:004:1 0:2 8 = 00000300 02000000
ffff8a35f0d2d840 679451653 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679459660 C Ii:6:004:1 0:2 8 = 00000300 03000000
ffff8a35f0d2d840 679459704 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679467634 C Ii:6:004:1 0:2 8 = 00000200 04000000
ffff8a35f0d2d840 679467674 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679473659 C Ii:6:004:1 0:2 8 = 00000200 02000000
ffff8a35f0d2d840 679473703 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679481664 C Ii:6:004:1 0:2 8 = 00000200 03000000
ffff8a35f0d2d840 679481715 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679489666 C Ii:6:004:1 0:2 8 = 00000200 01000000
ffff8a35f0d2d840 679489716 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679501665 C Ii:6:004:1 0:2 8 = 00000200 01000000
ffff8a35f0d2d840 679501726 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679505669 C Ii:6:004:1 0:2 8 = 00000100 00000000
ffff8a35f0d2d840 679505721 S Ii:6:004:1 -115:2 8 <
ffff8a35f006e9c0 679799926 S Ci:6:002:0 s a3 00 0000 0001 0004 4 <
ffff8a35f006e9c0 679800004 C Ci:6:002:0 0 4 = 07050000
ffff8a35165ce000 679800042 S Co:6:002:0 s 23 01 0002 0001 0000 0
ffff8a35165ce000 679800065 C Co:6:002:0 0 0
ffff8a35165ce000 679841561 S Ci:6:002:0 s a3 00 0000 0001 0004 4 <
ffff8a35165ce000 679841605 C Ci:6:002:0 0 4 = 03050400
ffff8a35f006e0c0 679853551 S Co:6:002:0 s 23 01 0012 0001 0000 0
ffff8a35f006e0c0 679853616 C Co:6:002:0 0 0
ffff8a35f006e0c0 679853673 S Ci:6:005:0 s 80 00 0000 0000 0002 2 <
ffff8a35f006e0c0 679853738 C Ci:6:005:0 0 2 = 0300
ffff8a35f006e0c0 679853801 S Co:6:005:0 s 00 01 0001 0000 0000 0
ffff8a35f006e0c0 679853850 C Co:6:005:0 0 0
ffff8a35f006e0c0 679853912 S Ci:6:005:0 s a3 00 0000 0001 0004 4 <
ffff8a35f006e0c0 679853976 C Ci:6:005:0 0 4 = 00010000
ffff8a35f006e0c0 679854030 S Ci:6:005:0 s a3 00 0000 0002 0004 4 <
ffff8a35f006e0c0 679854093 C Ci:6:005:0 0 4 = 00010000
ffff8a35f006e0c0 679854151 S Ci:6:005:0 s a3 00 0000 0003 0004 4 <
ffff8a35f006e0c0 679854215 C Ci:6:005:0 0 4 = 00010000
ffff8a35f006e0c0 679854273 S Ci:6:005:0 s a3 00 0000 0004 0004 4 <
ffff8a35f006e0c0 679854336 C Ci:6:005:0 0 4 = 00010000
ffff8a35ef1d5780 679854359 S Ii:6:005:1 -115:2048 1 <
ffff8a35ef1d5780 679854584 C Ii:6:005:1 -2:2048 0
ffff8a35f006e180 679854607 S Co:6:005:0 s 00 03 0001 0000 0000 0
ffff8a35f006e180 679854645 C Co:6:005:0 0 0
ffff8a35f006e180 679854656 S Co:6:002:0 s 23 03 0002 0001 0000 0
ffff8a35f006e180 679854689 C Co:6:002:0 0 0
ffff8a35f0d2d840 679857616 C Ii:6:004:1 0:2 8 = 00000000 01000000
ffff8a35f0d2d840 679857680 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679865622 C Ii:6:004:1 0:2 8 = 00000000 01000000
ffff8a35f0d2d840 679865698 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679873672 C Ii:6:004:1 0:2 8 = 00000000 01000000
ffff8a35f0d2d840 679873751 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679881664 C Ii:6:004:1 0:2 8 = 00000000 01000000
ffff8a35f0d2d840 679881740 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679895599 C Ii:6:004:1 0:2 8 = 00000000 02000000
ffff8a35f0d2d840 679895679 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679919663 C Ii:6:004:1 0:2 8 = 00000000 01000000
ffff8a35f0d2d840 679919753 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679935624 C Ii:6:004:1 0:2 8 = 0000feff 00000000
ffff8a35f0d2d840 679935704 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679981661 C Ii:6:004:1 0:2 8 = 0000feff 01000000
ffff8a35f0d2d840 679981753 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679991667 C Ii:6:004:1 0:2 8 = 0000feff 01000000
ffff8a35f0d2d840 679991756 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 679997620 C Ii:6:004:1 0:2 8 = 0000fcff 00000000
ffff8a35f0d2d840 679997707 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680005662 C Ii:6:004:1 0:2 8 = 0000fcff 00000000
ffff8a35f0d2d840 680005748 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680013659 C Ii:6:004:1 0:2 8 = 0000faff 00000000
ffff8a35f0d2d840 680013740 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680021596 C Ii:6:004:1 0:2 8 = 0000f9ff feff0000
ffff8a35f0d2d840 680021669 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680029628 C Ii:6:004:1 0:2 8 = 0000f7ff feff0000
ffff8a35f0d2d840 680029711 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680037661 C Ii:6:004:1 0:2 8 = 0000f5ff fcff0000
ffff8a35f0d2d840 680037751 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680045658 C Ii:6:004:1 0:2 8 = 0000f2ff f9ff0000
ffff8a35f0d2d840 680045747 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680053658 C Ii:6:004:1 0:2 8 = 0000f1ff f7ff0000
ffff8a35f0d2d840 680053749 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680061658 C Ii:6:004:1 0:2 8 = 0000edff f6ff0000
ffff8a35f0d2d840 680061747 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680067630 C Ii:6:004:1 0:2 8 = 0000ebff f4ff0000
ffff8a35f0d2d840 680067725 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680075639 C Ii:6:004:1 0:2 8 = 0000ebff f2ff0000
ffff8a35f0d2d840 680075725 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680083668 C Ii:6:004:1 0:2 8 = 0000e8ff f0ff0000
ffff8a35f0d2d840 680083746 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680091668 C Ii:6:004:1 0:2 8 = 0000e7ff f0ff0000
ffff8a35f0d2d840 680091747 S Ii:6:004:1 -115:2 8 <
ffff8a35275470c0 680095591 S Bo:6:006:2 -115 31 = 55534243 d1000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 680095651 C Bo:6:006:2 0 31 >
ffff8a35275470c0 680095673 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 680095698 C Bi:6:006:1 0 13 = 55534253 d1000000 00000000 00
ffff8a35f0d2d840 680099661 C Ii:6:004:1 0:2 8 = 0000e8ff eeff0000
ffff8a35f0d2d840 680099748 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680107656 C Ii:6:004:1 0:2 8 = 0000e8ff f1ff0000
ffff8a35f0d2d840 680107744 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680115660 C Ii:6:004:1 0:2 8 = 0000ebff f1ff0000
ffff8a35f0d2d840 680115748 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680123657 C Ii:6:004:1 0:2 8 = 0000ecff f3ff0000
ffff8a35f0d2d840 680123744 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680131596 C Ii:6:004:1 0:2 8 = 0000f1ff f4ff0000
ffff8a35f0d2d840 680131679 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680139624 C Ii:6:004:1 0:2 8 = 0000f1ff f6ff0000
ffff8a35f0d2d840 680139706 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680145659 C Ii:6:004:1 0:2 8 = 0000f3ff f7ff0000
ffff8a35f0d2d840 680145745 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680153626 C Ii:6:004:1 0:2 8 = 0000f3ff f7ff0000
ffff8a35f0d2d840 680153708 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680161654 C Ii:6:004:1 0:2 8 = 0000f3ff f9ff0000
ffff8a35f0d2d840 680161739 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680169658 C Ii:6:004:1 0:2 8 = 0000f4ff fbff0000
ffff8a35f0d2d840 680169744 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680177593 C Ii:6:004:1 0:2 8 = 0000f5ff fbff0000
ffff8a35f0d2d840 680177663 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680185628 C Ii:6:004:1 0:2 8 = 0000f6ff fdff0000
ffff8a35f0d2d840 680185713 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680193621 C Ii:6:004:1 0:2 8 = 0000f7ff fdff0000
ffff8a35f0d2d840 680193666 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680201649 C Ii:6:004:1 0:2 8 = 0000f8ff ffff0000
ffff8a35f0d2d840 680201701 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680209661 C Ii:6:004:1 0:2 8 = 0000f8ff fdff0000
ffff8a35f0d2d840 680209731 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680217613 C Ii:6:004:1 0:2 8 = 0000faff ffff0000
ffff8a35f0d2d840 680217683 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680223615 C Ii:6:004:1 0:2 8 = 0000f9ff feff0000
ffff8a35f0d2d840 680223685 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680231618 C Ii:6:004:1 0:2 8 = 0000f9ff feff0000
ffff8a35f0d2d840 680231690 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680239616 C Ii:6:004:1 0:2 8 = 0000f9ff ffff0000
ffff8a35f0d2d840 680239722 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680247657 C Ii:6:004:1 0:2 8 = 0000fbff feff0000
ffff8a35f0d2d840 680247742 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680255618 C Ii:6:004:1 0:2 8 = 0000fdff feff0000
ffff8a35f0d2d840 680255687 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680263619 C Ii:6:004:1 0:2 8 = 0000fdff feff0000
ffff8a35f0d2d840 680263709 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680271589 C Ii:6:004:1 0:2 8 = 0000ffff ffff0000
ffff8a35f0d2d840 680271656 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680279589 C Ii:6:004:1 0:2 8 = 0000ffff 00000000
ffff8a35f0d2d840 680279647 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680389661 C Ii:6:004:1 0:2 8 = 00000300 02000000
ffff8a35f0d2d840 680389752 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680395620 C Ii:6:004:1 0:2 8 = 00000500 03000000
ffff8a35f0d2d840 680395701 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680403653 C Ii:6:004:1 0:2 8 = 00000600 03000000
ffff8a35f0d2d840 680403739 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680411610 C Ii:6:004:1 0:2 8 = 00000800 03000000
ffff8a35f0d2d840 680411657 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680419638 C Ii:6:004:1 0:2 8 = 00000800 03000000
ffff8a35f0d2d840 680419693 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680427611 C Ii:6:004:1 0:2 8 = 00000800 03000000
ffff8a35f0d2d840 680427678 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680435637 C Ii:6:004:1 0:2 8 = 00000900 02000000
ffff8a35f0d2d840 680435712 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680443611 C Ii:6:004:1 0:2 8 = 00000700 02000000
ffff8a35f0d2d840 680443673 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680451592 C Ii:6:004:1 0:2 8 = 00000600 01000000
ffff8a35f0d2d840 680451616 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680459617 C Ii:6:004:1 0:2 8 = 00000500 02000000
ffff8a35f0d2d840 680459645 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680467638 C Ii:6:004:1 0:2 8 = 00000100 00000000
ffff8a35f0d2d840 680467673 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680505647 C Ii:6:004:1 0:2 8 = 0000fdff 00000000
ffff8a35f0d2d840 680505707 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680513629 C Ii:6:004:1 0:2 8 = 0000fdff 00000000
ffff8a35f0d2d840 680513694 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680521619 C Ii:6:004:1 0:2 8 = 0000feff 01000000
ffff8a35f0d2d840 680521684 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680529626 C Ii:6:004:1 0:2 8 = 0000fdff 00000000
ffff8a35f0d2d840 680529689 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680537585 C Ii:6:004:1 0:2 8 = 0000fcff 02000000
ffff8a35f0d2d840 680537646 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680545617 C Ii:6:004:1 0:2 8 = 0000feff 01000000
ffff8a35f0d2d840 680545671 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680553627 C Ii:6:004:1 0:2 8 = 0000feff 01000000
ffff8a35f0d2d840 680553720 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680559632 C Ii:6:004:1 0:2 8 = 0000ffff 01000000
ffff8a35f0d2d840 680559699 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680567614 C Ii:6:004:1 0:2 8 = 0000ffff 00000000
ffff8a35f0d2d840 680567686 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680575621 C Ii:6:004:1 0:2 8 = 00000000 01000000
ffff8a35f0d2d840 680575703 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680739621 C Ii:6:004:1 0:2 8 = 01000000 00000000
ffff8a35f0d2d840 680739712 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 680803587 C Ii:6:004:1 0:2 8 = 00000000 00000000
ffff8a35f0d2d840 680803618 S Ii:6:004:1 -115:2 8 <
ffff8a35f0776d80 680867490 S Bo:6:003:1 -115 31 = 55534243 1b0b0000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35f0776d80 680867536 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 680867556 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 680867909 C Bi:6:003:2 0 13 = 55534253 1b0b0000 00000000 01
ffff8a35f0776d80 680867934 S Bo:6:003:1 -115 31 = 55534243 1c0b0000 12000000 80000603 00000012 00000000 00000000 000000
ffff8a35f0776d80 680867972 C Bo:6:003:1 0 31 >
ffff8a35ff68a900 680867992 S Bi:6:003:2 -115 18 <
ffff8a35ff68a900 680868514 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 680868538 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 680868663 C Bi:6:003:2 0 13 = 55534253 1c0b0000 00000000 00
ffff8a35f0776d80 680868708 S Bo:6:003:1 -115 31 = 55534243 1d0b0000 00000000 00020600 00000000 00000000 00000000 000000
ffff8a35f0776d80 680868744 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 680868756 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 680869117 C Bi:6:003:2 0 13 = 55534253 1d0b0000 00000000 01
ffff8a35f0776d80 680869136 S Bo:6:003:1 -115 31 = 55534243 1e0b0000 12000000 80020603 00000012 00000000 00000000 000000
ffff8a35f0776d80 680869171 C Bo:6:003:1 0 31 >
ffff8a35ff68a900 680869174 S Bi:6:003:2 -115 18 <
ffff8a35ff68a900 680869713 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 680869732 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 680869857 C Bi:6:003:2 0 13 = 55534253 1e0b0000 00000000 00
ffff8a35f0776d80 680869891 S Bo:6:003:1 -115 31 = 55534243 1f0b0000 00000000 00010600 00000000 00000000 00000000 000000
ffff8a35f0776d80 680869927 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 680869940 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 680870299 C Bi:6:003:2 0 13 = 55534253 1f0b0000 00000000 01
ffff8a35f0776d80 680870315 S Bo:6:003:1 -115 31 = 55534243 200b0000 12000000 80010603 00000012 00000000 00000000 000000
ffff8a35f0776d80 680870350 C Bo:6:003:1 0 31 >
ffff8a35ff68a900 680870354 S Bi:6:003:2 -115 18 <
ffff8a35ff68a900 680870910 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 680870946 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 680871056 C Bi:6:003:2 0 13 = 55534253 200b0000 00000000 00
ffff8a35f0776d80 680927486 S Bo:6:003:1 -115 31 = 55534243 210b0000 00000000 00030600 00000000 00000000 00000000 000000
ffff8a35f0776d80 680927543 C Bo:6:003:1 0 31 >
ffff8a35f0776d80 680927571 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 680927924 C Bi:6:003:2 0 13 = 55534253 210b0000 00000000 01
ffff8a35f0776d80 680927936 S Bo:6:003:1 -115 31 = 55534243 220b0000 12000000 80030603 00000012 00000000 00000000 000000
ffff8a35f0776d80 680927973 C Bo:6:003:1 0 31 >
ffff8a352b385d80 680927980 S Bi:6:003:2 -115 18 <
ffff8a352b385d80 680928515 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff8a35f0776d80 680928540 S Bi:6:003:2 -115 13 <
ffff8a35f0776d80 680928667 C Bi:6:003:2 0 13 = 55534253 220b0000 00000000 00
ffff8a35f0d2d840 682045618 C Ii:6:004:1 0:2 8 = 00000b00 04000000
ffff8a35f0d2d840 682045712 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682053603 C Ii:6:004:1 0:2 8 = 00001800 07000000
ffff8a35f0d2d840 682053669 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682059598 C Ii:6:004:1 0:2 8 = 00001800 04000000
ffff8a35f0d2d840 682059660 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682067583 C Ii:6:004:1 0:2 8 = 00001800 01000000
ffff8a35f0d2d840 682067663 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682075627 C Ii:6:004:1 0:2 8 = 00001a00 00000000
ffff8a35f0d2d840 682075709 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682083584 C Ii:6:004:1 0:2 8 = 00001500 00000000
ffff8a35f0d2d840 682083647 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682091554 C Ii:6:004:1 0:2 8 = 00001300 00000000
ffff8a35f0d2d840 682091627 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682099620 C Ii:6:004:1 0:2 8 = 00001100 00000000
ffff8a35f0d2d840 682099706 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682107617 C Ii:6:004:1 0:2 8 = 00001200 feff0000
ffff8a35f0d2d840 682107706 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682115585 C Ii:6:004:1 0:2 8 = 00001000 00000000
ffff8a35f0d2d840 682115661 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682123590 C Ii:6:004:1 0:2 8 = 00001000 ffff0000
ffff8a35f0d2d840 682123662 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682131619 C Ii:6:004:1 0:2 8 = 00001000 00000000
ffff8a35f0d2d840 682131705 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682139619 C Ii:6:004:1 0:2 8 = 00000d00 ffff0000
ffff8a35f0d2d840 682139702 S Ii:6:004:1 -115:2 8 <
ffff8a35275470c0 682143548 S Bo:6:006:2 -115 31 = 55534243 d2000000 00000000 00000600 00000000 00000000 00000000 000000
ffff8a35275470c0 682143596 C Bo:6:006:2 0 31 >
ffff8a35275470c0 682143643 S Bi:6:006:1 -115 13 <
ffff8a35275470c0 682143670 C Bi:6:006:1 0 13 = 55534253 d2000000 00000000 00
ffff8a35f0d2d840 682145619 C Ii:6:004:1 0:2 8 = 00000a00 00000000
ffff8a35f0d2d840 682145706 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682153619 C Ii:6:004:1 0:2 8 = 00000700 ffff0000
ffff8a35f0d2d840 682153706 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682161616 C Ii:6:004:1 0:2 8 = 00000500 ffff0000
ffff8a35f0d2d840 682161703 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682231617 C Ii:6:004:1 0:2 8 = 00000000 02000000
ffff8a35f0d2d840 682231699 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682247617 C Ii:6:004:1 0:2 8 = 00000000 01000000
ffff8a35f0d2d840 682247704 S Ii:6:004:1 -115:2 8 <
ffff8a35f0d2d840 682263582 C Ii:6:004:1 0:2 8 = 00000000 01000000
ffff8a35f0d2d840 682263667 S Ii:6:004:1 -115:2 8 <

[-- Attachment #3: good.mon.out --]
[-- Type: text/plain, Size: 124851 bytes --]

ffff9067b1774780 915438503 S Bo:6:003:1 -115 31 = 55534243 a90e0000 00000000 00000600 00000000 00000000 00000000 000000
ffff9067b1774780 915438608 C Bo:6:003:1 0 31 >
ffff9067b1774780 915438664 S Bi:6:003:2 -115 13 <
ffff9067b1774780 915438987 C Bi:6:003:2 0 13 = 55534253 a90e0000 00000000 01
ffff9067b1774780 915439040 S Bo:6:003:1 -115 31 = 55534243 aa0e0000 12000000 80000603 00000012 00000000 00000000 000000
ffff9067b1774780 915439096 C Bo:6:003:1 0 31 >
ffff9067ae55e180 915439145 S Bi:6:003:2 -115 18 <
ffff9067ae55e180 915439648 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 915439705 S Bi:6:003:2 -115 13 <
ffff9067b1774780 915439776 C Bi:6:003:2 0 13 = 55534253 aa0e0000 00000000 00
ffff9067b1774780 915439883 S Bo:6:003:1 -115 31 = 55534243 ab0e0000 00000000 00010600 00000000 00000000 00000000 000000
ffff9067b1774780 915439946 C Bo:6:003:1 0 31 >
ffff9067b1774780 915439960 S Bi:6:003:2 -115 13 <
ffff9067b1774780 915440306 C Bi:6:003:2 0 13 = 55534253 ab0e0000 00000000 01
ffff9067b1774780 915440321 S Bo:6:003:1 -115 31 = 55534243 ac0e0000 12000000 80010603 00000012 00000000 00000000 000000
ffff9067b1774780 915440367 C Bo:6:003:1 0 31 >
ffff90673f63ae40 915440388 S Bi:6:003:2 -115 18 <
ffff90673f63ae40 915440896 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 915440911 S Bi:6:003:2 -115 13 <
ffff9067b1774780 915441048 C Bi:6:003:2 0 13 = 55534253 ac0e0000 00000000 00
ffff9067b1774780 915441127 S Bo:6:003:1 -115 31 = 55534243 ad0e0000 00000000 00020600 00000000 00000000 00000000 000000
ffff9067b1774780 915441173 C Bo:6:003:1 0 31 >
ffff9067b1774780 915441223 S Bi:6:003:2 -115 13 <
ffff9067b1774780 915441576 C Bi:6:003:2 0 13 = 55534253 ad0e0000 00000000 01
ffff9067b1774780 915441631 S Bo:6:003:1 -115 31 = 55534243 ae0e0000 12000000 80020603 00000012 00000000 00000000 000000
ffff9067b1774780 915441687 C Bo:6:003:1 0 31 >
ffff9067ae55e180 915441735 S Bi:6:003:2 -115 18 <
ffff9067ae55e180 915442242 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 915442293 S Bi:6:003:2 -115 13 <
ffff9067b1774780 915442374 C Bi:6:003:2 0 13 = 55534253 ae0e0000 00000000 00
ffff9067b1774780 915442462 S Bo:6:003:1 -115 31 = 55534243 af0e0000 00000000 00030600 00000000 00000000 00000000 000000
ffff9067b1774780 915442529 C Bo:6:003:1 0 31 >
ffff9067b1774780 915442543 S Bi:6:003:2 -115 13 <
ffff9067b1774780 915442916 C Bi:6:003:2 0 13 = 55534253 af0e0000 00000000 01
ffff9067b1774780 915442931 S Bo:6:003:1 -115 31 = 55534243 b00e0000 12000000 80030603 00000012 00000000 00000000 000000
ffff9067b1774780 915442981 C Bo:6:003:1 0 31 >
ffff90673f63ae40 915442994 S Bi:6:003:2 -115 18 <
ffff90673f63ae40 915443529 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 915443544 S Bi:6:003:2 -115 13 <
ffff9067b1774780 915443723 C Bi:6:003:2 0 13 = 55534253 b00e0000 00000000 00
ffff9067b13ada80 916806702 C Ii:6:002:1 0:2048 1 = 08
ffff9067b13ada80 916806724 S Ii:6:002:1 -115:2048 1 <
ffff90673f63ae40 916806749 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff90673f63ae40 916806785 C Ci:6:002:0 0 4 = 01010100
ffff90673f63ae40 916806814 S Co:6:002:0 s 23 01 0010 0003 0000 0
ffff90673f63ae40 916806830 C Co:6:002:0 0 0
ffff90673f63ae40 916806840 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff90673f63ae40 916806880 C Ci:6:002:0 0 4 = 01010000
ffff90673f63ae40 916833447 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff90673f63ae40 916833487 C Ci:6:002:0 0 4 = 01010000
ffff90673f63ae40 916860454 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff90673f63ae40 916860503 C Ci:6:002:0 0 4 = 01010000
ffff90673f63ae40 916887458 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff90673f63ae40 916887511 C Ci:6:002:0 0 4 = 01010000
ffff90673f63ae40 916914456 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff90673f63ae40 916914505 C Ci:6:002:0 0 4 = 01010000
ffff90673f63ae40 916914766 S Co:6:002:0 s 23 03 0004 0003 0000 0
ffff90673f63ae40 916914821 C Co:6:002:0 0 0
ffff90673f63ae40 916926425 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff90673f63ae40 916926469 C Ci:6:002:0 0 4 = 11050000
ffff90673f63ae40 916938456 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff90673f63ae40 916938510 C Ci:6:002:0 0 4 = 03051000
ffff90673f63ae40 916938525 S Co:6:002:0 s 23 01 0014 0003 0000 0
ffff90673f63ae40 916938544 C Co:6:002:0 0 0
ffff90673f63ae40 916990492 S Ci:6:000:0 s 80 06 0100 0000 0040 64 <
ffff90673f63ae40 916992792 C Ci:6:000:0 0 18 = 12011002 00000040 51096616 01000102 0301
ffff90673f63ae40 916992827 S Co:6:002:0 s 23 03 0004 0003 0000 0
ffff90673f63ae40 916992858 C Co:6:002:0 0 0
ffff90673f63ae40 917004425 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff90673f63ae40 917004461 C Ci:6:002:0 0 4 = 11050000
ffff90673f63ae40 917016442 S Ci:6:002:0 s a3 00 0000 0003 0004 4 <
ffff90673f63ae40 917016482 C Ci:6:002:0 0 4 = 03051000
ffff90673f63ae40 917016491 S Co:6:002:0 s 23 01 0014 0003 0000 0
ffff90673f63ae40 917016523 C Co:6:002:0 0 0
ffff90673f63ae40 917080468 S Ci:6:006:0 s 80 06 0100 0000 0012 18 <
ffff90673f63ae40 917082747 C Ci:6:006:0 0 18 = 12011002 00000040 51096616 01000102 0301
ffff90673f63ae40 917082784 S Ci:6:006:0 s 80 06 0f00 0000 0005 5 <
ffff90673f63ae40 917083314 C Ci:6:006:0 0 5 = 050f1600 02
ffff90673f63ae40 917083332 S Ci:6:006:0 s 80 06 0f00 0000 0016 22 <
ffff90673f63ae40 917083857 C Ci:6:006:0 0 22 = 050f1600 02071002 06000000 0a100300 0e00020a ff07
ffff90673f63ae40 917083881 S Ci:6:006:0 s 80 06 0200 0000 0009 9 <
ffff90673f63ae40 917086348 C Ci:6:006:0 0 9 = 09022000 01010080 fa
ffff90673f63ae40 917086410 S Ci:6:006:0 s 80 06 0200 0000 0020 32 <
ffff90673f63ae40 917088862 C Ci:6:006:0 0 32 = 09022000 01010080 fa090400 00020806 50000705 81020002 ff070502 020002ff
ffff90673f63aa80 917088910 S Ci:6:006:0 s 80 06 0300 0000 00ff 255 <
ffff90673f63aa80 917091339 C Ci:6:006:0 0 4 = 04030904
ffff90673f63aa80 917091392 S Ci:6:006:0 s 80 06 0302 0409 00ff 255 <
ffff90673f63aa80 917093802 C Ci:6:006:0 0 34 = 22034400 61007400 61005400 72006100 76006500 6c006500 72002000 33002e00
ffff90673f63aa80 917093855 S Ci:6:006:0 s 80 06 0301 0409 00ff 255 <
ffff90673f63aa80 917096300 C Ci:6:006:0 0 18 = 12034b00 69006e00 67007300 74006f00 6e00
ffff90673f63aa80 917096392 S Ci:6:006:0 s 80 06 0303 0409 00ff 255 <
ffff90673f63aa80 917098782 C Ci:6:006:0 0 50 = 32033600 30004100 34003400 43003400 32003500 36003800 43004600 46003700
ffff90673f63ac00 917102108 S Co:6:006:0 s 00 09 0001 0000 0000 0
ffff90673f63ac00 917102464 C Co:6:006:0 0 0
ffff9067b1774780 917486523 S Bo:6:003:1 -115 31 = 55534243 b10e0000 00000000 00030600 00000000 00000000 00000000 000000
ffff9067b1774780 917486621 C Bo:6:003:1 0 31 >
ffff9067b1774780 917486632 S Bi:6:003:2 -115 13 <
ffff9067b1774780 917487000 C Bi:6:003:2 0 13 = 55534253 b10e0000 00000000 01
ffff9067b1774780 917487017 S Bo:6:003:1 -115 31 = 55534243 b20e0000 12000000 80030603 00000012 00000000 00000000 000000
ffff9067b1774780 917487087 C Bo:6:003:1 0 31 >
ffff90673f63a480 917487098 S Bi:6:003:2 -115 18 <
ffff90673f63a480 917487704 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 917487719 S Bi:6:003:2 -115 13 <
ffff9067b1774780 917487834 C Bi:6:003:2 0 13 = 55534253 b20e0000 00000000 00
ffff9067b1774780 917487931 S Bo:6:003:1 -115 31 = 55534243 b30e0000 00000000 00020600 00000000 00000000 00000000 000000
ffff9067b1774780 917488002 C Bo:6:003:1 0 31 >
ffff9067b1774780 917488041 S Bi:6:003:2 -115 13 <
ffff9067b1774780 917488394 C Bi:6:003:2 0 13 = 55534253 b30e0000 00000000 01
ffff9067b1774780 917488452 S Bo:6:003:1 -115 31 = 55534243 b40e0000 12000000 80020603 00000012 00000000 00000000 000000
ffff9067b1774780 917488537 C Bo:6:003:1 0 31 >
ffff90673f6a79c0 917488584 S Bi:6:003:2 -115 18 <
ffff90673f6a79c0 917489046 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 917489077 S Bi:6:003:2 -115 13 <
ffff9067b1774780 917489184 C Bi:6:003:2 0 13 = 55534253 b40e0000 00000000 00
ffff9067b1774780 917489249 S Bo:6:003:1 -115 31 = 55534243 b50e0000 00000000 00010600 00000000 00000000 00000000 000000
ffff9067b1774780 917489318 C Bo:6:003:1 0 31 >
ffff9067b1774780 917489370 S Bi:6:003:2 -115 13 <
ffff9067b1774780 917489688 C Bi:6:003:2 0 13 = 55534253 b50e0000 00000000 01
ffff9067b1774780 917489742 S Bo:6:003:1 -115 31 = 55534243 b60e0000 12000000 80010603 00000012 00000000 00000000 000000
ffff9067b1774780 917489816 C Bo:6:003:1 0 31 >
ffff90673f6a79c0 917489864 S Bi:6:003:2 -115 18 <
ffff90673f6a79c0 917490309 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 917490384 S Bi:6:003:2 -115 13 <
ffff9067b1774780 917490442 C Bi:6:003:2 0 13 = 55534253 b60e0000 00000000 00
ffff9067b1774780 917490563 S Bo:6:003:1 -115 31 = 55534243 b70e0000 00000000 00000600 00000000 00000000 00000000 000000
ffff9067b1774780 917490616 C Bo:6:003:1 0 31 >
ffff9067b1774780 917490634 S Bi:6:003:2 -115 13 <
ffff9067b1774780 917490998 C Bi:6:003:2 0 13 = 55534253 b70e0000 00000000 01
ffff9067b1774780 917491009 S Bo:6:003:1 -115 31 = 55534243 b80e0000 12000000 80000603 00000012 00000000 00000000 000000
ffff9067b1774780 917491047 C Bo:6:003:1 0 31 >
ffff90673f46fa80 917491091 S Bi:6:003:2 -115 18 <
ffff90673f46fa80 917491615 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 917491668 S Bi:6:003:2 -115 13 <
ffff9067b1774780 917491735 C Bi:6:003:2 0 13 = 55534253 b80e0000 00000000 00
ffff9067b1774b40 917620842 C Ii:6:004:1 0:2 8 = 00001100 04000000
ffff9067b1774b40 917620910 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917628842 C Ii:6:004:1 0:2 8 = 00000d00 04000000
ffff9067b1774b40 917628898 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917634792 C Ii:6:004:1 0:2 8 = 00000d00 08000000
ffff9067b1774b40 917634846 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917642806 C Ii:6:004:1 0:2 8 = 00000b00 08000000
ffff9067b1774b40 917642853 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917650842 C Ii:6:004:1 0:2 8 = 00000800 08000000
ffff9067b1774b40 917650894 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917658803 C Ii:6:004:1 0:2 8 = 00000500 09000000
ffff9067b1774b40 917658849 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917666822 C Ii:6:004:1 0:2 8 = 00000400 0b000000
ffff9067b1774b40 917666878 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917674808 C Ii:6:004:1 0:2 8 = 00000100 0c000000
ffff9067b1774b40 917674861 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917682842 C Ii:6:004:1 0:2 8 = 00000100 0e000000
ffff9067b1774b40 917682904 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917690809 C Ii:6:004:1 0:2 8 = 00000000 0f000000
ffff9067b1774b40 917690859 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917698808 C Ii:6:004:1 0:2 8 = 00000000 10000000
ffff9067b1774b40 917698851 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917706803 C Ii:6:004:1 0:2 8 = 00000000 10000000
ffff9067b1774b40 917706829 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917714831 C Ii:6:004:1 0:2 8 = 00000000 0e000000
ffff9067b1774b40 917714862 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917720831 C Ii:6:004:1 0:2 8 = 00000000 0b000000
ffff9067b1774b40 917720860 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917728835 C Ii:6:004:1 0:2 8 = 00000000 0a000000
ffff9067b1774b40 917728873 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917736839 C Ii:6:004:1 0:2 8 = 00000000 04000000
ffff9067b1774b40 917736876 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917744841 C Ii:6:004:1 0:2 8 = 00000000 02000000
ffff9067b1774b40 917744883 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917798843 C Ii:6:004:1 0:2 8 = 00000000 01000000
ffff9067b1774b40 917798891 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917806843 C Ii:6:004:1 0:2 8 = 00000000 02000000
ffff9067b1774b40 917806889 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917814839 C Ii:6:004:1 0:2 8 = 00000200 03000000
ffff9067b1774b40 917814879 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917822806 C Ii:6:004:1 0:2 8 = 00000300 04000000
ffff9067b1774b40 917822845 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917830847 C Ii:6:004:1 0:2 8 = 00000400 04000000
ffff9067b1774b40 917830892 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917838847 C Ii:6:004:1 0:2 8 = 00000300 04000000
ffff9067b1774b40 917838899 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917846847 C Ii:6:004:1 0:2 8 = 00000300 04000000
ffff9067b1774b40 917846898 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917854845 C Ii:6:004:1 0:2 8 = 00000300 03000000
ffff9067b1774b40 917854901 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917862852 C Ii:6:004:1 0:2 8 = 00000100 03000000
ffff9067b1774b40 917862900 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917870849 C Ii:6:004:1 0:2 8 = 00000200 02000000
ffff9067b1774b40 917870907 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917878842 C Ii:6:004:1 0:2 8 = 00000100 02000000
ffff9067b1774b40 917878896 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917884837 C Ii:6:004:1 0:2 8 = 00000100 01000000
ffff9067b1774b40 917884897 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917892808 C Ii:6:004:1 0:2 8 = 00000100 01000000
ffff9067b1774b40 917892859 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917900841 C Ii:6:004:1 0:2 8 = 00000200 02000000
ffff9067b1774b40 917900902 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917908842 C Ii:6:004:1 0:2 8 = 00000100 03000000
ffff9067b1774b40 917908906 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917916836 C Ii:6:004:1 0:2 8 = 00000300 05000000
ffff9067b1774b40 917916898 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917924837 C Ii:6:004:1 0:2 8 = 00000500 07000000
ffff9067b1774b40 917924890 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917932841 C Ii:6:004:1 0:2 8 = 00000400 08000000
ffff9067b1774b40 917932894 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917940836 C Ii:6:004:1 0:2 8 = 00000700 0a000000
ffff9067b1774b40 917940897 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917948841 C Ii:6:004:1 0:2 8 = 00000800 09000000
ffff9067b1774b40 917948902 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917956840 C Ii:6:004:1 0:2 8 = 00000800 09000000
ffff9067b1774b40 917956903 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917962841 C Ii:6:004:1 0:2 8 = 00000900 08000000
ffff9067b1774b40 917962902 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917970802 C Ii:6:004:1 0:2 8 = 00000800 05000000
ffff9067b1774b40 917970852 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917978839 C Ii:6:004:1 0:2 8 = 00000700 03000000
ffff9067b1774b40 917978893 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917986802 C Ii:6:004:1 0:2 8 = 00000500 02000000
ffff9067b1774b40 917986852 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 917994839 C Ii:6:004:1 0:2 8 = 00000400 01000000
ffff9067b1774b40 917994900 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 918002801 C Ii:6:004:1 0:2 8 = 00000100 00000000
ffff9067b1774b40 918002843 S Ii:6:004:1 -115:2 8 <
ffff90673f63a3c0 918126468 S Ci:6:006:0 s a1 fe 0000 0000 0001 1 <
ffff90673f63a3c0 918126843 C Ci:6:006:0 0 1 = 00
ffff90673f63a3c0 918127110 S Bo:6:006:2 -115 31 = 55534243 01000000 24000000 80000612 00000024 00000000 00000000 000000
ffff90673f63a3c0 918127174 C Bo:6:006:2 0 31 >
ffff9067adb13480 918127223 S Bi:6:006:1 -115 36 <
ffff9067b1774b40 918174802 C Ii:6:004:1 0:2 8 = 00000000 fdff0000
ffff9067b1774b40 918174859 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 918182827 C Ii:6:004:1 0:2 8 = 00000000 fcff0000
ffff9067b1774b40 918182862 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 918190831 C Ii:6:004:1 0:2 8 = 00000000 feff0000
ffff9067b1774b40 918190876 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 918244840 C Ii:6:004:1 0:2 8 = 00000000 ffff0000
ffff9067b1774b40 918244898 S Ii:6:004:1 -115:2 8 <
ffff9067adb13480 918277656 C Bi:6:006:1 0 36 = 00800602 f8000000 4b696e67 73746f6e 44617461 54726176 656c6572 20332e30
ffff90673f63a3c0 918277717 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918277930 C Bi:6:006:1 0 13 = 55534253 01000000 00000000 00
ffff90673f63a3c0 918278985 S Bo:6:006:2 -115 31 = 55534243 02000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918279018 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918279061 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918279077 C Bi:6:006:1 0 13 = 55534253 02000000 00000000 00
ffff90673f63a3c0 918279141 S Bo:6:006:2 -115 31 = 55534243 03000000 08000000 80000a25 00000000 00000000 00000000 000000
ffff90673f63a3c0 918279157 C Bo:6:006:2 0 31 >
ffff9067adb133c0 918279755 S Bi:6:006:1 -115 8 <
ffff9067adb133c0 918279776 C Bi:6:006:1 0 8 = 039a33f3 00000200
ffff90673f63a3c0 918279793 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918279806 C Bi:6:006:1 0 13 = 55534253 03000000 00000000 00
ffff90673f63a3c0 918279834 S Bo:6:006:2 -115 31 = 55534243 04000000 c0000000 8000061a 003f00c0 00000000 00000000 000000
ffff90673f63a3c0 918279845 C Bo:6:006:2 0 31 >
ffff9067adb13840 918279863 S Bi:6:006:1 -115 192 <
ffff9067adb13840 918280124 C Bi:6:006:1 -121 80 = 4f000000 051ef000 00000200 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918280131 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918280368 C Bi:6:006:1 0 13 = 55534253 04000000 70000000 00
ffff90673f63a3c0 918280394 S Bo:6:006:2 -115 31 = 55534243 05000000 c0000000 8000061a 003f00c0 00000000 00000000 000000
ffff90673f63a3c0 918280406 C Bo:6:006:2 0 31 >
ffff90673f63a780 918280411 S Bi:6:006:1 -115 192 <
ffff90673f63a780 918280687 C Bi:6:006:1 -121 80 = 4f000000 051ef000 00000200 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918280693 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918280927 C Bi:6:006:1 0 13 = 55534253 05000000 70000000 00
ffff90673f63a3c0 918281292 S Bo:6:006:2 -115 31 = 55534243 06000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918281305 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918281308 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918281344 C Bi:6:006:1 0 13 = 55534253 06000000 00000000 00
ffff90673f63a3c0 918281365 S Bo:6:006:2 -115 31 = 55534243 07000000 00000000 0000061e 00000001 00000000 00000000 000000
ffff90673f63a3c0 918281384 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918281387 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918281864 C Bi:6:006:1 0 13 = 55534253 07000000 00000000 01
ffff90673f63a3c0 918281869 S Bo:6:006:2 -115 31 = 55534243 08000000 60000000 80000603 00000060 00000000 00000000 000000
ffff90673f63a3c0 918281903 C Bo:6:006:2 0 31 >
ffff90673f6a7480 918281907 S Bi:6:006:1 -115 96 <
ffff90673f6a7480 918282156 C Bi:6:006:1 0 96 = 70000500 0000000a 00000000 20000000 00000a7f 00000000 10000ae0 00400000
ffff90673f63a3c0 918282173 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918282399 C Bi:6:006:1 0 13 = 55534253 08000000 00000000 00
ffff90673f63a3c0 918282492 S Bo:6:006:2 -115 31 = 55534243 09000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918282541 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918282565 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918282607 C Bi:6:006:1 0 13 = 55534253 09000000 00000000 00
ffff90673f63a3c0 918282652 S Bo:6:006:2 -115 31 = 55534243 0a000000 08000000 80000a25 00000000 00000000 00000000 000000
ffff90673f63a3c0 918282664 C Bo:6:006:2 0 31 >
ffff90673f6a7480 918282668 S Bi:6:006:1 -115 8 <
ffff90673f6a7480 918282937 C Bi:6:006:1 0 8 = 039a33f3 00000200
ffff90673f63a3c0 918282944 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918283163 C Bi:6:006:1 0 13 = 55534253 0a000000 00000000 00
ffff90673f63a3c0 918283180 S Bo:6:006:2 -115 31 = 55534243 0b000000 c0000000 8000061a 003f00c0 00000000 00000000 000000
ffff90673f63a3c0 918283193 C Bo:6:006:2 0 31 >
ffff90673f63a540 918283197 S Bi:6:006:1 -115 192 <
ffff90673f63a540 918283477 C Bi:6:006:1 -121 80 = 4f000000 051ef000 00000200 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918283483 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918283700 C Bi:6:006:1 0 13 = 55534253 0b000000 70000000 00
ffff90673f63a3c0 918283719 S Bo:6:006:2 -115 31 = 55534243 0c000000 c0000000 8000061a 003f00c0 00000000 00000000 000000
ffff90673f63a3c0 918283742 C Bo:6:006:2 0 31 >
ffff90673f46f840 918283750 S Bi:6:006:1 -115 192 <
ffff90673f46f840 918284009 C Bi:6:006:1 -121 80 = 4f000000 051ef000 00000200 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918284020 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918284257 C Bi:6:006:1 0 13 = 55534253 0c000000 70000000 00
ffff90673f63a3c0 918284291 S Bo:6:006:2 -115 31 = 55534243 0d000000 00080000 80000a28 00000000 00000004 00000000 000000
ffff90673f63a3c0 918284308 C Bo:6:006:2 0 31 >
ffff90673f46f000 918284315 S Bi:6:006:1 -115 2048 <
ffff90673f46f000 918285776 C Bi:6:006:1 0 2048 = fab80000 8ed0bc00 7c8bf450 07501ffb fcbf0006 b90001f3 a5ea1e06 0000bebe
ffff90673f63a3c0 918285798 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918285824 C Bi:6:006:1 0 13 = 55534253 0d000000 00000000 00
ffff90673f63a3c0 918285856 S Bo:6:006:2 -115 31 = 55534243 0e000000 00080000 80000a28 00000000 04000004 00000000 000000
ffff90673f63a3c0 918285867 C Bo:6:006:2 0 31 >
ffff9067adb13480 918285916 S Bi:6:006:1 -115 2048 <
ffff9067adb13480 918285962 C Bi:6:006:1 0 2048 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918285980 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918286004 C Bi:6:006:1 0 13 = 55534253 0e000000 00000000 00
ffff90673f63a3c0 918286217 S Bo:6:006:2 -115 31 = 55534243 0f000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918286231 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918286233 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918286269 C Bi:6:006:1 0 13 = 55534253 0f000000 00000000 00
ffff90673f63a3c0 918286278 S Bo:6:006:2 -115 31 = 55534243 10000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918286309 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918286312 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918286349 C Bi:6:006:1 0 13 = 55534253 10000000 00000000 00
ffff90673f63a3c0 918286360 S Bo:6:006:2 -115 31 = 55534243 11000000 08000000 80000a25 00000000 00000000 00000000 000000
ffff90673f63a3c0 918286389 C Bo:6:006:2 0 31 >
ffff9067adb13480 918286392 S Bi:6:006:1 -115 8 <
ffff9067adb13480 918286670 C Bi:6:006:1 0 8 = 039a33f3 00000200
ffff90673f63a3c0 918286674 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918286885 C Bi:6:006:1 0 13 = 55534253 11000000 00000000 00
ffff90673f63a3c0 918286898 S Bo:6:006:2 -115 31 = 55534243 12000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918286926 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918286928 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918286965 C Bi:6:006:1 0 13 = 55534253 12000000 00000000 00
ffff90673f63a3c0 918286971 S Bo:6:006:2 -115 31 = 55534243 13000000 c0000000 8000061a 003f00c0 00000000 00000000 000000
ffff90673f63a3c0 918287005 C Bo:6:006:2 0 31 >
ffff9067adb13480 918287008 S Bi:6:006:1 -115 192 <
ffff9067adb13480 918287259 C Bi:6:006:1 -121 80 = 4f000000 051ef000 00000200 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918287263 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918287486 C Bi:6:006:1 0 13 = 55534253 13000000 70000000 00
ffff90673f63a3c0 918287540 S Bo:6:006:2 -115 31 = 55534243 14000000 c0000000 8000061a 003f00c0 00000000 00000000 000000
ffff90673f63a3c0 918287551 C Bo:6:006:2 0 31 >
ffff9067adb13480 918287568 S Bi:6:006:1 -115 192 <
ffff9067adb13480 918287822 C Bi:6:006:1 -121 80 = 4f000000 051ef000 00000200 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918287840 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918288070 C Bi:6:006:1 0 13 = 55534253 14000000 70000000 00
ffff90673f63a3c0 918288616 S Bo:6:006:2 -115 31 = 55534243 15000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918288639 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918288674 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918288682 C Bi:6:006:1 0 13 = 55534253 15000000 00000000 00
ffff90673f63a3c0 918288813 S Bo:6:006:2 -115 31 = 55534243 16000000 00100000 80000a28 00039a33 00000008 00000000 000000
ffff90673f63a3c0 918288834 C Bo:6:006:2 0 31 >
ffff9067adb13480 918288871 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918290341 C Bi:6:006:1 0 4096 = 4c776f62 69e41245 64d22d08 ae9db7b8 01c4e6fb 8b029833 8e4a9f4c 30a097df
ffff90673f63a3c0 918290390 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918290413 C Bi:6:006:1 0 13 = 55534253 16000000 00000000 00
ffff90673f63a3c0 918290517 S Bo:6:006:2 -115 31 = 55534243 17000000 00100000 80000a28 00039a33 e0000008 00000000 000000
ffff90673f63a3c0 918290549 C Bo:6:006:2 0 31 >
ffff9067adb13480 918290552 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918291238 C Bi:6:006:1 0 4096 = 85bfff3d 1c873fc8 99903146 ddceb7e5 10fe7570 715c5ff7 9343c2c9 c6ff195c
ffff90673f63a3c0 918291241 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918291256 C Bi:6:006:1 0 13 = 55534253 17000000 00000000 00
ffff90673f63a3c0 918291275 S Bo:6:006:2 -115 31 = 55534243 18000000 00100000 80000a28 00000000 00000008 00000000 000000
ffff90673f63a3c0 918291297 C Bo:6:006:2 0 31 >
ffff9067adb13480 918291300 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918292777 C Bi:6:006:1 0 4096 = fab80000 8ed0bc00 7c8bf450 07501ffb fcbf0006 b90001f3 a5ea1e06 0000bebe
ffff90673f63a3c0 918292781 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918292808 C Bi:6:006:1 0 13 = 55534253 18000000 00000000 00
ffff90673f63a3c0 918292838 S Bo:6:006:2 -115 31 = 55534243 19000000 00100000 80000a28 00000000 08000008 00000000 000000
ffff90673f63a3c0 918292849 C Bo:6:006:2 0 31 >
ffff9067adb13480 918292851 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918293515 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918293534 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918293555 C Bi:6:006:1 0 13 = 55534253 19000000 00000000 00
ffff90673f63a3c0 918293599 S Bo:6:006:2 -115 31 = 55534243 1a000000 00020000 80000a28 00039a33 f0000001 00000000 000000
ffff90673f63a3c0 918293610 C Bo:6:006:2 0 31 >
ffff9067adb13480 918293626 S Bi:6:006:1 -115 512 <
ffff9067adb13480 918295043 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff90673f63a3c0 918295058 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918295082 C Bi:6:006:1 0 13 = 55534253 1a000000 00000000 00
ffff90673f63a3c0 918295102 S Bo:6:006:2 -115 31 = 55534243 1b000000 00020000 80000a28 00039a33 f1000001 00000000 000000
ffff90673f63a3c0 918295122 C Bo:6:006:2 0 31 >
ffff9067adb13480 918295125 S Bi:6:006:1 -115 512 <
ffff9067adb13480 918295726 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff90673f63a3c0 918295757 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918295764 C Bi:6:006:1 0 13 = 55534253 1b000000 00000000 00
ffff90673f63a3c0 918295777 S Bo:6:006:2 -115 31 = 55534243 1c000000 00020000 80000a28 00039a33 f2000001 00000000 000000
ffff90673f63a3c0 918295805 C Bo:6:006:2 0 31 >
ffff9067adb13480 918295823 S Bi:6:006:1 -115 512 <
ffff9067adb13480 918296383 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff90673f63a3c0 918296394 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918296436 C Bi:6:006:1 0 13 = 55534253 1c000000 00000000 00
ffff90673f63a3c0 918296469 S Bo:6:006:2 -115 31 = 55534243 1d000000 00020000 80000a28 00039a33 f3000001 00000000 000000
ffff90673f63a3c0 918296479 C Bo:6:006:2 0 31 >
ffff9067adb13480 918296497 S Bi:6:006:1 -115 512 <
ffff9067adb13480 918297093 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff90673f63a3c0 918297113 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918297131 C Bi:6:006:1 0 13 = 55534253 1d000000 00000000 00
ffff90673f63a3c0 918297162 S Bo:6:006:2 -115 31 = 55534243 1e000000 00100000 80000a28 00039a32 f0000008 00000000 000000
ffff90673f63a3c0 918297174 C Bo:6:006:2 0 31 >
ffff9067adb13480 918297178 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918297851 C Bi:6:006:1 0 4096 = 8add6e3c 2432a9cf 357c4940 a360e0ab a20f802f 1d321990 325a7363 73ab2a85
ffff90673f63a3c0 918297856 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918297880 C Bi:6:006:1 0 13 = 55534253 1e000000 00000000 00
ffff90673f63a3c0 918297936 S Bo:6:006:2 -115 31 = 55534243 1f000000 00100000 80000a28 00039a33 b0000008 00000000 000000
ffff90673f63a3c0 918297948 C Bo:6:006:2 0 31 >
ffff9067adb13480 918297968 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918298628 C Bi:6:006:1 0 4096 = fd53643f 276dee14 bca4e177 d85ab32c 5a52be3a 75d14c46 8b06f53f 39e85942
ffff90673f63a3c0 918298666 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918298674 C Bi:6:006:1 0 13 = 55534253 1f000000 00000000 00
ffff90673f63a3c0 918298740 S Bo:6:006:2 -115 31 = 55534243 20000000 00100000 80000a28 00039a32 60000008 00000000 000000
ffff90673f63a3c0 918298763 C Bo:6:006:2 0 31 >
ffff9067adb13480 918298785 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918299446 C Bi:6:006:1 0 4096 = 8841fcc0 f141f63c 74b09b13 582943df 1f84ee19 ca7625f1 1e71698a d8e9723f
ffff90673f63a3c0 918299485 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918299506 C Bi:6:006:1 0 13 = 55534253 20000000 00000000 00
ffff90673f63a3c0 918299567 S Bo:6:006:2 -115 31 = 55534243 21000000 00100000 80000a28 00039a31 a0000008 00000000 000000
ffff90673f63a3c0 918299581 C Bo:6:006:2 0 31 >
ffff9067adb13480 918299584 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918300233 C Bi:6:006:1 0 4096 = e83e664b f50389eb 1b408ee4 65d0216d 44c3e132 eb58ebd2 3b8f99e3 f0cf5765
ffff90673f63a3c0 918300238 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918300274 C Bi:6:006:1 0 13 = 55534253 21000000 00000000 00
ffff90673f63a3c0 918300296 S Bo:6:006:2 -115 31 = 55534243 22000000 00100000 80000a28 00039a31 50000008 00000000 000000
ffff90673f63a3c0 918300314 C Bo:6:006:2 0 31 >
ffff9067adb13480 918300318 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918300997 C Bi:6:006:1 0 4096 = f92826b9 0f9982fd 742b040f f35d44f5 d590c27e 669f6b9f c7c8491f c9a8cfa7
ffff90673f63a3c0 918301003 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918301027 C Bi:6:006:1 0 13 = 55534253 22000000 00000000 00
ffff90673f63a3c0 918301085 S Bo:6:006:2 -115 31 = 55534243 23000000 00100000 80000a28 00039a31 10000008 00000000 000000
ffff90673f63a3c0 918301099 C Bo:6:006:2 0 31 >
ffff9067adb13480 918301107 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918301802 C Bi:6:006:1 0 4096 = 2dbbe381 8f0e583d 890d82b4 4d73de7e 7b0a3609 2658d90d ba959458 1aed842c
ffff90673f63a3c0 918301840 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918301848 C Bi:6:006:1 0 13 = 55534253 23000000 00000000 00
ffff90673f63a3c0 918301896 S Bo:6:006:2 -115 31 = 55534243 24000000 00100000 80000a28 00039a30 60000008 00000000 000000
ffff90673f63a3c0 918301909 C Bo:6:006:2 0 31 >
ffff9067adb13480 918301916 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918302619 C Bi:6:006:1 0 4096 = 2ec65134 a721e3ca 2a1f8931 2f630dea 7a552159 fab8e3da 7d6c7ef9 2c814b72
ffff90673f63a3c0 918302674 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918302692 C Bi:6:006:1 0 13 = 55534253 24000000 00000000 00
ffff90673f63a3c0 918302776 S Bo:6:006:2 -115 31 = 55534243 25000000 00100000 80000a28 00039a30 20000008 00000000 000000
ffff90673f63a3c0 918302831 C Bo:6:006:2 0 31 >
ffff9067adb13480 918302851 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918303494 C Bi:6:006:1 0 4096 = fa0cab42 80b853f2 b7ad415c d4c255af 9c7a1f34 b93c10a9 f257457b 2cc3df29
ffff90673f63a3c0 918303524 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918303549 C Bi:6:006:1 0 13 = 55534253 25000000 00000000 00
ffff90673f63a3c0 918303672 S Bo:6:006:2 -115 31 = 55534243 26000000 00100000 80000a28 00039a30 10000008 00000000 000000
ffff90673f63a3c0 918303688 C Bo:6:006:2 0 31 >
ffff9067adb13480 918303698 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918304361 C Bi:6:006:1 0 4096 = 560e4b62 84950a4a a3c3ec6c 4ac0e404 a0220047 a97c1a1c 00758e94 58b20f2b
ffff90673f63a3c0 918304372 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918304401 C Bi:6:006:1 0 13 = 55534253 26000000 00000000 00
ffff90673f63a3c0 918304451 S Bo:6:006:2 -115 31 = 55534243 27000000 00100000 80000a28 00039a30 38000008 00000000 000000
ffff90673f63a3c0 918304465 C Bo:6:006:2 0 31 >
ffff9067adb13480 918304473 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918305157 C Bi:6:006:1 0 4096 = c87d4091 ebf570d5 61fb8e44 21a5418f a14adebf f11a4627 b196f961 216317b1
ffff90673f63a3c0 918305164 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918305201 C Bi:6:006:1 0 13 = 55534253 27000000 00000000 00
ffff90673f63a3c0 918305239 S Bo:6:006:2 -115 31 = 55534243 28000000 00100000 80000a28 00039a27 e0000008 00000000 000000
ffff90673f63a3c0 918305253 C Bo:6:006:2 0 31 >
ffff9067adb13480 918305258 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918305926 C Bi:6:006:1 0 4096 = 4d758507 4874e7ab 733c4eaf a1b9c050 e0c0867e 61b66631 c080b58e 18107d17
ffff90673f63a3c0 918305933 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918305994 C Bi:6:006:1 0 13 = 55534253 28000000 00000000 00
ffff90673f63a3c0 918306116 S Bo:6:006:2 -115 31 = 55534243 29000000 00080000 80000a28 00039a33 e8000004 00000000 000000
ffff90673f63a3c0 918306178 C Bo:6:006:2 0 31 >
ffff9067adb13480 918306196 S Bi:6:006:1 -115 2048 <
ffff9067adb13480 918306833 C Bi:6:006:1 0 2048 = c71672c3 62f9e441 93556627 49f7d2f6 c1556d39 f35c478c b7303075 c5c622c8
ffff90673f63a3c0 918306871 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918306899 C Bi:6:006:1 0 13 = 55534253 29000000 00000000 00
ffff90673f63a3c0 918306943 S Bo:6:006:2 -115 31 = 55534243 2a000000 00020000 80000a28 00039a33 ec000001 00000000 000000
ffff90673f63a3c0 918306973 C Bo:6:006:2 0 31 >
ffff9067adb13480 918306996 S Bi:6:006:1 -115 512 <
ffff9067adb13480 918307621 C Bi:6:006:1 0 512 = 809f4d0a bc5667de 57467a8e 646e8913 8d0bae35 d0f85a27 97065766 1ac31874
ffff90673f63a3c0 918307660 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918307686 C Bi:6:006:1 0 13 = 55534253 2a000000 00000000 00
ffff90673f63a3c0 918307728 S Bo:6:006:2 -115 31 = 55534243 2b000000 00020000 80000a28 00039a33 ed000001 00000000 000000
ffff90673f63a3c0 918307781 C Bo:6:006:2 0 31 >
ffff9067adb13480 918307802 S Bi:6:006:1 -115 512 <
ffff9067adb13480 918308343 C Bi:6:006:1 0 512 = 97bc843d bb40e8dd 37106a17 8b97f602 c439d3b5 e4dfd162 bd8e6a42 09f83484
ffff90673f63a3c0 918308385 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918308412 C Bi:6:006:1 0 13 = 55534253 2b000000 00000000 00
ffff90673f63a3c0 918308483 S Bo:6:006:2 -115 31 = 55534243 2c000000 00020000 80000a28 00039a33 ee000001 00000000 000000
ffff90673f63a3c0 918308536 C Bo:6:006:2 0 31 >
ffff9067adb13480 918308557 S Bi:6:006:1 -115 512 <
ffff9067adb13480 918309109 C Bi:6:006:1 0 512 = 66f05e02 eb030290 ceab76c6 e426f4b0 6ed9b467 31e125f2 28e6c092 135d8134
ffff90673f63a3c0 918309172 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918309190 C Bi:6:006:1 0 13 = 55534253 2c000000 00000000 00
ffff90673f63a3c0 918309214 S Bo:6:006:2 -115 31 = 55534243 2d000000 00020000 80000a28 00039a33 ef000001 00000000 000000
ffff90673f63a3c0 918309262 C Bo:6:006:2 0 31 >
ffff9067adb13480 918309275 S Bi:6:006:1 -115 512 <
ffff9067adb13480 918309834 C Bi:6:006:1 0 512 = 3a43294b 77481b39 4cec9d97 da766318 358af9ff e8c5dfce 6bbfff83 e78d3f1c
ffff90673f63a3c0 918309880 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918309905 C Bi:6:006:1 0 13 = 55534253 2d000000 00000000 00
ffff90673f63a3c0 918310051 S Bo:6:006:2 -115 31 = 55534243 2e000000 00100000 80000a28 00000000 20000008 00000000 000000
ffff90673f63a3c0 918310065 C Bo:6:006:2 0 31 >
ffff9067adb13480 918310086 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918311565 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918311585 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918311604 C Bi:6:006:1 0 13 = 55534253 2e000000 00000000 00
ffff90673f63a3c0 918311689 S Bo:6:006:2 -115 31 = 55534243 2f000000 00100000 80000a28 00000000 40000008 00000000 000000
ffff90673f63a3c0 918311732 C Bo:6:006:2 0 31 >
ffff9067adb13480 918311755 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918312401 C Bi:6:006:1 0 4096 = eb589054 48524545 53202000 02204000 02000000 00f80000 40002000 40000000
ffff90673f63a3c0 918312415 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918312445 C Bi:6:006:1 0 13 = 55534253 2f000000 00000000 00
ffff90673f63a3c0 918312503 S Bo:6:006:2 -115 31 = 55534243 30000000 00100000 80000a28 00000000 80000008 00000000 000000
ffff90673f63a3c0 918312532 C Bo:6:006:2 0 31 >
ffff9067adb13480 918312535 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918313199 C Bi:6:006:1 0 4096 = f8ffffff ffffff0f ffffff0f 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918313204 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918313240 C Bi:6:006:1 0 13 = 55534253 30000000 00000000 00
ffff90673f63a3c0 918313274 S Bo:6:006:2 -115 31 = 55534243 31000000 00100000 80000a28 00000001 00000008 00000000 000000
ffff90673f63a3c0 918313292 C Bo:6:006:2 0 31 >
ffff9067adb13480 918313295 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918313960 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918313965 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918314019 C Bi:6:006:1 0 13 = 55534253 31000000 00000000 00
ffff90673f63a3c0 918314099 S Bo:6:006:2 -115 31 = 55534243 32000000 00100000 80000a28 00000002 00000008 00000000 000000
ffff90673f63a3c0 918314154 C Bo:6:006:2 0 31 >
ffff9067adb13480 918314168 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918314803 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918314839 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918314869 C Bi:6:006:1 0 13 = 55534253 32000000 00000000 00
ffff90673f63a3c0 918314926 S Bo:6:006:2 -115 31 = 55534243 33000000 00100000 80000a28 00000004 00000008 00000000 000000
ffff90673f63a3c0 918314963 C Bo:6:006:2 0 31 >
ffff9067adb13480 918314987 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918315628 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918315675 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918315704 C Bi:6:006:1 0 13 = 55534253 33000000 00000000 00
ffff90673f63a3c0 918315786 S Bo:6:006:2 -115 31 = 55534243 34000000 00100000 80000a28 00000008 00000008 00000000 000000
ffff90673f63a3c0 918315840 C Bo:6:006:2 0 31 >
ffff9067adb13480 918315854 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918316500 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918316547 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918316576 C Bi:6:006:1 0 13 = 55534253 34000000 00000000 00
ffff90673f63a3c0 918316661 S Bo:6:006:2 -115 31 = 55534243 35000000 00100000 80000a28 00000010 00000008 00000000 000000
ffff90673f63a3c0 918316694 C Bo:6:006:2 0 31 >
ffff9067adb13480 918316720 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918317379 C Bi:6:006:1 0 4096 = 00810000 00810100 00810200 00810300 00810400 00810c00 00810d00 00811800
ffff90673f63a3c0 918317441 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918317469 C Bi:6:006:1 0 13 = 55534253 35000000 00000000 00
ffff90673f63a3c0 918317527 S Bo:6:006:2 -115 31 = 55534243 36000000 00100000 80000a28 00000020 00000008 00000000 000000
ffff90673f63a3c0 918317555 C Bo:6:006:2 0 31 >
ffff9067adb13480 918317558 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918318242 C Bi:6:006:1 0 4096 = 00830000 00830100 00830200 00830300 00830400 00830c00 00830d00 00831800
ffff90673f63a3c0 918318247 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918318261 C Bi:6:006:1 0 13 = 55534253 36000000 00000000 00
ffff90673f63a3c0 918318307 S Bo:6:006:2 -115 31 = 55534243 37000000 00100000 80000a28 00000000 18000008 00000000 000000
ffff90673f63a3c0 918318320 C Bo:6:006:2 0 31 >
ffff9067adb13480 918318323 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918318997 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918319002 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918319056 C Bi:6:006:1 0 13 = 55534253 37000000 00000000 00
ffff90673f63a3c0 918319113 S Bo:6:006:2 -115 31 = 55534243 38000000 00100000 80000a28 00000000 38000008 00000000 000000
ffff90673f63a3c0 918319167 C Bo:6:006:2 0 31 >
ffff9067adb13480 918319181 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918319802 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918319849 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918319891 C Bi:6:006:1 0 13 = 55534253 38000000 00000000 00
ffff90673f63a3c0 918320009 S Bo:6:006:2 -115 31 = 55534243 39000000 00100000 80000a28 00000000 78000008 00000000 000000
ffff90673f63a3c0 918320022 C Bo:6:006:2 0 31 >
ffff9067adb13480 918320026 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918320715 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918320762 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918320791 C Bi:6:006:1 0 13 = 55534253 39000000 00000000 00
ffff90673f63a3c0 918320907 S Bo:6:006:2 -115 31 = 55534243 3a000000 00100000 80000a28 00000000 10000008 00000000 000000
ffff90673f63a3c0 918320923 C Bo:6:006:2 0 31 >
ffff9067adb13480 918320940 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918321591 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918321637 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918321667 C Bi:6:006:1 0 13 = 55534253 3a000000 00000000 00
ffff90673f63a3c0 918321692 S Bo:6:006:2 -115 31 = 55534243 3b000000 00200000 80000a28 00000000 28000010 00000000 000000
ffff90673f63a3c0 918321703 C Bo:6:006:2 0 31 >
ffff9067adb13480 918321725 S Bi:6:006:1 -115 8192 <
ffff9067adb13480 918322466 C Bi:6:006:1 0 8192 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918322513 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918322543 C Bi:6:006:1 0 13 = 55534253 3b000000 00000000 00
ffff90673f63a3c0 918322564 S Bo:6:006:2 -115 31 = 55534243 3c000000 00600000 80000a28 00000000 48000030 00000000 000000
ffff90673f63a3c0 918322576 C Bo:6:006:2 0 31 >
ffff9067adb13480 918322598 S Bi:6:006:1 -115 24576 <
ffff9067adb13480 918323676 C Bi:6:006:1 0 24576 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918323723 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918323766 C Bi:6:006:1 0 13 = 55534253 3c000000 00000000 00
ffff90673f63a3c0 918323826 S Bo:6:006:2 -115 31 = 55534243 3d000000 00f00000 80000a28 00000000 88000078 00000000 000000
ffff90673f63a3c0 918323846 C Bo:6:006:2 0 31 >
ffff9067adb13480 918323854 S Bi:6:006:1 -115 61440 <
ffff9067adb13480 918325873 C Bi:6:006:1 0 61440 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918325925 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918325945 C Bi:6:006:1 0 13 = 55534253 3d000000 00000000 00
ffff90673f63a3c0 918325976 S Bo:6:006:2 -115 31 = 55534243 3e000000 00100000 80000a28 00000001 08000008 00000000 000000
ffff90673f63a3c0 918325997 C Bo:6:006:2 0 31 >
ffff9067adb13480 918326008 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918326677 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918326739 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918326774 C Bi:6:006:1 0 13 = 55534253 3e000000 00000000 00
ffff90673f63a3c0 918326868 S Bo:6:006:2 -115 31 = 55534243 3f000000 00e00100 80000a28 00000001 100000f0 00000000 000000
ffff90673f63a3c0 918326932 C Bo:6:006:2 0 31 >
ffff9067adb13480 918326948 S Bi:6:006:1 -115 122880 <
ffff9067adb13480 918330498 C Bi:6:006:1 0 122880 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918330550 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918330570 C Bi:6:006:1 0 13 = 55534253 3f000000 00000000 00
ffff90673f63a3c0 918330884 S Bo:6:006:2 -115 31 = 55534243 40000000 00e00100 80000a28 00000002 080000f0 00000000 000000
ffff90673f63a3c0 918330924 C Bo:6:006:2 0 31 >
ffff9067adb13480 918330963 S Bi:6:006:1 -115 122880 <
ffff9067adb13480 918334586 C Bi:6:006:1 0 122880 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918334650 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918334672 C Bi:6:006:1 0 13 = 55534253 40000000 00000000 00
ffff90673f63a3c0 918334726 S Bo:6:006:2 -115 31 = 55534243 41000000 00200000 80000a28 00000002 f8000010 00000000 000000
ffff90673f63a3c0 918334754 C Bo:6:006:2 0 31 >
ffff9067adb13480 918334775 S Bi:6:006:1 -115 8192 <
ffff9067adb13480 918335552 C Bi:6:006:1 0 8192 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918335623 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918335644 C Bi:6:006:1 0 13 = 55534253 41000000 00000000 00
ffff90673f63a3c0 918335842 S Bo:6:006:2 -115 31 = 55534243 42000000 00e00100 80000a28 00000003 080000f0 00000000 000000
ffff90673f63a3c0 918335871 C Bo:6:006:2 0 31 >
ffff9067adb13480 918335904 S Bi:6:006:1 -115 122880 <
ffff9067adb13480 918339591 C Bi:6:006:1 0 122880 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918339641 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918339675 C Bi:6:006:1 0 13 = 55534253 42000000 00000000 00
ffff90673f63a3c0 918339749 S Bo:6:006:2 -115 31 = 55534243 43000000 00100000 80000a28 00000003 f8000008 00000000 000000
ffff90673f63a3c0 918339788 C Bo:6:006:2 0 31 >
ffff9067adb13480 918339801 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918339942 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918340016 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918340053 C Bi:6:006:1 0 13 = 55534253 43000000 00000000 00
ffff90673f63a3c0 918340585 S Bo:6:006:2 -115 31 = 55534243 44000000 00e00100 80000a28 00039a2e 000000f0 00000000 000000
ffff90673f63a3c0 918340610 C Bo:6:006:2 0 31 >
ffff9067adb13480 918340638 S Bi:6:006:1 -115 122880 <
ffff9067adb13480 918345187 C Bi:6:006:1 0 122880 = c77ccf3d b18bbc20 49a7c771 ab78fd88 353bdfd1 c2f59b72 ee1c7082 2c62c6f6
ffff90673f63a3c0 918345247 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918345287 C Bi:6:006:1 0 13 = 55534253 44000000 00000000 00
ffff90673f63a3c0 918345388 S Bo:6:006:2 -115 31 = 55534243 45000000 00200000 80000a28 00039a2e f0000010 00000000 000000
ffff90673f63a3c0 918345418 C Bo:6:006:2 0 31 >
ffff9067adb13480 918345439 S Bi:6:006:1 -115 8192 <
ffff9067adb13480 918346249 C Bi:6:006:1 0 8192 = 2d1e8c4f 0d6d6356 384be8bc b5cfb50e cdb3025f bcfecddd 4f80d2e5 eb6b3284
ffff90673f63a3c0 918346307 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918346349 C Bi:6:006:1 0 13 = 55534253 45000000 00000000 00
ffff90673f63a3c0 918346594 S Bo:6:006:2 -115 31 = 55534243 46000000 00e00100 80000a28 00039a2f 000000f0 00000000 000000
ffff90673f63a3c0 918346669 C Bo:6:006:2 0 31 >
ffff9067adb13480 918346708 S Bi:6:006:1 -115 122880 <
ffff9067adb13480 918350366 C Bi:6:006:1 0 122880 = aa1c9eea 8ab1cada d7bd4902 c36a01ea 724ec451 e5509315 ca093a94 9dcab358
ffff90673f63a3c0 918350423 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918350462 C Bi:6:006:1 0 13 = 55534253 46000000 00000000 00
ffff90673f63a3c0 918350550 S Bo:6:006:2 -115 31 = 55534243 47000000 00200000 80000a28 00039a2f f0000010 00000000 000000
ffff90673f63a3c0 918350596 C Bo:6:006:2 0 31 >
ffff9067adb13480 918350610 S Bi:6:006:1 -115 8192 <
ffff9067adb13480 918350883 C Bi:6:006:1 0 8192 = 6ddcf87d 817edc15 a527992a 77e6c620 1adc6410 77ef845a d44f3327 5ac54da3
ffff90673f63a3c0 918350937 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918350977 C Bi:6:006:1 0 13 = 55534253 47000000 00000000 00
ffff90673f63a3c0 918351473 S Bo:6:006:2 -115 31 = 55534243 48000000 00200000 80000a28 00039a30 00000010 00000000 000000
ffff90673f63a3c0 918351503 C Bo:6:006:2 0 31 >
ffff9067adb13480 918351528 S Bi:6:006:1 -115 8192 <
ffff9067adb13480 918351738 C Bi:6:006:1 0 8192 = c2bb8f02 e6227d01 1316d5d4 f8589196 3738834e 7c5c90cb 17868ef4 9eb65148
ffff90673f63a3c0 918351803 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918351826 C Bi:6:006:1 0 13 = 55534253 48000000 00000000 00
ffff90673f63a3c0 918351878 S Bo:6:006:2 -115 31 = 55534243 49000000 00100000 80000a28 00039a30 18000008 00000000 000000
ffff90673f63a3c0 918351897 C Bo:6:006:2 0 31 >
ffff9067adb13480 918351917 S Bi:6:006:1 -115 4096 <
ffff9067adb13480 918352629 C Bi:6:006:1 0 4096 = 3fa743bc cc91e902 d72dfafc 5ceb71ec 3a4d1ec0 d33baba5 e7323bf0 e6eec907
ffff90673f63a3c0 918352700 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918352723 C Bi:6:006:1 0 13 = 55534253 49000000 00000000 00
ffff90673f63a3c0 918352777 S Bo:6:006:2 -115 31 = 55534243 4a000000 00200000 80000a28 00039a30 28000010 00000000 000000
ffff90673f63a3c0 918352802 C Bo:6:006:2 0 31 >
ffff9067adb13480 918352816 S Bi:6:006:1 -115 8192 <
ffff9067adb13480 918353596 C Bi:6:006:1 0 8192 = 04529ef3 d43afe6d 30080475 edaff8de 8de2c099 9f5f242b 139da8b7 840879d0
ffff90673f63a3c0 918353667 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918353690 C Bi:6:006:1 0 13 = 55534253 4a000000 00000000 00
ffff90673f63a3c0 918353746 S Bo:6:006:2 -115 31 = 55534243 4b000000 00400000 80000a28 00039a30 40000020 00000000 000000
ffff90673f63a3c0 918353767 C Bo:6:006:2 0 31 >
ffff9067adb13480 918353785 S Bi:6:006:1 -115 16384 <
ffff9067adb13480 918354716 C Bi:6:006:1 0 16384 = 877035b3 af82887f 9c7bab44 76656b91 64401358 4211b9e0 165863b9 e453e6f3
ffff90673f63a3c0 918354799 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918354821 C Bi:6:006:1 0 13 = 55534253 4b000000 00000000 00
ffff90673f63a3c0 918354870 S Bo:6:006:2 -115 31 = 55534243 4c000000 00300100 80000a28 00039a30 68000098 00000000 000000
ffff90673f63a3c0 918354890 C Bo:6:006:2 0 31 >
ffff9067adb13480 918354909 S Bi:6:006:1 -115 77824 <
ffff9067adb13480 918357456 C Bi:6:006:1 0 77824 = 8a0e21ae 43b2cb2b 22eab472 19a923c0 5c814e49 7a105fc9 79938e83 79833c64
ffff90673f63a3c0 918357513 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918357537 C Bi:6:006:1 0 13 = 55534253 4c000000 00000000 00
ffff90673f63a3c0 918357647 S Bo:6:006:2 -115 31 = 55534243 4d000000 00200000 80000a28 00039a31 00000010 00000000 000000
ffff90673f63a3c0 918357664 C Bo:6:006:2 0 31 >
ffff9067adb13480 918357681 S Bi:6:006:1 -115 8192 <
ffff9067adb13480 918357928 C Bi:6:006:1 0 8192 = 315bbba0 afae98ec 6e222b9d ba2bc3bd 1a3f0717 1688e8e5 c30b4584 6f4f58ae
ffff90673f63a3c0 918357968 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918358001 C Bi:6:006:1 0 13 = 55534253 4d000000 00000000 00
ffff90673f63a3c0 918358035 S Bo:6:006:2 -115 31 = 55534243 4e000000 00700000 80000a28 00039a31 18000038 00000000 000000
ffff90673f63a3c0 918358052 C Bo:6:006:2 0 31 >
ffff9067adb13480 918358058 S Bi:6:006:1 -115 28672 <
ffff9067adb13480 918359430 C Bi:6:006:1 0 28672 = 3f13f01d 8a7e9065 563a44f7 0ef01f79 42c5d9cf 91d9e2cd 0f878026 67c1a426
ffff90673f63a3c0 918359514 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918359553 C Bi:6:006:1 0 13 = 55534253 4e000000 00000000 00
ffff90673f63a3c0 918359635 S Bo:6:006:2 -115 31 = 55534243 4f000000 00900000 80000a28 00039a31 58000048 00000000 000000
ffff90673f63a3c0 918359658 C Bo:6:006:2 0 31 >
ffff9067adb13480 918359688 S Bi:6:006:1 -115 36864 <
ffff9067adb13480 918361209 C Bi:6:006:1 0 36864 = 063e5961 7fe48390 3d427f2a 2dfd5e3a b589c6d3 1c72b289 dc3b9458 a96cf1e0
ffff90673f63a3c0 918361291 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918361317 C Bi:6:006:1 0 13 = 55534253 4f000000 00000000 00
ffff90673f63a3c0 918361399 S Bo:6:006:2 -115 31 = 55534243 50000000 00b00000 80000a28 00039a31 a8000058 00000000 000000
ffff90673f63a3c0 918361423 C Bo:6:006:2 0 31 >
ffff9067adb13480 918361452 S Bi:6:006:1 -115 45056 <
ffff9067adb13480 918363097 C Bi:6:006:1 0 45056 = 98fc9677 d852f513 9fbf7375 d7fdf2d4 5d650447 eb96f5b2 8292a863 52207607
ffff90673f63a3c0 918363144 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918363157 C Bi:6:006:1 0 13 = 55534253 50000000 00000000 00
ffff90673f63a3c0 918363409 S Bo:6:006:2 -115 31 = 55534243 51000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918363433 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918363465 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918363475 C Bi:6:006:1 0 13 = 55534253 51000000 00000000 00
ffff90673f63a3c0 918367127 S Bo:6:006:2 -115 31 = 55534243 52000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918367146 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918367151 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918367184 C Bi:6:006:1 0 13 = 55534253 52000000 00000000 00
ffff90673f63a3c0 918367987 S Bo:6:006:2 -115 31 = 55534243 53000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918368010 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918368027 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918368056 C Bi:6:006:1 0 13 = 55534253 53000000 00000000 00
ffff90673f63a3c0 918368674 S Bo:6:006:2 -115 31 = 55534243 54000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918368712 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918368727 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918368758 C Bi:6:006:1 0 13 = 55534253 54000000 00000000 00
ffff90673f63a3c0 918370223 S Bo:6:006:2 -115 31 = 55534243 55000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918370236 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918370239 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918370276 C Bi:6:006:1 0 13 = 55534253 55000000 00000000 00
ffff90673f63a3c0 918371102 S Bo:6:006:2 -115 31 = 55534243 56000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918371116 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918371118 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918371155 C Bi:6:006:1 0 13 = 55534253 56000000 00000000 00
ffff90673f63a3c0 918371287 S Bo:6:006:2 -115 31 = 55534243 57000000 00100000 80000a28 00039a33 00000008 00000000 000000
ffff90673f63a3c0 918371299 C Bo:6:006:2 0 31 >
ffff90673f6a7000 918371301 S Bi:6:006:1 -115 4096 <
ffff90673f6a7000 918371964 C Bi:6:006:1 0 4096 = 4c776f62 69e41245 64d22d08 ae9db7b8 01c4e6fb 8b029833 8e4a9f4c 30a097df
ffff90673f63a3c0 918371968 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918372004 C Bi:6:006:1 0 13 = 55534253 57000000 00000000 00
ffff90673f63a3c0 918372026 S Bo:6:006:2 -115 31 = 55534243 58000000 00100000 80000a28 00039a33 e0000008 00000000 000000
ffff90673f63a3c0 918372044 C Bo:6:006:2 0 31 >
ffff90673f6a7000 918372047 S Bi:6:006:1 -115 4096 <
ffff90673f6a7000 918372698 C Bi:6:006:1 0 4096 = 85bfff3d 1c873fc8 99903146 ddceb7e5 10fe7570 715c5ff7 9343c2c9 c6ff195c
ffff90673f63a3c0 918372716 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918372740 C Bi:6:006:1 0 13 = 55534253 58000000 00000000 00
ffff90673f63a3c0 918372778 S Bo:6:006:2 -115 31 = 55534243 59000000 00100000 80000a28 00000008 00000008 00000000 000000
ffff90673f63a3c0 918372788 C Bo:6:006:2 0 31 >
ffff90673f6a7000 918372822 S Bi:6:006:1 -115 4096 <
ffff90673f6a7000 918374304 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918374308 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918374344 C Bi:6:006:1 0 13 = 55534253 59000000 00000000 00
ffff90673f63a3c0 918374433 S Bo:6:006:2 -115 31 = 55534243 5a000000 00100000 80000a28 00000008 08000008 00000000 000000
ffff90673f63a3c0 918374471 C Bo:6:006:2 0 31 >
ffff90673f6a7000 918374509 S Bi:6:006:1 -115 4096 <
ffff90673f6a7000 918375175 C Bi:6:006:1 0 4096 = 01040000 11040000 21040000 e85bdf1f 02000400 00000000 00000000 dc1f8466
ffff90673f63a3c0 918375199 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918375207 C Bi:6:006:1 0 13 = 55534253 5a000000 00000000 00
ffff90673f63a3c0 918375243 S Bo:6:006:2 -115 31 = 55534243 5b000000 00020000 80000a28 00039a33 f0000001 00000000 000000
ffff90673f63a3c0 918375258 C Bo:6:006:2 0 31 >
ffff90673f6a7000 918375261 S Bi:6:006:1 -115 512 <
ffff90673f6a7000 918376997 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff90673f63a3c0 918377021 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918377045 C Bi:6:006:1 0 13 = 55534253 5b000000 00000000 00
ffff90673f63a3c0 918377075 S Bo:6:006:2 -115 31 = 55534243 5c000000 00020000 80000a28 00039a33 f1000001 00000000 000000
ffff90673f63a3c0 918377086 C Bo:6:006:2 0 31 >
ffff90673f6a7000 918377089 S Bi:6:006:1 -115 512 <
ffff90673f6a7000 918377709 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff90673f63a3c0 918377733 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918377757 C Bi:6:006:1 0 13 = 55534253 5c000000 00000000 00
ffff90673f63a3c0 918377785 S Bo:6:006:2 -115 31 = 55534243 5d000000 00020000 80000a28 00039a33 f2000001 00000000 000000
ffff90673f63a3c0 918377807 C Bo:6:006:2 0 31 >
ffff90673f6a7000 918377831 S Bi:6:006:1 -115 512 <
ffff90673f6a7000 918378399 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff90673f63a3c0 918378446 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918378474 C Bi:6:006:1 0 13 = 55534253 5d000000 00000000 00
ffff90673f63a3c0 918378533 S Bo:6:006:2 -115 31 = 55534243 5e000000 00020000 80000a28 00039a33 f3000001 00000000 000000
ffff90673f63a3c0 918378577 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918378590 S Bi:6:006:1 -115 512 <
ffff90673f46ff00 918379150 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff90673f63a3c0 918379154 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918379180 C Bi:6:006:1 0 13 = 55534253 5e000000 00000000 00
ffff90673f63a3c0 918379222 S Bo:6:006:2 -115 31 = 55534243 5f000000 00100000 80000a28 00039a32 f0000008 00000000 000000
ffff90673f63a3c0 918379233 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918379236 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918379903 C Bi:6:006:1 0 4096 = 8add6e3c 2432a9cf 357c4940 a360e0ab a20f802f 1d321990 325a7363 73ab2a85
ffff90673f63a3c0 918379908 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918379963 C Bi:6:006:1 0 13 = 55534253 5f000000 00000000 00
ffff90673f63a3c0 918380084 S Bo:6:006:2 -115 31 = 55534243 60000000 00080000 80000a28 00039a33 e8000004 00000000 000000
ffff90673f63a3c0 918380140 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918380155 S Bi:6:006:1 -115 2048 <
ffff90673f46ff00 918380745 C Bi:6:006:1 0 2048 = c71672c3 62f9e441 93556627 49f7d2f6 c1556d39 f35c478c b7303075 c5c622c8
ffff90673f63a3c0 918380796 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918380817 C Bi:6:006:1 0 13 = 55534253 60000000 00000000 00
ffff90673f63a3c0 918380889 S Bo:6:006:2 -115 31 = 55534243 61000000 00020000 80000a28 00039a33 ec000001 00000000 000000
ffff90673f63a3c0 918380913 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918380927 S Bi:6:006:1 -115 512 <
ffff90673f46ff00 918381533 C Bi:6:006:1 0 512 = 809f4d0a bc5667de 57467a8e 646e8913 8d0bae35 d0f85a27 97065766 1ac31874
ffff90673f63a3c0 918381569 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918381596 C Bi:6:006:1 0 13 = 55534253 61000000 00000000 00
ffff90673f63a3c0 918381615 S Bo:6:006:2 -115 31 = 55534243 62000000 00020000 80000a28 00039a33 ed000001 00000000 000000
ffff90673f63a3c0 918381661 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918381675 S Bi:6:006:1 -115 512 <
ffff90673f46ff00 918382265 C Bi:6:006:1 0 512 = 97bc843d bb40e8dd 37106a17 8b97f602 c439d3b5 e4dfd162 bd8e6a42 09f83484
ffff90673f63a3c0 918382286 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918382292 C Bi:6:006:1 0 13 = 55534253 62000000 00000000 00
ffff90673f63a3c0 918382298 S Bo:6:006:2 -115 31 = 55534243 63000000 00020000 80000a28 00039a33 ee000001 00000000 000000
ffff90673f63a3c0 918382333 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918382335 S Bi:6:006:1 -115 512 <
ffff90673f46ff00 918382888 C Bi:6:006:1 0 512 = 66f05e02 eb030290 ceab76c6 e426f4b0 6ed9b467 31e125f2 28e6c092 135d8134
ffff90673f63a3c0 918382901 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918382953 C Bi:6:006:1 0 13 = 55534253 63000000 00000000 00
ffff90673f63a3c0 918382996 S Bo:6:006:2 -115 31 = 55534243 64000000 00020000 80000a28 00039a33 ef000001 00000000 000000
ffff90673f63a3c0 918383009 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918383012 S Bi:6:006:1 -115 512 <
ffff90673f46ff00 918383663 C Bi:6:006:1 0 512 = 3a43294b 77481b39 4cec9d97 da766318 358af9ff e8c5dfce 6bbfff83 e78d3f1c
ffff90673f63a3c0 918383697 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918383721 C Bi:6:006:1 0 13 = 55534253 64000000 00000000 00
ffff90673f63a3c0 918383858 S Bo:6:006:2 -115 31 = 55534243 65000000 00100000 80000a28 00000008 20000008 00000000 000000
ffff90673f63a3c0 918383909 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918383928 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918385420 C Bi:6:006:1 0 4096 = 04800000 04800100 04800200 04800300 04800400 04800c00 04800d00 04801800
ffff90673f63a3c0 918385455 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918385475 C Bi:6:006:1 0 13 = 55534253 65000000 00000000 00
ffff90673f63a3c0 918385580 S Bo:6:006:2 -115 31 = 55534243 66000000 00100000 80000a28 00000008 40000008 00000000 000000
ffff90673f63a3c0 918385617 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918385646 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918386310 C Bi:6:006:1 0 4096 = 08800000 08800100 08800200 08800300 08800400 08800c00 08800d00 08801800
ffff90673f63a3c0 918386314 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918386323 C Bi:6:006:1 0 13 = 55534253 66000000 00000000 00
ffff90673f63a3c0 918386405 S Bo:6:006:2 -115 31 = 55534243 67000000 00100000 80000a28 00000008 80000008 00000000 000000
ffff90673f63a3c0 918386449 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918386494 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918387120 C Bi:6:006:1 0 4096 = 10800000 10800100 10800200 10800300 10800400 10800c00 10800d00 10801800
ffff90673f63a3c0 918387126 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918387181 C Bi:6:006:1 0 13 = 55534253 67000000 00000000 00
ffff90673f63a3c0 918387287 S Bo:6:006:2 -115 31 = 55534243 68000000 00100000 80000a28 00000009 00000008 00000000 000000
ffff90673f63a3c0 918387355 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918387374 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918388054 C Bi:6:006:1 0 4096 = 20800000 20800100 20800200 20800300 20800400 20800c00 20800d00 20801800
ffff90673f63a3c0 918388059 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918388088 C Bi:6:006:1 0 13 = 55534253 68000000 00000000 00
ffff90673f63a3c0 918388153 S Bo:6:006:2 -115 31 = 55534243 69000000 00100000 80000a28 0000000a 00000008 00000000 000000
ffff90673f63a3c0 918388180 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918388184 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918388853 C Bi:6:006:1 0 4096 = 40800000 40800100 40800200 40800300 40800400 40800c00 40800d00 40801800
ffff90673f63a3c0 918388859 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918388915 C Bi:6:006:1 0 13 = 55534253 69000000 00000000 00
ffff90673f63a3c0 918389036 S Bo:6:006:2 -115 31 = 55534243 6a000000 00100000 80000a28 0000000c 00000008 00000000 000000
ffff90673f63a3c0 918389090 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918389108 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918389792 C Bi:6:006:1 0 4096 = 80800000 80800100 80800200 80800300 80800400 80800c00 80800d00 80801800
ffff90673f63a3c0 918389826 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918389850 C Bi:6:006:1 0 13 = 55534253 6a000000 00000000 00
ffff90673f63a3c0 918389974 S Bo:6:006:2 -115 31 = 55534243 6b000000 00100000 80000a28 00000010 00000008 00000000 000000
ffff90673f63a3c0 918390028 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918390041 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918390694 C Bi:6:006:1 0 4096 = 00810000 00810100 00810200 00810300 00810400 00810c00 00810d00 00811800
ffff90673f63a3c0 918390740 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918390769 C Bi:6:006:1 0 13 = 55534253 6b000000 00000000 00
ffff90673f63a3c0 918390857 S Bo:6:006:2 -115 31 = 55534243 6c000000 00100000 80000a28 00000018 00000008 00000000 000000
ffff90673f63a3c0 918390894 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918390918 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918391570 C Bi:6:006:1 0 4096 = 00820000 00820100 00820200 00820300 00820400 00820c00 00820d00 00821800
ffff90673f63a3c0 918391614 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918391643 C Bi:6:006:1 0 13 = 55534253 6c000000 00000000 00
ffff90673f63a3c0 918391729 S Bo:6:006:2 -115 31 = 55534243 6d000000 00100000 80000a28 00000028 00000008 00000000 000000
ffff90673f63a3c0 918391766 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918391790 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918392465 C Bi:6:006:1 0 4096 = 00840000 00840100 00840200 00840300 00840400 00840c00 00840d00 00841800
ffff90673f63a3c0 918392511 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918392539 C Bi:6:006:1 0 13 = 55534253 6d000000 00000000 00
ffff90673f63a3c0 918392636 S Bo:6:006:2 -115 31 = 55534243 6e000000 00100000 80000a28 00000008 18000008 00000000 000000
ffff90673f63a3c0 918392673 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918392696 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918393345 C Bi:6:006:1 0 4096 = 03800000 03800100 03800200 03800300 03800400 03800c00 03800d00 03801800
ffff90673f63a3c0 918393349 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918393393 C Bi:6:006:1 0 13 = 55534253 6e000000 00000000 00
ffff90673f63a3c0 918393520 S Bo:6:006:2 -115 31 = 55534243 6f000000 00100000 80000a28 00000008 38000008 00000000 000000
ffff90673f63a3c0 918393545 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918393548 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918394239 C Bi:6:006:1 0 4096 = 07800000 07800100 07800200 07800300 07800400 07800c00 07800d00 07801800
ffff90673f63a3c0 918394243 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918394298 C Bi:6:006:1 0 13 = 55534253 6f000000 00000000 00
ffff90673f63a3c0 918394365 S Bo:6:006:2 -115 31 = 55534243 70000000 00100000 80000a28 00000008 78000008 00000000 000000
ffff90673f63a3c0 918394406 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918394429 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918395099 C Bi:6:006:1 0 4096 = 0f800000 0f800100 0f800200 0f800300 0f800400 0f800c00 0f800d00 0f801800
ffff90673f63a3c0 918395103 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918395129 C Bi:6:006:1 0 13 = 55534253 70000000 00000000 00
ffff90673f63a3c0 918395243 S Bo:6:006:2 -115 31 = 55534243 71000000 00100000 80000a28 00000008 10000008 00000000 000000
ffff90673f63a3c0 918395270 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918395284 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918395965 C Bi:6:006:1 0 4096 = 00004000 10004000 20004000 f05ff01f 00000500 00000000 00000000 f01f7270
ffff90673f63a3c0 918395994 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918396018 C Bi:6:006:1 0 13 = 55534253 71000000 00000000 00
ffff90673f63a3c0 918396062 S Bo:6:006:2 -115 31 = 55534243 72000000 00200000 80000a28 00000008 28000010 00000000 000000
ffff90673f63a3c0 918396072 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918396089 S Bi:6:006:1 -115 8192 <
ffff90673f46ff00 918396862 C Bi:6:006:1 0 8192 = 05800000 05800100 05800200 05800300 05800400 05800c00 05800d00 05801800
ffff90673f63a3c0 918396906 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918396934 C Bi:6:006:1 0 13 = 55534253 72000000 00000000 00
ffff90673f63a3c0 918396954 S Bo:6:006:2 -115 31 = 55534243 73000000 00600000 80000a28 00000008 48000030 00000000 000000
ffff90673f63a3c0 918397003 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918397015 S Bi:6:006:1 -115 24576 <
ffff90673f46ff00 918398090 C Bi:6:006:1 0 24576 = 09800000 09800100 09800200 09800300 09800400 09800c00 09800d00 09801800
ffff90673f63a3c0 918398135 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918398164 C Bi:6:006:1 0 13 = 55534253 73000000 00000000 00
ffff90673f63a3c0 918398183 S Bo:6:006:2 -115 31 = 55534243 74000000 00f00000 80000a28 00000008 88000078 00000000 000000
ffff90673f63a3c0 918398218 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918398234 S Bi:6:006:1 -115 61440 <
ffff90673f46ff00 918400215 C Bi:6:006:1 0 61440 = 11800000 11800100 11800200 11800300 11800400 11800c00 11800d00 11801800
ffff90673f63a3c0 918400263 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918400297 C Bi:6:006:1 0 13 = 55534253 74000000 00000000 00
ffff90673f63a3c0 918400325 S Bo:6:006:2 -115 31 = 55534243 75000000 00100000 80000a28 00000009 08000008 00000000 000000
ffff90673f63a3c0 918400376 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918400391 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918401051 C Bi:6:006:1 0 4096 = 21800000 21800100 21800200 21800300 21800400 21800c00 21800d00 21801800
ffff90673f63a3c0 918401098 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918401142 C Bi:6:006:1 0 13 = 55534253 75000000 00000000 00
ffff90673f63a3c0 918401260 S Bo:6:006:2 -115 31 = 55534243 76000000 00e00100 80000a28 00000009 100000f0 00000000 000000
ffff90673f63a3c0 918401299 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918401326 S Bi:6:006:1 -115 122880 <
ffff90673f46ff00 918404946 C Bi:6:006:1 0 122880 = 22800000 22800100 22800200 22800300 22800400 22800c00 22800d00 22801800
ffff90673f63a3c0 918404994 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918405031 C Bi:6:006:1 0 13 = 55534253 76000000 00000000 00
ffff90673f63a3c0 918405324 S Bo:6:006:2 -115 31 = 55534243 77000000 00e00100 80000a28 0000000a 080000f0 00000000 000000
ffff90673f63a3c0 918405341 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918405372 S Bi:6:006:1 -115 122880 <
ffff90673f46ff00 918409083 C Bi:6:006:1 0 122880 = 41800000 41800100 41800200 41800300 41800400 41800c00 41800d00 41801800
ffff90673f63a3c0 918409124 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918409153 C Bi:6:006:1 0 13 = 55534253 77000000 00000000 00
ffff90673f63a3c0 918409205 S Bo:6:006:2 -115 31 = 55534243 78000000 00200000 80000a28 0000000a f8000010 00000000 000000
ffff90673f63a3c0 918409240 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918409249 S Bi:6:006:1 -115 8192 <
ffff90673f46ff00 918410103 C Bi:6:006:1 0 8192 = 5f800000 5f800100 5f800200 5f800300 5f800400 5f800c00 5f800d00 5f801800
ffff90673f63a3c0 918410144 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918410166 C Bi:6:006:1 0 13 = 55534253 78000000 00000000 00
ffff90673f63a3c0 918410332 S Bo:6:006:2 -115 31 = 55534243 79000000 00e00100 80000a28 0000000b 080000f0 00000000 000000
ffff90673f63a3c0 918410365 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918410389 S Bi:6:006:1 -115 122880 <
ffff90673f46ff00 918413991 C Bi:6:006:1 0 122880 = 61800000 61800100 61800200 61800300 61800400 61800c00 61800d00 61801800
ffff90673f63a3c0 918414025 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918414046 C Bi:6:006:1 0 13 = 55534253 79000000 00000000 00
ffff90673f63a3c0 918414095 S Bo:6:006:2 -115 31 = 55534243 7a000000 00100000 80000a28 0000000b f8000008 00000000 000000
ffff90673f63a3c0 918414115 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918414133 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918414242 C Bi:6:006:1 0 4096 = 7f800000 7f800100 7f800200 7f800300 7f800400 7f800c00 7f800d00 7f801800
ffff90673f63a3c0 918414273 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918414282 C Bi:6:006:1 0 13 = 55534253 7a000000 00000000 00
ffff90673f63a3c0 918414554 S Bo:6:006:2 -115 31 = 55534243 7b000000 00e00100 80000a28 00039a2e 000000f0 00000000 000000
ffff90673f63a3c0 918414593 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918414619 S Bi:6:006:1 -115 122880 <
ffff90673f46ff00 918419259 C Bi:6:006:1 0 122880 = c77ccf3d b18bbc20 49a7c771 ab78fd88 353bdfd1 c2f59b72 ee1c7082 2c62c6f6
ffff90673f63a3c0 918419307 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918419320 C Bi:6:006:1 0 13 = 55534253 7b000000 00000000 00
ffff90673f63a3c0 918419382 S Bo:6:006:2 -115 31 = 55534243 7c000000 00200000 80000a28 00039a2e f0000010 00000000 000000
ffff90673f63a3c0 918419413 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918419475 S Bi:6:006:1 -115 8192 <
ffff90673f46ff00 918420204 C Bi:6:006:1 0 8192 = 2d1e8c4f 0d6d6356 384be8bc b5cfb50e cdb3025f bcfecddd 4f80d2e5 eb6b3284
ffff90673f63a3c0 918420241 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918420254 C Bi:6:006:1 0 13 = 55534253 7c000000 00000000 00
ffff90673f63a3c0 918420453 S Bo:6:006:2 -115 31 = 55534243 7d000000 00e00100 80000a28 00039a2f 000000f0 00000000 000000
ffff90673f63a3c0 918420483 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918420512 S Bi:6:006:1 -115 122880 <
ffff90673f46ff00 918424188 C Bi:6:006:1 0 122880 = aa1c9eea 8ab1cada d7bd4902 c36a01ea 724ec451 e5509315 ca093a94 9dcab358
ffff90673f63a3c0 918424226 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918424256 C Bi:6:006:1 0 13 = 55534253 7d000000 00000000 00
ffff90673f63a3c0 918424300 S Bo:6:006:2 -115 31 = 55534243 7e000000 00200000 80000a28 00039a2f f0000010 00000000 000000
ffff90673f63a3c0 918424334 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918424340 S Bi:6:006:1 -115 8192 <
ffff90673f46ff00 918424625 C Bi:6:006:1 0 8192 = 6ddcf87d 817edc15 a527992a 77e6c620 1adc6410 77ef845a d44f3327 5ac54da3
ffff90673f63a3c0 918424672 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918424696 C Bi:6:006:1 0 13 = 55534253 7e000000 00000000 00
ffff90673f63a3c0 918425049 S Bo:6:006:2 -115 31 = 55534243 7f000000 00e00100 80000a28 00039a30 000000f0 00000000 000000
ffff90673f63a3c0 918425094 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918425123 S Bi:6:006:1 -115 122880 <
ffff90673f46ff00 918428301 C Bi:6:006:1 0 122880 = c2bb8f02 e6227d01 1316d5d4 f8589196 3738834e 7c5c90cb 17868ef4 9eb65148
ffff90673f63a3c0 918428350 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918428396 C Bi:6:006:1 0 13 = 55534253 7f000000 00000000 00
ffff90673f63a3c0 918428454 S Bo:6:006:2 -115 31 = 55534243 80000000 00200000 80000a28 00039a30 f0000010 00000000 000000
ffff90673f63a3c0 918428494 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918428550 S Bi:6:006:1 -115 8192 <
ffff90673f46ff00 918428799 C Bi:6:006:1 0 8192 = 2378ad20 bc2a7d26 58ef9fc8 48429609 72a99082 f13ed042 ba93cad1 de25670f
ffff90673f63a3c0 918428847 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918428882 C Bi:6:006:1 0 13 = 55534253 80000000 00000000 00
ffff90673f63a3c0 918429098 S Bo:6:006:2 -115 31 = 55534243 81000000 00e00100 80000a28 00039a31 000000f0 00000000 000000
ffff90673f63a3c0 918429141 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918429160 S Bi:6:006:1 -115 122880 <
ffff90673f46ff00 918432317 C Bi:6:006:1 0 122880 = 315bbba0 afae98ec 6e222b9d ba2bc3bd 1a3f0717 1688e8e5 c30b4584 6f4f58ae
ffff90673f63a3c0 918432370 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918432388 C Bi:6:006:1 0 13 = 55534253 81000000 00000000 00
ffff90673f63a3c0 918432453 S Bo:6:006:2 -115 31 = 55534243 82000000 00200000 80000a28 00039a31 f0000010 00000000 000000
ffff90673f63a3c0 918432513 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918432550 S Bi:6:006:1 -115 8192 <
ffff90673f46ff00 918432810 C Bi:6:006:1 0 8192 = 15c5f9a5 4fdcf7b5 d4013ce6 82011093 87fb73d9 5b8553c4 c7965bfb e4ae9642
ffff90673f63a3c0 918432862 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918432903 C Bi:6:006:1 0 13 = 55534253 82000000 00000000 00
ffff90673f63a3c0 918433281 S Bo:6:006:2 -115 31 = 55534243 83000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918433330 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918433368 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918433382 C Bi:6:006:1 0 13 = 55534253 83000000 00000000 00
ffff90673f63a3c0 918433640 S Bo:6:006:2 -115 31 = 55534243 84000000 00100000 80000a28 00000000 00000008 00000000 000000
ffff90673f63a3c0 918433686 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918433728 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918435192 C Bi:6:006:1 0 4096 = fab80000 8ed0bc00 7c8bf450 07501ffb fcbf0006 b90001f3 a5ea1e06 0000bebe
ffff90673f63a3c0 918435237 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918435267 C Bi:6:006:1 0 13 = 55534253 84000000 00000000 00
ffff90673f63a3c0 918435686 S Bo:6:006:2 -115 31 = 55534243 85000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918435724 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918435763 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918435805 C Bi:6:006:1 0 13 = 55534253 85000000 00000000 00
ffff90673f63a3c0 918435994 S Bo:6:006:2 -115 31 = 55534243 86000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918436025 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918436048 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918436070 C Bi:6:006:1 0 13 = 55534253 86000000 00000000 00
ffff90673f63a3c0 918442387 S Bo:6:006:2 -115 31 = 55534243 87000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918442426 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918442445 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918442462 C Bi:6:006:1 0 13 = 55534253 87000000 00000000 00
ffff90673f63a3c0 918450451 S Bo:6:006:2 -115 31 = 55534243 88000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918450471 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918450496 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918450508 C Bi:6:006:1 0 13 = 55534253 88000000 00000000 00
ffff90673f63a3c0 918450578 S Bo:6:006:2 -115 31 = 55534243 89000000 00400000 80000a28 00000008 00000020 00000000 000000
ffff90673f63a3c0 918450590 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918450607 S Bi:6:006:1 -115 16384 <
ffff90673f46ff00 918451582 C Bi:6:006:1 0 16384 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918451603 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918451629 C Bi:6:006:1 0 13 = 55534253 89000000 00000000 00
ffff90673f63a3c0 918451682 S Bo:6:006:2 -115 31 = 55534243 8a000000 00800000 80000a28 00000008 20000040 00000000 000000
ffff90673f63a3c0 918451694 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918451697 S Bi:6:006:1 -115 32768 <
ffff90673f46ff00 918452988 C Bi:6:006:1 0 32768 = 04800000 04800100 04800200 04800300 04800400 04800c00 04800d00 04801800
ffff90673f63a3c0 918453009 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918453027 C Bi:6:006:1 0 13 = 55534253 8a000000 00000000 00
ffff90673f63a3c0 918453047 S Bo:6:006:2 -115 31 = 55534243 8b000000 00100000 80000a28 00000029 08000008 00000000 000000
ffff90673f63a3c0 918453067 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918453080 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918454147 C Bi:6:006:1 0 4096 = 00000000 00000000 33f72257 33f72257 33f72257 00000000 00000000 00000000
ffff90673f63a3c0 918454167 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918454178 C Bi:6:006:1 0 13 = 55534253 8b000000 00000000 00
ffff90673f63a3c0 918454208 S Bo:6:006:2 -115 31 = 55534243 8c000000 00100000 80000a28 0001c408 00000008 00000000 000000
ffff90673f63a3c0 918454219 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918454223 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918455772 C Bi:6:006:1 0 4096 = c03b3998 00000004 00000000 00001000 00008000 00000001 00031061 00000000
ffff90673f63a3c0 918455792 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918455803 C Bi:6:006:1 0 13 = 55534253 8c000000 00000000 00
ffff90673f63a3c0 918455853 S Bo:6:006:2 -115 31 = 55534243 8d000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918455865 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918455881 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918455911 C Bi:6:006:1 0 13 = 55534253 8d000000 00000000 00
ffff90673f63a3c0 918457946 S Bo:6:006:2 -115 31 = 55534243 8e000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918457963 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918457979 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918458007 C Bi:6:006:1 0 13 = 55534253 8e000000 00000000 00
ffff90673f63a3c0 918458068 S Bo:6:006:2 -115 31 = 55534243 8f000000 00400000 80000a28 00000008 00000020 00000000 000000
ffff90673f63a3c0 918458080 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918458083 S Bi:6:006:1 -115 16384 <
ffff90673f46ff00 918459887 C Bi:6:006:1 0 16384 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 918459918 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918459925 C Bi:6:006:1 0 13 = 55534253 8f000000 00000000 00
ffff90673f63a3c0 918459982 S Bo:6:006:2 -115 31 = 55534243 90000000 00800000 80000a28 00000008 20000040 00000000 000000
ffff90673f63a3c0 918459995 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918460011 S Bi:6:006:1 -115 32768 <
ffff90673f46ff00 918461310 C Bi:6:006:1 0 32768 = 04800000 04800100 04800200 04800300 04800400 04800c00 04800d00 04801800
ffff90673f63a3c0 918461341 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918461348 C Bi:6:006:1 0 13 = 55534253 90000000 00000000 00
ffff90673f63a3c0 918461365 S Bo:6:006:2 -115 31 = 55534243 91000000 00100000 80000a28 00000029 08000008 00000000 000000
ffff90673f63a3c0 918461400 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918461427 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918462445 C Bi:6:006:1 0 4096 = 00000000 00000000 33f72257 33f72257 33f72257 00000000 00000000 00000000
ffff90673f63a3c0 918462507 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918462526 C Bi:6:006:1 0 13 = 55534253 91000000 00000000 00
ffff90673f63a3c0 918462587 S Bo:6:006:2 -115 31 = 55534243 92000000 00100000 80000a28 0001c408 00000008 00000000 000000
ffff90673f63a3c0 918462628 C Bo:6:006:2 0 31 >
ffff90673f46ff00 918462651 S Bi:6:006:1 -115 4096 <
ffff90673f46ff00 918464138 C Bi:6:006:1 0 4096 = c03b3998 00000004 00000000 00001000 00008000 00000001 00031061 00000000
ffff90673f63a3c0 918464143 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918464178 C Bi:6:006:1 0 13 = 55534253 92000000 00000000 00
ffff90673f63a3c0 918464213 S Bo:6:006:2 -115 31 = 55534243 93000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 918464224 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 918464227 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 918464263 C Bi:6:006:1 0 13 = 55534253 93000000 00000000 00
ffff9067b1774b40 919198818 C Ii:6:004:1 0:2 8 = 01000000 00000000
ffff9067b1774b40 919198886 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 919244759 C Ii:6:004:1 0:2 8 = 00000000 00000000
ffff9067b1774b40 919244799 S Ii:6:004:1 -115:2 8 <
ffff9067b1774780 919534496 S Bo:6:003:1 -115 31 = 55534243 b90e0000 00000000 00000600 00000000 00000000 00000000 000000
ffff9067b1774780 919534553 C Bo:6:003:1 0 31 >
ffff9067b1774780 919534588 S Bi:6:003:2 -115 13 <
ffff9067b1774780 919534989 C Bi:6:003:2 0 13 = 55534253 b90e0000 00000000 01
ffff9067b1774780 919535029 S Bo:6:003:1 -115 31 = 55534243 ba0e0000 12000000 80000603 00000012 00000000 00000000 000000
ffff9067b1774780 919535093 C Bo:6:003:1 0 31 >
ffff90673f46f480 919535141 S Bi:6:003:2 -115 18 <
ffff90673f46f480 919535676 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 919535718 S Bi:6:003:2 -115 13 <
ffff9067b1774780 919535786 C Bi:6:003:2 0 13 = 55534253 ba0e0000 00000000 00
ffff9067b1774780 919535874 S Bo:6:003:1 -115 31 = 55534243 bb0e0000 00000000 00010600 00000000 00000000 00000000 000000
ffff9067b1774780 919535939 C Bo:6:003:1 0 31 >
ffff9067b1774780 919535947 S Bi:6:003:2 -115 13 <
ffff9067b1774780 919536307 C Bi:6:003:2 0 13 = 55534253 bb0e0000 00000000 01
ffff9067b1774780 919536318 S Bo:6:003:1 -115 31 = 55534243 bc0e0000 12000000 80010603 00000012 00000000 00000000 000000
ffff9067b1774780 919536363 C Bo:6:003:1 0 31 >
ffff9067c0aac3c0 919536378 S Bi:6:003:2 -115 18 <
ffff9067c0aac3c0 919536938 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 919536958 S Bi:6:003:2 -115 13 <
ffff9067b1774780 919537046 C Bi:6:003:2 0 13 = 55534253 bc0e0000 00000000 00
ffff9067b1774780 919537101 S Bo:6:003:1 -115 31 = 55534243 bd0e0000 00000000 00020600 00000000 00000000 00000000 000000
ffff9067b1774780 919537139 C Bo:6:003:1 0 31 >
ffff9067b1774780 919537143 S Bi:6:003:2 -115 13 <
ffff9067b1774780 919537551 C Bi:6:003:2 0 13 = 55534253 bd0e0000 00000000 01
ffff9067b1774780 919537586 S Bo:6:003:1 -115 31 = 55534243 be0e0000 12000000 80020603 00000012 00000000 00000000 000000
ffff9067b1774780 919537653 C Bo:6:003:1 0 31 >
ffff9067adb13840 919537658 S Bi:6:003:2 -115 18 <
ffff9067adb13840 919538185 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 919538191 S Bi:6:003:2 -115 13 <
ffff9067b1774780 919538329 C Bi:6:003:2 0 13 = 55534253 be0e0000 00000000 00
ffff9067b1774780 919538350 S Bo:6:003:1 -115 31 = 55534243 bf0e0000 00000000 00030600 00000000 00000000 00000000 000000
ffff9067b1774780 919538419 C Bo:6:003:1 0 31 >
ffff9067b1774780 919538426 S Bi:6:003:2 -115 13 <
ffff9067b1774780 919538812 C Bi:6:003:2 0 13 = 55534253 bf0e0000 00000000 01
ffff9067b1774780 919538819 S Bo:6:003:1 -115 31 = 55534243 c00e0000 12000000 80030603 00000012 00000000 00000000 000000
ffff9067b1774780 919538869 C Bo:6:003:1 0 31 >
ffff9067c0aac3c0 919538875 S Bi:6:003:2 -115 18 <
ffff9067c0aac3c0 919539423 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 919539439 S Bi:6:003:2 -115 13 <
ffff9067b1774780 919539559 C Bi:6:003:2 0 13 = 55534253 c00e0000 00000000 00
ffff90673f63a3c0 920494529 S Bo:6:006:2 -115 31 = 55534243 94000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 920494569 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 920494599 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 920494635 C Bi:6:006:1 0 13 = 55534253 94000000 00000000 00
ffff90673f63a3c0 921008726 S Bo:6:006:2 -115 31 = 55534243 95000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 921008766 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 921008773 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921008788 C Bi:6:006:1 0 13 = 55534253 95000000 00000000 00
ffff90673f63a3c0 921009055 S Bo:6:006:2 -115 31 = 55534243 96000000 00100000 80000a28 00039a33 00000008 00000000 000000
ffff90673f63a3c0 921009070 C Bo:6:006:2 0 31 >
ffff90673f46f480 921009076 S Bi:6:006:1 -115 4096 <
ffff90673f46f480 921010615 C Bi:6:006:1 0 4096 = 4c776f62 69e41245 64d22d08 ae9db7b8 01c4e6fb 8b029833 8e4a9f4c 30a097df
ffff90673f63a3c0 921010662 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921010672 C Bi:6:006:1 0 13 = 55534253 96000000 00000000 00
ffff90673f63a3c0 921010735 S Bo:6:006:2 -115 31 = 55534243 97000000 00100000 80000a28 00039a33 e0000008 00000000 000000
ffff90673f63a3c0 921010751 C Bo:6:006:2 0 31 >
ffff90673f46f480 921010765 S Bi:6:006:1 -115 4096 <
ffff90673f46f480 921011444 C Bi:6:006:1 0 4096 = 85bfff3d 1c873fc8 99903146 ddceb7e5 10fe7570 715c5ff7 9343c2c9 c6ff195c
ffff90673f63a3c0 921011490 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921011500 C Bi:6:006:1 0 13 = 55534253 97000000 00000000 00
ffff90673f63a3c0 921011557 S Bo:6:006:2 -115 31 = 55534243 98000000 00100000 80000a28 00000008 00000008 00000000 000000
ffff90673f63a3c0 921011572 C Bo:6:006:2 0 31 >
ffff90673f46f480 921011587 S Bi:6:006:1 -115 4096 <
ffff90673f46f480 921013122 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 921013130 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921013150 C Bi:6:006:1 0 13 = 55534253 98000000 00000000 00
ffff90673f63a3c0 921013214 S Bo:6:006:2 -115 31 = 55534243 99000000 00100000 80000a28 00000008 08000008 00000000 000000
ffff90673f63a3c0 921013230 C Bo:6:006:2 0 31 >
ffff90673f46f480 921013244 S Bi:6:006:1 -115 4096 <
ffff90673f46f480 921013933 C Bi:6:006:1 0 4096 = 01040000 11040000 21040000 e85bdf1f 02000400 00000000 00000000 dc1f8466
ffff90673f63a3c0 921013978 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921013988 C Bi:6:006:1 0 13 = 55534253 99000000 00000000 00
ffff90673f63a3c0 921014046 S Bo:6:006:2 -115 31 = 55534243 9a000000 00020000 80000a28 00039a33 f0000001 00000000 000000
ffff90673f63a3c0 921014062 C Bo:6:006:2 0 31 >
ffff90673f46f480 921014076 S Bi:6:006:1 -115 512 <
ffff90673f46f480 921015775 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff90673f63a3c0 921015846 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921015867 C Bi:6:006:1 0 13 = 55534253 9a000000 00000000 00
ffff90673f63a3c0 921015919 S Bo:6:006:2 -115 31 = 55534243 9b000000 00020000 80000a28 00039a33 f1000001 00000000 000000
ffff90673f63a3c0 921015938 C Bo:6:006:2 0 31 >
ffff9067b478e000 921015946 S Bi:6:006:1 -115 512 <
ffff9067b478e000 921016532 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff90673f63a3c0 921016567 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921016582 C Bi:6:006:1 0 13 = 55534253 9b000000 00000000 00
ffff90673f63a3c0 921016621 S Bo:6:006:2 -115 31 = 55534243 9c000000 00020000 80000a28 00039a33 f2000001 00000000 000000
ffff90673f63a3c0 921016646 C Bo:6:006:2 0 31 >
ffff9067c0aac3c0 921016653 S Bi:6:006:1 -115 512 <
ffff9067c0aac3c0 921017231 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff90673f63a3c0 921017238 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921017258 C Bi:6:006:1 0 13 = 55534253 9c000000 00000000 00
ffff90673f63a3c0 921017316 S Bo:6:006:2 -115 31 = 55534243 9d000000 00020000 80000a28 00039a33 f3000001 00000000 000000
ffff90673f63a3c0 921017338 C Bo:6:006:2 0 31 >
ffff90673f46f480 921017348 S Bi:6:006:1 -115 512 <
ffff90673f46f480 921017946 C Bi:6:006:1 0 512 = ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
ffff90673f63a3c0 921017997 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921018012 C Bi:6:006:1 0 13 = 55534253 9d000000 00000000 00
ffff90673f63a3c0 921018200 S Bo:6:006:2 -115 31 = 55534243 9e000000 00100000 80000a28 00039a32 f0000008 00000000 000000
ffff90673f63a3c0 921018234 C Bo:6:006:2 0 31 >
ffff90673f46f480 921018253 S Bi:6:006:1 -115 4096 <
ffff90673f46f480 921018967 C Bi:6:006:1 0 4096 = 8add6e3c 2432a9cf 357c4940 a360e0ab a20f802f 1d321990 325a7363 73ab2a85
ffff90673f63a3c0 921019024 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921019063 C Bi:6:006:1 0 13 = 55534253 9e000000 00000000 00
ffff90673f63a3c0 921019267 S Bo:6:006:2 -115 31 = 55534243 9f000000 00080000 80000a28 00039a33 e8000004 00000000 000000
ffff90673f63a3c0 921019304 C Bo:6:006:2 0 31 >
ffff90673f46f480 921019322 S Bi:6:006:1 -115 2048 <
ffff90673f46f480 921019989 C Bi:6:006:1 0 2048 = c71672c3 62f9e441 93556627 49f7d2f6 c1556d39 f35c478c b7303075 c5c622c8
ffff90673f63a3c0 921020044 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921020084 C Bi:6:006:1 0 13 = 55534253 9f000000 00000000 00
ffff90673f63a3c0 921020225 S Bo:6:006:2 -115 31 = 55534243 a0000000 00020000 80000a28 00039a33 ec000001 00000000 000000
ffff90673f63a3c0 921020273 C Bo:6:006:2 0 31 >
ffff9067b478e000 921020326 S Bi:6:006:1 -115 512 <
ffff9067b478e000 921020916 C Bi:6:006:1 0 512 = 809f4d0a bc5667de 57467a8e 646e8913 8d0bae35 d0f85a27 97065766 1ac31874
ffff90673f63a3c0 921020973 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921021013 C Bi:6:006:1 0 13 = 55534253 a0000000 00000000 00
ffff90673f63a3c0 921021119 S Bo:6:006:2 -115 31 = 55534243 a1000000 00020000 80000a28 00039a33 ed000001 00000000 000000
ffff90673f63a3c0 921021154 C Bo:6:006:2 0 31 >
ffff9067c0aac3c0 921021166 S Bi:6:006:1 -115 512 <
ffff9067c0aac3c0 921021820 C Bi:6:006:1 0 512 = 97bc843d bb40e8dd 37106a17 8b97f602 c439d3b5 e4dfd162 bd8e6a42 09f83484
ffff90673f63a3c0 921021855 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921021866 C Bi:6:006:1 0 13 = 55534253 a1000000 00000000 00
ffff90673f63a3c0 921021941 S Bo:6:006:2 -115 31 = 55534243 a2000000 00020000 80000a28 00039a33 ee000001 00000000 000000
ffff90673f63a3c0 921021996 C Bo:6:006:2 0 31 >
ffff90673f46f480 921022038 S Bi:6:006:1 -115 512 <
ffff90673f46f480 921022629 C Bi:6:006:1 0 512 = 66f05e02 eb030290 ceab76c6 e426f4b0 6ed9b467 31e125f2 28e6c092 135d8134
ffff90673f63a3c0 921022684 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921022737 C Bi:6:006:1 0 13 = 55534253 a2000000 00000000 00
ffff90673f63a3c0 921022881 S Bo:6:006:2 -115 31 = 55534243 a3000000 00020000 80000a28 00039a33 ef000001 00000000 000000
ffff90673f63a3c0 921022929 C Bo:6:006:2 0 31 >
ffff9067b478e000 921022955 S Bi:6:006:1 -115 512 <
ffff9067b478e000 921023540 C Bi:6:006:1 0 512 = 3a43294b 77481b39 4cec9d97 da766318 358af9ff e8c5dfce 6bbfff83 e78d3f1c
ffff90673f63a3c0 921023597 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921023637 C Bi:6:006:1 0 13 = 55534253 a3000000 00000000 00
ffff90673f63a3c0 921023889 S Bo:6:006:2 -115 31 = 55534243 a4000000 00100000 80000a28 00000008 20000008 00000000 000000
ffff90673f63a3c0 921023937 C Bo:6:006:2 0 31 >
ffff9067b478e000 921023949 S Bi:6:006:1 -115 4096 <
ffff9067b478e000 921025496 C Bi:6:006:1 0 4096 = 04800000 04800100 04800200 04800300 04800400 04800c00 04800d00 04801800
ffff90673f63a3c0 921025552 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921025592 C Bi:6:006:1 0 13 = 55534253 a4000000 00000000 00
ffff90673f63a3c0 921025784 S Bo:6:006:2 -115 31 = 55534243 a5000000 00100000 80000a28 00000008 40000008 00000000 000000
ffff90673f63a3c0 921025811 C Bo:6:006:2 0 31 >
ffff9067b478e000 921025837 S Bi:6:006:1 -115 4096 <
ffff9067b478e000 921026488 C Bi:6:006:1 0 4096 = 08800000 08800100 08800200 08800300 08800400 08800c00 08800d00 08801800
ffff90673f63a3c0 921026555 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921026591 C Bi:6:006:1 0 13 = 55534253 a5000000 00000000 00
ffff90673f63a3c0 921026793 S Bo:6:006:2 -115 31 = 55534243 a6000000 00100000 80000a28 00000008 80000008 00000000 000000
ffff90673f63a3c0 921026826 C Bo:6:006:2 0 31 >
ffff9067b478e000 921026847 S Bi:6:006:1 -115 4096 <
ffff9067b478e000 921027526 C Bi:6:006:1 0 4096 = 10800000 10800100 10800200 10800300 10800400 10800c00 10800d00 10801800
ffff90673f63a3c0 921027568 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921027584 C Bi:6:006:1 0 13 = 55534253 a6000000 00000000 00
ffff90673f63a3c0 921027713 S Bo:6:006:2 -115 31 = 55534243 a7000000 00100000 80000a28 00000009 00000008 00000000 000000
ffff90673f63a3c0 921027784 C Bo:6:006:2 0 31 >
ffff9067b478e000 921027823 S Bi:6:006:1 -115 4096 <
ffff9067b478e000 921028471 C Bi:6:006:1 0 4096 = 20800000 20800100 20800200 20800300 20800400 20800c00 20800d00 20801800
ffff90673f63a3c0 921028516 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921028529 C Bi:6:006:1 0 13 = 55534253 a7000000 00000000 00
ffff90673f63a3c0 921028699 S Bo:6:006:2 -115 31 = 55534243 a8000000 00100000 80000a28 0000000a 00000008 00000000 000000
ffff90673f63a3c0 921028773 C Bo:6:006:2 0 31 >
ffff9067b478e000 921028815 S Bi:6:006:1 -115 4096 <
ffff9067b478e000 921029429 C Bi:6:006:1 0 4096 = 40800000 40800100 40800200 40800300 40800400 40800c00 40800d00 40801800
ffff90673f63a3c0 921029446 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921029459 C Bi:6:006:1 0 13 = 55534253 a8000000 00000000 00
ffff90673f63a3c0 921029539 S Bo:6:006:2 -115 31 = 55534243 a9000000 00100000 80000a28 0000000c 00000008 00000000 000000
ffff90673f63a3c0 921029561 C Bo:6:006:2 0 31 >
ffff9067b478e000 921029587 S Bi:6:006:1 -115 4096 <
ffff9067b478e000 921030299 C Bi:6:006:1 0 4096 = 80800000 80800100 80800200 80800300 80800400 80800c00 80800d00 80801800
ffff90673f63a3c0 921030314 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921030332 C Bi:6:006:1 0 13 = 55534253 a9000000 00000000 00
ffff90673f63a3c0 921030438 S Bo:6:006:2 -115 31 = 55534243 aa000000 00100000 80000a28 00000010 00000008 00000000 000000
ffff90673f63a3c0 921030463 C Bo:6:006:2 0 31 >
ffff9067b478e000 921030471 S Bi:6:006:1 -115 4096 <
ffff9067b478e000 921031193 C Bi:6:006:1 0 4096 = 00810000 00810100 00810200 00810300 00810400 00810c00 00810d00 00811800
ffff90673f63a3c0 921031208 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921031217 C Bi:6:006:1 0 13 = 55534253 aa000000 00000000 00
ffff90673f63a3c0 921031318 S Bo:6:006:2 -115 31 = 55534243 ab000000 00100000 80000a28 00000018 00000008 00000000 000000
ffff90673f63a3c0 921031338 C Bo:6:006:2 0 31 >
ffff9067b478e000 921031345 S Bi:6:006:1 -115 4096 <
ffff9067b478e000 921032114 C Bi:6:006:1 0 4096 = 00820000 00820100 00820200 00820300 00820400 00820c00 00820d00 00821800
ffff90673f63a3c0 921032172 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921032187 C Bi:6:006:1 0 13 = 55534253 ab000000 00000000 00
ffff90673f63a3c0 921032369 S Bo:6:006:2 -115 31 = 55534243 ac000000 00100000 80000a28 00000028 00000008 00000000 000000
ffff90673f63a3c0 921032412 C Bo:6:006:2 0 31 >
ffff9067b478e000 921032459 S Bi:6:006:1 -115 4096 <
ffff9067b478e000 921033169 C Bi:6:006:1 0 4096 = 00840000 00840100 00840200 00840300 00840400 00840c00 00840d00 00841800
ffff90673f63a3c0 921033225 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921033241 C Bi:6:006:1 0 13 = 55534253 ac000000 00000000 00
ffff90673f63a3c0 921033466 S Bo:6:006:2 -115 31 = 55534243 ad000000 00100000 80000a28 00000008 18000008 00000000 000000
ffff90673f63a3c0 921033497 C Bo:6:006:2 0 31 >
ffff9067b478e000 921033524 S Bi:6:006:1 -115 4096 <
ffff9067b478e000 921034229 C Bi:6:006:1 0 4096 = 03800000 03800100 03800200 03800300 03800400 03800c00 03800d00 03801800
ffff90673f63a3c0 921034270 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921034284 C Bi:6:006:1 0 13 = 55534253 ad000000 00000000 00
ffff90673f63a3c0 921034476 S Bo:6:006:2 -115 31 = 55534243 ae000000 00100000 80000a28 00000008 38000008 00000000 000000
ffff90673f63a3c0 921034551 C Bo:6:006:2 0 31 >
ffff9067b478e000 921034592 S Bi:6:006:1 -115 4096 <
ffff9067b478e000 921035212 C Bi:6:006:1 0 4096 = 07800000 07800100 07800200 07800300 07800400 07800c00 07800d00 07801800
ffff90673f63a3c0 921035254 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921035275 C Bi:6:006:1 0 13 = 55534253 ae000000 00000000 00
ffff90673f63a3c0 921035429 S Bo:6:006:2 -115 31 = 55534243 af000000 00100000 80000a28 00000008 78000008 00000000 000000
ffff90673f63a3c0 921035505 C Bo:6:006:2 0 31 >
ffff9067b478e000 921035546 S Bi:6:006:1 -115 4096 <
ffff9067b478e000 921036159 C Bi:6:006:1 0 4096 = 0f800000 0f800100 0f800200 0f800300 0f800400 0f800c00 0f800d00 0f801800
ffff90673f63a3c0 921036203 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921036223 C Bi:6:006:1 0 13 = 55534253 af000000 00000000 00
ffff90673f63a3c0 921036414 S Bo:6:006:2 -115 31 = 55534243 b0000000 00100000 80000a28 00000008 10000008 00000000 000000
ffff90673f63a3c0 921036452 C Bo:6:006:2 0 31 >
ffff9067b478e000 921036499 S Bi:6:006:1 -115 4096 <
ffff9067b478e000 921037142 C Bi:6:006:1 0 4096 = 00004000 10004000 20004000 f05ff01f 00000500 00000000 00000000 f01f7270
ffff90673f63a3c0 921037183 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921037209 C Bi:6:006:1 0 13 = 55534253 b0000000 00000000 00
ffff90673f63a3c0 921037273 S Bo:6:006:2 -115 31 = 55534243 b1000000 00200000 80000a28 00000008 28000010 00000000 000000
ffff90673f63a3c0 921037293 C Bo:6:006:2 0 31 >
ffff9067c0aac3c0 921037300 S Bi:6:006:1 -115 8192 <
ffff9067c0aac3c0 921038065 C Bi:6:006:1 0 8192 = 05800000 05800100 05800200 05800300 05800400 05800c00 05800d00 05801800
ffff90673f63a3c0 921038076 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921038137 C Bi:6:006:1 0 13 = 55534253 b1000000 00000000 00
ffff90673f63a3c0 921038182 S Bo:6:006:2 -115 31 = 55534243 b2000000 00600000 80000a28 00000008 48000030 00000000 000000
ffff90673f63a3c0 921038195 C Bo:6:006:2 0 31 >
ffff9067c0aac3c0 921038202 S Bi:6:006:1 -115 24576 <
ffff9067c0aac3c0 921039306 C Bi:6:006:1 0 24576 = 09800000 09800100 09800200 09800300 09800400 09800c00 09800d00 09801800
ffff90673f63a3c0 921039315 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921039344 C Bi:6:006:1 0 13 = 55534253 b2000000 00000000 00
ffff90673f63a3c0 921039449 S Bo:6:006:2 -115 31 = 55534243 b3000000 00f00000 80000a28 00000008 88000078 00000000 000000
ffff90673f63a3c0 921039479 C Bo:6:006:2 0 31 >
ffff9067aedb4cc0 921039503 S Bi:6:006:1 -115 61440 <
ffff9067aedb4cc0 921041539 C Bi:6:006:1 0 61440 = 11800000 11800100 11800200 11800300 11800400 11800c00 11800d00 11801800
ffff90673f63a3c0 921041586 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921041600 C Bi:6:006:1 0 13 = 55534253 b3000000 00000000 00
ffff90673f63a3c0 921041678 S Bo:6:006:2 -115 31 = 55534243 b4000000 00100000 80000a28 00000009 08000008 00000000 000000
ffff90673f63a3c0 921041703 C Bo:6:006:2 0 31 >
ffff9067b478e000 921041735 S Bi:6:006:1 -115 4096 <
ffff9067b478e000 921042404 C Bi:6:006:1 0 4096 = 21800000 21800100 21800200 21800300 21800400 21800c00 21800d00 21801800
ffff90673f63a3c0 921042421 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921042440 C Bi:6:006:1 0 13 = 55534253 b4000000 00000000 00
ffff90673f63a3c0 921042685 S Bo:6:006:2 -115 31 = 55534243 b5000000 00e00100 80000a28 00000009 100000f0 00000000 000000
ffff90673f63a3c0 921042740 C Bo:6:006:2 0 31 >
ffff9067aedb4cc0 921042781 S Bi:6:006:1 -115 122880 <
ffff9067aedb4cc0 921046356 C Bi:6:006:1 0 122880 = 22800000 22800100 22800200 22800300 22800400 22800c00 22800d00 22801800
ffff90673f63a3c0 921046408 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921046450 C Bi:6:006:1 0 13 = 55534253 b5000000 00000000 00
ffff90673f63a3c0 921046856 S Bo:6:006:2 -115 31 = 55534243 b6000000 00e00100 80000a28 0000000a 080000f0 00000000 000000
ffff90673f63a3c0 921046897 C Bo:6:006:2 0 31 >
ffff9067aedb4cc0 921046944 S Bi:6:006:1 -115 122880 <
ffff9067aedb4cc0 921050599 C Bi:6:006:1 0 122880 = 41800000 41800100 41800200 41800300 41800400 41800c00 41800d00 41801800
ffff90673f63a3c0 921050663 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921050708 C Bi:6:006:1 0 13 = 55534253 b6000000 00000000 00
ffff90673f63a3c0 921050799 S Bo:6:006:2 -115 31 = 55534243 b7000000 00200000 80000a28 0000000a f8000010 00000000 000000
ffff90673f63a3c0 921050836 C Bo:6:006:2 0 31 >
ffff9067aedb4cc0 921050855 S Bi:6:006:1 -115 8192 <
ffff9067aedb4cc0 921051676 C Bi:6:006:1 0 8192 = 5f800000 5f800100 5f800200 5f800300 5f800400 5f800c00 5f800d00 5f801800
ffff90673f63a3c0 921051758 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921051784 C Bi:6:006:1 0 13 = 55534253 b7000000 00000000 00
ffff90673f63a3c0 921052192 S Bo:6:006:2 -115 31 = 55534243 b8000000 00e00100 80000a28 0000000b 080000f0 00000000 000000
ffff90673f63a3c0 921052252 C Bo:6:006:2 0 31 >
ffff9067aedb4cc0 921052285 S Bi:6:006:1 -115 122880 <
ffff9067aedb4cc0 921055928 C Bi:6:006:1 0 122880 = 61800000 61800100 61800200 61800300 61800400 61800c00 61800d00 61801800
ffff90673f63a3c0 921055991 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921056038 C Bi:6:006:1 0 13 = 55534253 b8000000 00000000 00
ffff90673f63a3c0 921056123 S Bo:6:006:2 -115 31 = 55534243 b9000000 00100000 80000a28 0000000b f8000008 00000000 000000
ffff90673f63a3c0 921056162 C Bo:6:006:2 0 31 >
ffff9067aedb4cc0 921056192 S Bi:6:006:1 -115 4096 <
ffff9067aedb4cc0 921056331 C Bi:6:006:1 0 4096 = 7f800000 7f800100 7f800200 7f800300 7f800400 7f800c00 7f800d00 7f801800
ffff90673f63a3c0 921056379 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921056419 C Bi:6:006:1 0 13 = 55534253 b9000000 00000000 00
ffff90673f63a3c0 921056854 S Bo:6:006:2 -115 31 = 55534243 ba000000 00e00100 80000a28 00039a2e 000000f0 00000000 000000
ffff90673f63a3c0 921056914 C Bo:6:006:2 0 31 >
ffff9067aedb4cc0 921056959 S Bi:6:006:1 -115 122880 <
ffff9067aedb4cc0 921061444 C Bi:6:006:1 0 122880 = c77ccf3d b18bbc20 49a7c771 ab78fd88 353bdfd1 c2f59b72 ee1c7082 2c62c6f6
ffff90673f63a3c0 921061545 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921061592 C Bi:6:006:1 0 13 = 55534253 ba000000 00000000 00
ffff90673f63a3c0 921061693 S Bo:6:006:2 -115 31 = 55534243 bb000000 00200000 80000a28 00039a2e f0000010 00000000 000000
ffff90673f63a3c0 921061750 C Bo:6:006:2 0 31 >
ffff9067aedb4cc0 921061781 S Bi:6:006:1 -115 8192 <
ffff9067aedb4cc0 921062505 C Bi:6:006:1 0 8192 = 2d1e8c4f 0d6d6356 384be8bc b5cfb50e cdb3025f bcfecddd 4f80d2e5 eb6b3284
ffff90673f63a3c0 921062582 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921062624 C Bi:6:006:1 0 13 = 55534253 bb000000 00000000 00
ffff90673f63a3c0 921063069 S Bo:6:006:2 -115 31 = 55534243 bc000000 00e00100 80000a28 00039a2f 000000f0 00000000 000000
ffff90673f63a3c0 921063122 C Bo:6:006:2 0 31 >
ffff9067aedb4cc0 921063166 S Bi:6:006:1 -115 122880 <
ffff9067aedb4cc0 921066839 C Bi:6:006:1 0 122880 = aa1c9eea 8ab1cada d7bd4902 c36a01ea 724ec451 e5509315 ca093a94 9dcab358
ffff90673f63a3c0 921066902 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921066930 C Bi:6:006:1 0 13 = 55534253 bc000000 00000000 00
ffff90673f63a3c0 921067011 S Bo:6:006:2 -115 31 = 55534243 bd000000 00200000 80000a28 00039a2f f0000010 00000000 000000
ffff90673f63a3c0 921067063 C Bo:6:006:2 0 31 >
ffff9067aedb4cc0 921067086 S Bi:6:006:1 -115 8192 <
ffff9067aedb4cc0 921067304 C Bi:6:006:1 0 8192 = 6ddcf87d 817edc15 a527992a 77e6c620 1adc6410 77ef845a d44f3327 5ac54da3
ffff90673f63a3c0 921067335 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921067348 C Bi:6:006:1 0 13 = 55534253 bd000000 00000000 00
ffff90673f63a3c0 921067777 S Bo:6:006:2 -115 31 = 55534243 be000000 00e00100 80000a28 00039a30 000000f0 00000000 000000
ffff90673f63a3c0 921067861 C Bo:6:006:2 0 31 >
ffff9067aedb4cc0 921067901 S Bi:6:006:1 -115 122880 <
ffff9067aedb4cc0 921071059 C Bi:6:006:1 0 122880 = c2bb8f02 e6227d01 1316d5d4 f8589196 3738834e 7c5c90cb 17868ef4 9eb65148
ffff90673f63a3c0 921071126 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921071176 C Bi:6:006:1 0 13 = 55534253 be000000 00000000 00
ffff90673f63a3c0 921071287 S Bo:6:006:2 -115 31 = 55534243 bf000000 00200000 80000a28 00039a30 f0000010 00000000 000000
ffff90673f63a3c0 921071322 C Bo:6:006:2 0 31 >
ffff9067aedb4cc0 921071345 S Bi:6:006:1 -115 8192 <
ffff9067aedb4cc0 921071545 C Bi:6:006:1 0 8192 = 2378ad20 bc2a7d26 58ef9fc8 48429609 72a99082 f13ed042 ba93cad1 de25670f
ffff90673f63a3c0 921071600 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921071631 C Bi:6:006:1 0 13 = 55534253 bf000000 00000000 00
ffff90673f63a3c0 921072063 S Bo:6:006:2 -115 31 = 55534243 c0000000 00e00100 80000a28 00039a31 000000f0 00000000 000000
ffff90673f63a3c0 921072091 C Bo:6:006:2 0 31 >
ffff9067aedb4cc0 921072106 S Bi:6:006:1 -115 122880 <
ffff9067aedb4cc0 921075218 C Bi:6:006:1 0 122880 = 315bbba0 afae98ec 6e222b9d ba2bc3bd 1a3f0717 1688e8e5 c30b4584 6f4f58ae
ffff90673f63a3c0 921075272 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921075289 C Bi:6:006:1 0 13 = 55534253 c0000000 00000000 00
ffff90673f63a3c0 921075339 S Bo:6:006:2 -115 31 = 55534243 c1000000 00200000 80000a28 00039a31 f0000010 00000000 000000
ffff90673f63a3c0 921075417 C Bo:6:006:2 0 31 >
ffff9067aedb4cc0 921075441 S Bi:6:006:1 -115 8192 <
ffff9067aedb4cc0 921075633 C Bi:6:006:1 0 8192 = 15c5f9a5 4fdcf7b5 d4013ce6 82011093 87fb73d9 5b8553c4 c7965bfb e4ae9642
ffff90673f63a3c0 921075683 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921075708 C Bi:6:006:1 0 13 = 55534253 c1000000 00000000 00
ffff90673f63a3c0 921076287 S Bo:6:006:2 -115 31 = 55534243 c2000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 921076313 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 921076327 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921076355 C Bi:6:006:1 0 13 = 55534253 c2000000 00000000 00
ffff90673f63a3c0 921076613 S Bo:6:006:2 -115 31 = 55534243 c3000000 00100000 80000a28 00000000 00000008 00000000 000000
ffff90673f63a3c0 921076633 C Bo:6:006:2 0 31 >
ffff9067c0aac3c0 921076668 S Bi:6:006:1 -115 4096 <
ffff9067c0aac3c0 921078132 C Bi:6:006:1 0 4096 = fab80000 8ed0bc00 7c8bf450 07501ffb fcbf0006 b90001f3 a5ea1e06 0000bebe
ffff90673f63a3c0 921078149 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921078170 C Bi:6:006:1 0 13 = 55534253 c3000000 00000000 00
ffff90673f63a3c0 921078483 S Bo:6:006:2 -115 31 = 55534243 c4000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 921078510 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 921078517 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921078548 C Bi:6:006:1 0 13 = 55534253 c4000000 00000000 00
ffff90673f63a3c0 921078998 S Bo:6:006:2 -115 31 = 55534243 c5000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 921079014 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 921079021 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921079053 C Bi:6:006:1 0 13 = 55534253 c5000000 00000000 00
ffff90673f63a3c0 921079433 S Bo:6:006:2 -115 31 = 55534243 c6000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 921079476 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 921079486 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921079498 C Bi:6:006:1 0 13 = 55534253 c6000000 00000000 00
ffff90673f63a3c0 921079646 S Bo:6:006:2 -115 31 = 55534243 c7000000 00040000 80000a28 00000008 02000002 00000000 000000
ffff90673f63a3c0 921079662 C Bo:6:006:2 0 31 >
ffff9067c0aac3c0 921079670 S Bi:6:006:1 -115 1024 <
ffff9067c0aac3c0 921080294 C Bi:6:006:1 0 1024 = 90d11c00 7e457300 79c30500 2fa65700 7fd11c00 00000000 02000000 02000000
ffff90673f63a3c0 921080306 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921080334 C Bi:6:006:1 0 13 = 55534253 c7000000 00000000 00
ffff90673f63a3c0 921080463 S Bo:6:006:2 -115 31 = 55534243 c8000000 00100000 80000a28 00000008 00000008 00000000 000000
ffff90673f63a3c0 921080482 C Bo:6:006:2 0 31 >
ffff9067c0aac3c0 921080511 S Bi:6:006:1 -115 4096 <
ffff9067c0aac3c0 921081185 C Bi:6:006:1 0 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff90673f63a3c0 921081199 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921081223 C Bi:6:006:1 0 13 = 55534253 c8000000 00000000 00
ffff90673f63a3c0 921081288 S Bo:6:006:2 -115 31 = 55534243 c9000000 00100000 80000a28 00000008 08000008 00000000 000000
ffff90673f63a3c0 921081303 C Bo:6:006:2 0 31 >
ffff9067c0aac3c0 921081310 S Bi:6:006:1 -115 4096 <
ffff9067c0aac3c0 921082022 C Bi:6:006:1 0 4096 = 01040000 11040000 21040000 e85bdf1f 02000400 00000000 00000000 dc1f8466
ffff90673f63a3c0 921082035 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921082046 C Bi:6:006:1 0 13 = 55534253 c9000000 00000000 00
ffff90673f63a3c0 921082075 S Bo:6:006:2 -115 31 = 55534243 ca000000 00100000 80000a28 00000008 10000008 00000000 000000
ffff90673f63a3c0 921082088 C Bo:6:006:2 0 31 >
ffff9067c0aac3c0 921082095 S Bi:6:006:1 -115 4096 <
ffff9067c0aac3c0 921082216 C Bi:6:006:1 0 4096 = 00004000 10004000 20004000 f05ff01f 00000500 00000000 00000000 f01f7270
ffff90673f63a3c0 921082225 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921082256 C Bi:6:006:1 0 13 = 55534253 ca000000 00000000 00
ffff90673f63a3c0 921082468 S Bo:6:006:2 -115 31 = 55534243 cb000000 00100000 80000a28 00000029 10000008 00000000 000000
ffff90673f63a3c0 921082511 C Bo:6:006:2 0 31 >
ffff9067c0aac3c0 921082522 S Bi:6:006:1 -115 4096 <
ffff9067c0aac3c0 921083198 C Bi:6:006:1 0 4096 = a4810000 0080841e 2a0d4d5d 7b3c4d5d 7b3c4d5d 00000000 00000100 40420f00
ffff90673f63a3c0 921083212 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921083223 C Bi:6:006:1 0 13 = 55534253 cb000000 00000000 00
ffff90673f63a3c0 921083245 S Bo:6:006:2 -115 31 = 55534243 cc000000 00100000 80000a28 00000029 08000008 00000000 000000
ffff90673f63a3c0 921083262 C Bo:6:006:2 0 31 >
ffff9067c0aac3c0 921083269 S Bi:6:006:1 -115 4096 <
ffff9067c0aac3c0 921083991 C Bi:6:006:1 0 4096 = 00000000 00000000 33f72257 33f72257 33f72257 00000000 00000000 00000000
ffff90673f63a3c0 921084031 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921084040 C Bi:6:006:1 0 13 = 55534253 cc000000 00000000 00
ffff90673f63a3c0 921084127 S Bo:6:006:2 -115 31 = 55534243 cd000000 00100000 80000a28 0001c408 00000008 00000000 000000
ffff90673f63a3c0 921084143 C Bo:6:006:2 0 31 >
ffff9067c0aac3c0 921084149 S Bi:6:006:1 -115 4096 <
ffff9067c0aac3c0 921085710 C Bi:6:006:1 0 4096 = c03b3998 00000004 00000000 00001000 00008000 00000001 00031061 00000000
ffff90673f63a3c0 921085723 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921085748 C Bi:6:006:1 0 13 = 55534253 cd000000 00000000 00
ffff90673f63a3c0 921086900 S Bo:6:006:2 -115 31 = 55534243 ce000000 00100000 00000a2a 00000008 00000008 00000000 000000
ffff90673f63a3c0 921086953 C Bo:6:006:2 0 31 >
ffff9067c0aac3c0 921086976 S Bo:6:006:2 -115 4096 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff9067c0aac3c0 921087119 C Bo:6:006:2 0 4096 >
ffff90673f63a3c0 921087131 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921087140 C Bi:6:006:1 0 13 = 55534253 ce000000 00000000 00
ffff90673f63a3c0 921094817 S Bo:6:006:2 -115 31 = 55534243 cf000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 921094845 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 921094851 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 921094890 C Bi:6:006:1 0 13 = 55534253 cf000000 00000000 00
ffff9067b1774780 921582503 S Bo:6:003:1 -115 31 = 55534243 c10e0000 00000000 00030600 00000000 00000000 00000000 000000
ffff9067b1774780 921582595 C Bo:6:003:1 0 31 >
ffff9067b1774780 921582607 S Bi:6:003:2 -115 13 <
ffff9067b1774780 921582968 C Bi:6:003:2 0 13 = 55534253 c10e0000 00000000 01
ffff9067b1774780 921582990 S Bo:6:003:1 -115 31 = 55534243 c20e0000 12000000 80030603 00000012 00000000 00000000 000000
ffff9067b1774780 921583042 C Bo:6:003:1 0 31 >
ffff9067c0aac600 921583054 S Bi:6:003:2 -115 18 <
ffff9067c0aac600 921583647 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 921583671 S Bi:6:003:2 -115 13 <
ffff9067b1774780 921583735 C Bi:6:003:2 0 13 = 55534253 c20e0000 00000000 00
ffff9067b1774780 921583814 S Bo:6:003:1 -115 31 = 55534243 c30e0000 00000000 00020600 00000000 00000000 00000000 000000
ffff9067b1774780 921583877 C Bo:6:003:1 0 31 >
ffff9067b1774780 921583918 S Bi:6:003:2 -115 13 <
ffff9067b1774780 921584266 C Bi:6:003:2 0 13 = 55534253 c30e0000 00000000 01
ffff9067b1774780 921584312 S Bo:6:003:1 -115 31 = 55534243 c40e0000 12000000 80020603 00000012 00000000 00000000 000000
ffff9067b1774780 921584368 C Bo:6:003:1 0 31 >
ffff9067aedb4cc0 921584387 S Bi:6:003:2 -115 18 <
ffff9067aedb4cc0 921584933 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 921585002 S Bi:6:003:2 -115 13 <
ffff9067b1774780 921585044 C Bi:6:003:2 0 13 = 55534253 c40e0000 00000000 00
ffff9067b1774780 921585115 S Bo:6:003:1 -115 31 = 55534243 c50e0000 00000000 00010600 00000000 00000000 00000000 000000
ffff9067b1774780 921585162 C Bo:6:003:1 0 31 >
ffff9067b1774780 921585168 S Bi:6:003:2 -115 13 <
ffff9067b1774780 921585603 C Bi:6:003:2 0 13 = 55534253 c50e0000 00000000 01
ffff9067b1774780 921585617 S Bo:6:003:1 -115 31 = 55534243 c60e0000 12000000 80010603 00000012 00000000 00000000 000000
ffff9067b1774780 921585668 C Bo:6:003:1 0 31 >
ffff9067c0aac600 921585679 S Bi:6:003:2 -115 18 <
ffff9067c0aac600 921586194 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 921586208 S Bi:6:003:2 -115 13 <
ffff9067b1774780 921586332 C Bi:6:003:2 0 13 = 55534253 c60e0000 00000000 00
ffff9067b1774780 921586399 S Bo:6:003:1 -115 31 = 55534243 c70e0000 00000000 00000600 00000000 00000000 00000000 000000
ffff9067b1774780 921586448 C Bo:6:003:1 0 31 >
ffff9067b1774780 921586470 S Bi:6:003:2 -115 13 <
ffff9067b1774780 921586857 C Bi:6:003:2 0 13 = 55534253 c70e0000 00000000 01
ffff9067b1774780 921587075 S Bo:6:003:1 -115 31 = 55534243 c80e0000 12000000 80000603 00000012 00000000 00000000 000000
ffff9067b1774780 921587134 C Bo:6:003:1 0 31 >
ffff9067aedb4cc0 921587174 S Bi:6:003:2 -115 18 <
ffff9067aedb4cc0 921587677 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 921587750 S Bi:6:003:2 -115 13 <
ffff9067b1774780 921587789 C Bi:6:003:2 0 13 = 55534253 c80e0000 00000000 00
ffff9067b1774b40 921768761 C Ii:6:004:1 0:2 8 = 0000f6ff e9ff0000
ffff9067b1774b40 921768798 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921776755 C Ii:6:004:1 0:2 8 = 0000f6ff e7ff0000
ffff9067b1774b40 921776780 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921784754 C Ii:6:004:1 0:2 8 = 0000f9ff edff0000
ffff9067b1774b40 921784778 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921792727 C Ii:6:004:1 0:2 8 = 0000fdff f8ff0000
ffff9067b1774b40 921792754 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921814755 C Ii:6:004:1 0:2 8 = 00000500 02000000
ffff9067b1774b40 921814785 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921822756 C Ii:6:004:1 0:2 8 = 00000800 02000000
ffff9067b1774b40 921822783 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921830755 C Ii:6:004:1 0:2 8 = 00000600 01000000
ffff9067b1774b40 921830784 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921838755 C Ii:6:004:1 0:2 8 = 00000400 00000000
ffff9067b1774b40 921838782 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921846756 C Ii:6:004:1 0:2 8 = 00000300 00000000
ffff9067b1774b40 921846779 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921854757 C Ii:6:004:1 0:2 8 = 00000000 fdff0000
ffff9067b1774b40 921854788 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921862755 C Ii:6:004:1 0:2 8 = 0000feff fdff0000
ffff9067b1774b40 921862781 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921870754 C Ii:6:004:1 0:2 8 = 0000fcff fbff0000
ffff9067b1774b40 921870780 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921878754 C Ii:6:004:1 0:2 8 = 0000fbff f7ff0000
ffff9067b1774b40 921878783 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921884706 C Ii:6:004:1 0:2 8 = 0000f9ff f5ff0000
ffff9067b1774b40 921884731 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921892754 C Ii:6:004:1 0:2 8 = 0000f8ff f3ff0000
ffff9067b1774b40 921892781 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921900754 C Ii:6:004:1 0:2 8 = 0000f9ff f2ff0000
ffff9067b1774b40 921900780 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921908754 C Ii:6:004:1 0:2 8 = 0000f7ff f1ff0000
ffff9067b1774b40 921908782 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921916751 C Ii:6:004:1 0:2 8 = 0000f7ff f4ff0000
ffff9067b1774b40 921916779 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921924723 C Ii:6:004:1 0:2 8 = 0000f9ff f6ff0000
ffff9067b1774b40 921924750 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921932755 C Ii:6:004:1 0:2 8 = 0000fbff fbff0000
ffff9067b1774b40 921932784 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921940724 C Ii:6:004:1 0:2 8 = 0000fdff feff0000
ffff9067b1774b40 921940748 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 921948753 C Ii:6:004:1 0:2 8 = 0000ffff 00000000
ffff9067b1774b40 921948781 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922228761 C Ii:6:004:1 0:2 8 = 0000fcff 02000000
ffff9067b1774b40 922228827 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922236757 C Ii:6:004:1 0:2 8 = 0000faff 02000000
ffff9067b1774b40 922236819 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922244761 C Ii:6:004:1 0:2 8 = 0000f7ff 00000000
ffff9067b1774b40 922244828 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922252756 C Ii:6:004:1 0:2 8 = 0000f6ff fcff0000
ffff9067b1774b40 922252812 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922260761 C Ii:6:004:1 0:2 8 = 0000f2ff f9ff0000
ffff9067b1774b40 922260824 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922268760 C Ii:6:004:1 0:2 8 = 0000efff f8ff0000
ffff9067b1774b40 922268827 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922276760 C Ii:6:004:1 0:2 8 = 0000ecff f6ff0000
ffff9067b1774b40 922276822 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922284760 C Ii:6:004:1 0:2 8 = 0000e7ff f4ff0000
ffff9067b1774b40 922284822 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922292755 C Ii:6:004:1 0:2 8 = 0000e7ff efff0000
ffff9067b1774b40 922292816 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922300761 C Ii:6:004:1 0:2 8 = 0000e5ff f1ff0000
ffff9067b1774b40 922300813 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922306760 C Ii:6:004:1 0:2 8 = 0000e1ff f2ff0000
ffff9067b1774b40 922306828 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922316756 C Ii:6:004:1 0:2 8 = 0000e1ff f2ff0000
ffff9067b1774b40 922316831 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922322729 C Ii:6:004:1 0:2 8 = 0000e4ff f3ff0000
ffff9067b1774b40 922322763 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922330754 C Ii:6:004:1 0:2 8 = 0000e7ff f5ff0000
ffff9067b1774b40 922330800 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922338754 C Ii:6:004:1 0:2 8 = 0000ebff f7ff0000
ffff9067b1774b40 922338814 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922346760 C Ii:6:004:1 0:2 8 = 0000ebff faff0000
ffff9067b1774b40 922346816 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922354764 C Ii:6:004:1 0:2 8 = 0000f0ff fcff0000
ffff9067b1774b40 922354817 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922362746 C Ii:6:004:1 0:2 8 = 0000f2ff fdff0000
ffff9067b1774b40 922362775 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922370691 C Ii:6:004:1 0:2 8 = 0000f5ff ffff0000
ffff9067b1774b40 922370714 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922378745 C Ii:6:004:1 0:2 8 = 0000f8ff 00000000
ffff9067b1774b40 922378775 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922384744 C Ii:6:004:1 0:2 8 = 0000fbff 00000000
ffff9067b1774b40 922384771 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922392745 C Ii:6:004:1 0:2 8 = 0000ffff 00000000
ffff9067b1774b40 922392773 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922510757 C Ii:6:004:1 0:2 8 = 01000000 00000000
ffff9067b1774b40 922510810 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922572744 C Ii:6:004:1 0:2 8 = 00000000 00000000
ffff9067b1774b40 922572782 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922970706 C Ii:6:004:1 0:2 8 = 00000200 00000000
ffff9067b1774b40 922970733 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922978735 C Ii:6:004:1 0:2 8 = 00000800 02000000
ffff9067b1774b40 922978767 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922986734 C Ii:6:004:1 0:2 8 = 00000f00 05000000
ffff9067b1774b40 922986766 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 922994740 C Ii:6:004:1 0:2 8 = 00001000 08000000
ffff9067b1774b40 922994782 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 923002742 C Ii:6:004:1 0:2 8 = 00001400 09000000
ffff9067b1774b40 923002786 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 923010749 C Ii:6:004:1 0:2 8 = 00001500 09000000
ffff9067b1774b40 923010799 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 923018749 C Ii:6:004:1 0:2 8 = 00001400 09000000
ffff9067b1774b40 923018802 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 923026748 C Ii:6:004:1 0:2 8 = 00001500 0a000000
ffff9067b1774b40 923026798 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 923034744 C Ii:6:004:1 0:2 8 = 00001300 0a000000
ffff9067b1774b40 923034794 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 923042748 C Ii:6:004:1 0:2 8 = 00001200 0d000000
ffff9067b1774b40 923042799 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 923048744 C Ii:6:004:1 0:2 8 = 00001000 09000000
ffff9067b1774b40 923048793 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 923056750 C Ii:6:004:1 0:2 8 = 00000e00 09000000
ffff9067b1774b40 923056805 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 923064750 C Ii:6:004:1 0:2 8 = 00000c00 07000000
ffff9067b1774b40 923064802 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 923072746 C Ii:6:004:1 0:2 8 = 00000a00 05000000
ffff9067b1774b40 923072798 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 923080749 C Ii:6:004:1 0:2 8 = 00000900 03000000
ffff9067b1774b40 923080804 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 923088753 C Ii:6:004:1 0:2 8 = 00000700 02000000
ffff9067b1774b40 923088810 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 923096748 C Ii:6:004:1 0:2 8 = 00000500 00000000
ffff9067b1774b40 923096805 S Ii:6:004:1 -115:2 8 <
ffff9067b1774b40 923104748 C Ii:6:004:1 0:2 8 = 00000300 00000000
ffff9067b1774b40 923104797 S Ii:6:004:1 -115:2 8 <
ffff90673f63a3c0 923118536 S Bo:6:006:2 -115 31 = 55534243 d0000000 00000000 00000600 00000000 00000000 00000000 000000
ffff90673f63a3c0 923118581 C Bo:6:006:2 0 31 >
ffff90673f63a3c0 923118594 S Bi:6:006:1 -115 13 <
ffff90673f63a3c0 923118612 C Bi:6:006:1 0 13 = 55534253 d0000000 00000000 00
ffff9067b1774780 923630497 S Bo:6:003:1 -115 31 = 55534243 c90e0000 00000000 00000600 00000000 00000000 00000000 000000
ffff9067b1774780 923630592 C Bo:6:003:1 0 31 >
ffff9067b1774780 923630631 S Bi:6:003:2 -115 13 <
ffff9067b1774780 923630988 C Bi:6:003:2 0 13 = 55534253 c90e0000 00000000 01
ffff9067b1774780 923631057 S Bo:6:003:1 -115 31 = 55534243 ca0e0000 12000000 80000603 00000012 00000000 00000000 000000
ffff9067b1774780 923631155 C Bo:6:003:1 0 31 >
ffff9067aedb40c0 923631224 S Bi:6:003:2 -115 18 <
ffff9067aedb40c0 923631675 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 923631744 S Bi:6:003:2 -115 13 <
ffff9067b1774780 923631792 C Bi:6:003:2 0 13 = 55534253 ca0e0000 00000000 00
ffff9067b1774780 923631866 S Bo:6:003:1 -115 31 = 55534243 cb0e0000 00000000 00010600 00000000 00000000 00000000 000000
ffff9067b1774780 923631918 C Bo:6:003:1 0 31 >
ffff9067b1774780 923631927 S Bi:6:003:2 -115 13 <
ffff9067b1774780 923632330 C Bi:6:003:2 0 13 = 55534253 cb0e0000 00000000 01
ffff9067b1774780 923632346 S Bo:6:003:1 -115 31 = 55534243 cc0e0000 12000000 80010603 00000012 00000000 00000000 000000
ffff9067b1774780 923632382 C Bo:6:003:1 0 31 >
ffff90673f46fb40 923632415 S Bi:6:003:2 -115 18 <
ffff90673f46fb40 923632947 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 923632985 S Bi:6:003:2 -115 13 <
ffff9067b1774780 923633061 C Bi:6:003:2 0 13 = 55534253 cc0e0000 00000000 00
ffff9067b1774780 923633123 S Bo:6:003:1 -115 31 = 55534243 cd0e0000 00000000 00020600 00000000 00000000 00000000 000000
ffff9067b1774780 923633185 C Bo:6:003:1 0 31 >
ffff9067b1774780 923633219 S Bi:6:003:2 -115 13 <
ffff9067b1774780 923633573 C Bi:6:003:2 0 13 = 55534253 cd0e0000 00000000 01
ffff9067b1774780 923633612 S Bo:6:003:1 -115 31 = 55534243 ce0e0000 12000000 80020603 00000012 00000000 00000000 000000
ffff9067b1774780 923633670 C Bo:6:003:1 0 31 >
ffff90673f46fb40 923633706 S Bi:6:003:2 -115 18 <
ffff90673f46fb40 923634180 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 923634213 S Bi:6:003:2 -115 13 <
ffff9067b1774780 923634343 C Bi:6:003:2 0 13 = 55534253 ce0e0000 00000000 00
ffff9067b1774780 923634409 S Bo:6:003:1 -115 31 = 55534243 cf0e0000 00000000 00030600 00000000 00000000 00000000 000000
ffff9067b1774780 923634506 C Bo:6:003:1 0 31 >
ffff9067b1774780 923634563 S Bi:6:003:2 -115 13 <
ffff9067b1774780 923634842 C Bi:6:003:2 0 13 = 55534253 cf0e0000 00000000 01
ffff9067b1774780 923634850 S Bo:6:003:1 -115 31 = 55534243 d00e0000 12000000 80030603 00000012 00000000 00000000 000000
ffff9067b1774780 923634929 C Bo:6:003:1 0 31 >
ffff9067c0aac900 923634946 S Bi:6:003:2 -115 18 <
ffff9067c0aac900 923635470 C Bi:6:003:2 0 18 = 70000200 0000000a 00000000 3a000000 0000
ffff9067b1774780 923635490 S Bi:6:003:2 -115 13 <
ffff9067b1774780 923635599 C Bi:6:003:2 0 13 = 55534253 d00e0000 00000000 00
ffff9067b5fba240 923741669 S Ci:6:002:0 s a3 00 0000 0001 0004 4 <
ffff9067b5fba240 923741716 C Ci:6:002:0 0 4 = 07050000
ffff9067aedb40c0 923741740 S Co:6:002:0 s 23 01 0002 0001 0000 0
ffff9067aedb40c0 923741765 C Co:6:002:0 0 0
ffff9067aedb40c0 923783452 S Ci:6:002:0 s a3 00 0000 0001 0004 4 <
ffff9067aedb40c0 923783501 C Ci:6:002:0 0 4 = 03050400
ffff9067aedb40c0 923795469 S Co:6:002:0 s 23 01 0012 0001 0000 0
ffff9067aedb40c0 923795517 C Co:6:002:0 0 0
ffff9067aedb40c0 923795554 S Ci:6:005:0 s 80 00 0000 0000 0002 2 <
ffff9067aedb40c0 923795610 C Ci:6:005:0 0 2 = 0300
ffff9067aedb40c0 923795658 S Co:6:005:0 s 00 01 0001 0000 0000 0
ffff9067aedb40c0 923795692 C Co:6:005:0 0 0
ffff9067aedb40c0 923795725 S Ci:6:005:0 s a3 00 0000 0001 0004 4 <
ffff9067aedb40c0 923795767 C Ci:6:005:0 0 4 = 00010000
ffff9067aedb40c0 923795790 S Ci:6:005:0 s a3 00 0000 0002 0004 4 <
ffff9067aedb40c0 923795839 C Ci:6:005:0 0 4 = 00010000
ffff9067aedb40c0 923795877 S Ci:6:005:0 s a3 00 0000 0003 0004 4 <
ffff9067aedb40c0 923795934 C Ci:6:005:0 0 4 = 00010000
ffff9067aedb40c0 923795943 S Ci:6:005:0 s a3 00 0000 0004 0004 4 <
ffff9067aedb40c0 923795969 C Ci:6:005:0 0 4 = 00010000
ffff9067aedb4c00 923795977 S Ii:6:005:1 -115:2048 1 <
ffff9067aedb4c00 923796109 C Ii:6:005:1 -2:2048 0
ffff90673f63a540 923796125 S Co:6:005:0 s 00 03 0001 0000 0000 0
ffff90673f63a540 923796138 C Co:6:005:0 0 0
ffff90673f63a540 923796143 S Co:6:002:0 s 23 03 0002 0001 0000 0
ffff90673f63a540 923796177 C Co:6:002:0 0 0

^ permalink raw reply	[flat|nested] 104+ messages in thread

end of thread, other threads:[~2020-01-07  7:51 UTC | newest]

Thread overview: 104+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <307581a490b610c3025ee80f79a465a89d68ed19.camel@unipv.it>
2019-08-20 17:13 ` Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Alan Stern
2019-08-23 10:39   ` Andrea Vai
2019-08-23 20:42     ` Alan Stern
2019-08-26  6:09       ` Andrea Vai
2019-08-26 16:33         ` Alan Stern
2019-09-18 15:25           ` Andrea Vai
2019-09-18 16:30             ` Alan Stern
2019-09-19  7:33               ` Andrea Vai
2019-09-19 17:54                 ` Alan Stern
2019-09-20  7:25                   ` Andrea Vai
2019-09-20  7:44                     ` Greg KH
2019-09-19  8:26               ` Damien Le Moal
2019-09-19  8:55                 ` Ming Lei
2019-09-19  9:09                   ` Damien Le Moal
2019-09-19  9:21                     ` Ming Lei
2019-09-19 14:01                 ` Alan Stern
2019-09-19 14:14                   ` Damien Le Moal
2019-09-20  7:03                     ` Andrea Vai
2019-09-25 19:30                       ` Alan Stern
2019-09-25 19:36                         ` Jens Axboe
2019-09-27 15:47                           ` Andrea Vai
2019-11-04 16:00                             ` Andrea Vai
2019-11-04 18:20                               ` Alan Stern
2019-11-05 11:48                                 ` Andrea Vai
2019-11-05 18:31                                   ` Alan Stern
2019-11-05 23:29                                     ` Jens Axboe
2019-11-06 16:03                                       ` Alan Stern
2019-11-06 22:13                                         ` Damien Le Moal
2019-11-07  7:04                                           ` Andrea Vai
2019-11-07  7:54                                             ` Damien Le Moal
2019-11-07 18:59                                               ` Andrea Vai
2019-11-08  8:42                                                 ` Damien Le Moal
2019-11-08 14:33                                                   ` Jens Axboe
2019-11-11 10:46                                                     ` Andrea Vai
2019-11-09 10:09                                                   ` Ming Lei
2019-11-09 22:28                                                 ` Ming Lei
2019-11-11 10:50                                                   ` Andrea Vai
2019-11-11 11:05                                                     ` Ming Lei
2019-11-11 11:13                                                       ` Andrea Vai
2019-11-22 19:16                                                   ` Andrea Vai
2019-11-23  7:28                                                     ` Ming Lei
2019-11-23 15:44                                                       ` Andrea Vai
2019-11-25  3:54                                                         ` Ming Lei
2019-11-25 10:11                                                           ` Andrea Vai
2019-11-25 10:29                                                             ` Ming Lei
2019-11-25 14:58                                                               ` Andrea Vai
2019-11-25 15:15                                                                 ` Ming Lei
2019-11-25 18:51                                                                   ` Andrea Vai
2019-11-26  2:32                                                                     ` Ming Lei
2019-11-26  7:46                                                                       ` Andrea Vai
2019-11-26  9:15                                                                         ` Ming Lei
2019-11-26 10:24                                                                           ` Ming Lei
2019-11-26 11:14                                                                           ` Andrea Vai
2019-11-27  2:05                                                                             ` Ming Lei
2019-11-27  9:39                                                                               ` Andrea Vai
2019-11-27 13:08                                                                                 ` Ming Lei
2019-11-27 15:01                                                                                   ` Andrea Vai
2019-11-27  0:21                                                                         ` Finn Thain
2019-11-27  8:14                                                                           ` AW: " Schmid, Carsten
2019-11-27 21:49                                                                             ` Finn Thain
2019-11-28  7:46                                                                             ` Andrea Vai
2019-11-28  8:12                                                                               ` AW: " Schmid, Carsten
2019-11-28 11:40                                                                                 ` Andrea Vai
2019-11-28 17:39                                                                                 ` Alan Stern
2019-11-28  9:17                                                                               ` Ming Lei
2019-11-28 17:34                                                                                 ` Andrea Vai
2019-11-29  0:57                                                                                   ` Ming Lei
2019-11-29  2:35                                                                                     ` Ming Lei
2019-11-29 14:41                                                                                       ` Andrea Vai
2019-12-03  2:23                                                                                         ` Ming Lei
2019-12-10  7:35                                                                                           ` Andrea Vai
2019-12-10  8:05                                                                                             ` Ming Lei
2019-12-11  2:41                                                                                               ` Theodore Y. Ts'o
2019-12-11  4:00                                                                                                 ` Ming Lei
2019-12-11 16:07                                                                                                   ` Theodore Y. Ts'o
2019-12-11 21:33                                                                                                     ` Ming Lei
2019-12-12  7:34                                                                                                       ` Andrea Vai
2019-12-18  8:25                                                                                                       ` Andrea Vai
2019-12-18  9:48                                                                                                         ` Ming Lei
     [not found]                                                                                                           ` <b1b6a0e9d690ecd9432025acd2db4ac09f834040.camel@unipv.it>
2019-12-23 13:08                                                                                                             ` Ming Lei
2019-12-23 14:02                                                                                                               ` Andrea Vai
2019-12-24  1:32                                                                                                                 ` Ming Lei
2019-12-24  8:04                                                                                                                   ` Andrea Vai
2019-12-24  8:47                                                                                                                     ` Ming Lei
2019-12-23 16:26                                                                                                               ` Theodore Y. Ts'o
2019-12-23 16:29                                                                                                                 ` Andrea Vai
2019-12-23 17:22                                                                                                                   ` Theodore Y. Ts'o
2019-12-23 18:45                                                                                                                     ` Andrea Vai
2019-12-23 19:53                                                                                                                       ` Theodore Y. Ts'o
2019-12-24  1:27                                                                                                                         ` Ming Lei
2019-12-24  6:49                                                                                                                           ` Andrea Vai
2019-12-24  8:51                                                                                                                           ` Andrea Vai
2019-12-24  9:35                                                                                                                             ` Ming Lei
2019-12-25  5:17                                                                                                                           ` Theodore Y. Ts'o
2019-12-26  2:27                                                                                                                             ` Ming Lei
2019-12-26  3:30                                                                                                                               ` Theodore Y. Ts'o
2019-12-26  8:37                                                                                                                                 ` Ming Lei
2020-01-07  7:51                                                                                                                                   ` Andrea Vai
     [not found]                                                                                                                                 ` <20200101074310.10904-1-hdanton@sina.com>
2020-01-01 13:53                                                                                                                                   ` slow IO on USB media Ming Lei
2019-11-29 11:44                                                                                     ` AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Bernd Schubert
2019-12-02  7:01                                                                                       ` Andrea Vai
2019-11-28 17:10                                                                           ` Andrea Vai
     [not found] <Pine.LNX.4.44L0.1907081101260.1424-100000@iolanthe.rowland.org>
2019-08-09 12:07 ` Andrea Vai
2019-08-13 19:52   ` Alan Stern

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).