All of lore.kernel.org
 help / color / mirror / Atom feed
From: Peter Rosin <peda@axentia.se>
To: "Tudor.Ambarus@microchip.com" <Tudor.Ambarus@microchip.com>,
	"regressions@leemhuis.info" <regressions@leemhuis.info>,
	"Nicolas.Ferre@microchip.com" <Nicolas.Ferre@microchip.com>,
	"alexandre.belloni@bootlin.com" <alexandre.belloni@bootlin.com>
Cc: Daniels Umanovskis <du@axentia.se>,
	"Patrice.Vilchez@microchip.com" <Patrice.Vilchez@microchip.com>,
	"Cristian.Birsan@microchip.com" <Cristian.Birsan@microchip.com>,
	"Ludovic.Desroches@microchip.com"
	<Ludovic.Desroches@microchip.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-arm-kernel@lists.infradead.org" 
	<linux-arm-kernel@lists.infradead.org>,
	"gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
	"saravanak@google.com" <saravanak@google.com>
Subject: RE: Regression: memory corruption on Atmel SAMA5D31
Date: Tue, 21 Jun 2022 10:46:25 +0000	[thread overview]
Message-ID: <AM0PR02MB4436C535FDD72EFE422D8B10BCB39@AM0PR02MB4436.eurprd02.prod.outlook.com> (raw)
In-Reply-To: <0d8b2d9c-af85-7148-ff13-aa968a7f51ad@microchip.com>

2022-06-20 at 16:22, Tudor.Ambarus@microchip.com wrote:
> 
>>
>> git@github.com:ambarus/linux-0day.git, branch dma-regression-hdmac-v5.18-rc7-4th-attempt
>>
> 
> Hi, Peter,
> 
> I've just forced pushed on this branch, I had a typo somewhere and with that fixed I could
> no longer reproduce the bug. Tested for ~20 minutes. Would you please test last 3 patches
> and tell me if you can still reproduce the bug?

Hi!

I rebased your patches onto my current branch which is v5.18.2 plus a few unrelated
changes (at least they are unrelated after removing the previous workaround to disable
nand-dma entirely).

The unrelated patches are two backports so that drivers recognize new compatibles [1][2],
which should be completely harmless, plus a couple of proposed fixes that happens to fix
eeprom issues with the at91 I2C driver from Codrin Ciubotariu [3].

On that kernel, I can still reproduce. It seems a bit harder to reproduce the problem now
though. If the system is otherwise idle, the sha256sum test did not reproduce in a run of
150+ attempts, but if I let the "real" application run while I do the test, I get a failure rate
of about 10%, see below. The real application burns some CPU (but not all of it) and
communicates with HW using I2C, native UARTs and two of the four USB-serial ports
(FTDI, with the latency set to 1ms as mentioned earlier), so I guess there is more DMA
pressure or something? There is a 100mbps network connection, but it was left "idle"
during this test.

Cheers,
Peter

$ dd if=/dev/urandom of=testfile bs=1024 count=40000
40000+0 records in
40000+0 records out
40960000 bytes (41 MB, 39 MiB) copied, 80.0485 s, 512 kB/s
$ while :; do cat testfile | sha256sum; done
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
a4850c1bb0226f14659035cdf1461c7df03d50bff8af560e3bd204942556b73f  -
43c1941e15bd7e048e9d5f1d41ce67517cb6e59dae1d3af256d1507168100fcb  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
4a35af384455853a24b943ef94353663e8c22a9aa29d2e275194fd544d0b194a  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
f6710849b36e6954c26ff62cd974ecb082b93fa6e53ecf0aea7e0c93acc0a445  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
2c2f4ac91f435439d2d640c34ee89b4d1ebf3adb8438efbf064a4139247241c5  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
^C

[1] https://lore.kernel.org/linux-kernel/42db911c-5eba-0511-3e8c-8011a2a5b44a@axentia.se/
[2] https://lore.kernel.org/linux-kernel/ea4cd16b-4a04-8857-d08a-53be58b00d28@axentia.se/
[3] https://lore.kernel.org/linux-kernel/20220614101347.16910-1-codrin.ciubotariu@microchip.com/

WARNING: multiple messages have this Message-ID (diff)
From: Peter Rosin <peda@axentia.se>
To: "Tudor.Ambarus@microchip.com" <Tudor.Ambarus@microchip.com>,
	"regressions@leemhuis.info" <regressions@leemhuis.info>,
	"Nicolas.Ferre@microchip.com" <Nicolas.Ferre@microchip.com>,
	"alexandre.belloni@bootlin.com" <alexandre.belloni@bootlin.com>
Cc: Daniels Umanovskis <du@axentia.se>,
	"Patrice.Vilchez@microchip.com" <Patrice.Vilchez@microchip.com>,
	"Cristian.Birsan@microchip.com" <Cristian.Birsan@microchip.com>,
	"Ludovic.Desroches@microchip.com"
	<Ludovic.Desroches@microchip.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>,
	"gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
	"saravanak@google.com" <saravanak@google.com>
Subject: RE: Regression: memory corruption on Atmel SAMA5D31
Date: Tue, 21 Jun 2022 10:46:25 +0000	[thread overview]
Message-ID: <AM0PR02MB4436C535FDD72EFE422D8B10BCB39@AM0PR02MB4436.eurprd02.prod.outlook.com> (raw)
In-Reply-To: <0d8b2d9c-af85-7148-ff13-aa968a7f51ad@microchip.com>

2022-06-20 at 16:22, Tudor.Ambarus@microchip.com wrote:
> 
>>
>> git@github.com:ambarus/linux-0day.git, branch dma-regression-hdmac-v5.18-rc7-4th-attempt
>>
> 
> Hi, Peter,
> 
> I've just forced pushed on this branch, I had a typo somewhere and with that fixed I could
> no longer reproduce the bug. Tested for ~20 minutes. Would you please test last 3 patches
> and tell me if you can still reproduce the bug?

Hi!

I rebased your patches onto my current branch which is v5.18.2 plus a few unrelated
changes (at least they are unrelated after removing the previous workaround to disable
nand-dma entirely).

The unrelated patches are two backports so that drivers recognize new compatibles [1][2],
which should be completely harmless, plus a couple of proposed fixes that happens to fix
eeprom issues with the at91 I2C driver from Codrin Ciubotariu [3].

On that kernel, I can still reproduce. It seems a bit harder to reproduce the problem now
though. If the system is otherwise idle, the sha256sum test did not reproduce in a run of
150+ attempts, but if I let the "real" application run while I do the test, I get a failure rate
of about 10%, see below. The real application burns some CPU (but not all of it) and
communicates with HW using I2C, native UARTs and two of the four USB-serial ports
(FTDI, with the latency set to 1ms as mentioned earlier), so I guess there is more DMA
pressure or something? There is a 100mbps network connection, but it was left "idle"
during this test.

Cheers,
Peter

$ dd if=/dev/urandom of=testfile bs=1024 count=40000
40000+0 records in
40000+0 records out
40960000 bytes (41 MB, 39 MiB) copied, 80.0485 s, 512 kB/s
$ while :; do cat testfile | sha256sum; done
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
a4850c1bb0226f14659035cdf1461c7df03d50bff8af560e3bd204942556b73f  -
43c1941e15bd7e048e9d5f1d41ce67517cb6e59dae1d3af256d1507168100fcb  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
4a35af384455853a24b943ef94353663e8c22a9aa29d2e275194fd544d0b194a  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
f6710849b36e6954c26ff62cd974ecb082b93fa6e53ecf0aea7e0c93acc0a445  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
2c2f4ac91f435439d2d640c34ee89b4d1ebf3adb8438efbf064a4139247241c5  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
b93e0c56cfba75cb9e2b35ff6769abdb6c1a5d17cadc28cec1979188e044cf3d  -
^C

[1] https://lore.kernel.org/linux-kernel/42db911c-5eba-0511-3e8c-8011a2a5b44a@axentia.se/
[2] https://lore.kernel.org/linux-kernel/ea4cd16b-4a04-8857-d08a-53be58b00d28@axentia.se/
[3] https://lore.kernel.org/linux-kernel/20220614101347.16910-1-codrin.ciubotariu@microchip.com/
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

  parent reply	other threads:[~2022-06-21 10:47 UTC|newest]

Thread overview: 77+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-03-03  0:29 Regression: memory corruption on Atmel SAMA5D31 Peter Rosin
2022-03-03  3:02 ` Saravana Kannan
2022-03-03  3:02   ` Saravana Kannan
2022-03-03  9:17   ` Peter Rosin
2022-03-03  9:17     ` Peter Rosin
2022-03-04  3:55     ` Saravana Kannan
2022-03-04  3:55       ` Saravana Kannan
2022-03-04  6:57       ` Peter Rosin
2022-03-04  6:57         ` Peter Rosin
2022-03-04 10:57         ` Peter Rosin
2022-03-04 10:57           ` Peter Rosin
2022-03-04 11:12           ` Tudor.Ambarus
2022-03-04 11:12             ` Tudor.Ambarus
2022-03-04 12:38             ` Peter Rosin
2022-03-04 12:38               ` Peter Rosin
2022-03-04 16:48               ` Tudor.Ambarus
2022-03-04 16:48                 ` Tudor.Ambarus
2022-03-07  9:45                 ` Tudor.Ambarus
2022-03-07  9:45                   ` Tudor.Ambarus
2022-03-07 11:32                   ` Peter Rosin
2022-03-07 11:32                     ` Peter Rosin
2022-03-07 20:32                     ` Peter Rosin
2022-03-07 20:32                       ` Peter Rosin
2022-03-08  7:55                       ` Nicolas Ferre
2022-03-08  7:55                         ` Nicolas Ferre
2022-03-09  8:30                         ` Peter Rosin
2022-03-09  8:30                           ` Peter Rosin
     [not found]                           ` <6d9561a4-39e4-3dbe-5fe2-c6f88ee2a4c6@axentia.se>
     [not found]                             ` <ed24a281-1790-8e24-5f5a-25b66527044b@microchip.com>
     [not found]                               ` <d563c7ba-6431-2639-9f2a-2e2c6788e625@axentia.se>
     [not found]                                 ` <e5a715c5-ad9f-6fd4-071e-084ab950603e@microchip.com>
2022-03-10  9:58                                   ` Peter Rosin
2022-03-10  9:58                                     ` Peter Rosin
2022-03-10 10:40                                     ` Peter Rosin
2022-03-10 10:40                                       ` Peter Rosin
2022-04-09 13:02                                       ` Thorsten Leemhuis
2022-04-09 13:02                                         ` Thorsten Leemhuis
2022-04-11  6:21                                         ` Tudor.Ambarus
2022-04-11  6:21                                           ` Tudor.Ambarus
2022-05-17 14:50                                           ` Peter Rosin
2022-05-17 14:50                                             ` Peter Rosin
2022-05-18  6:21                                             ` Tudor.Ambarus
2022-05-18  6:21                                               ` Tudor.Ambarus
2022-05-18  7:51                                               ` Peter Rosin
2022-05-18  7:51                                                 ` Peter Rosin
2022-06-20  7:04                                                 ` Thorsten Leemhuis
2022-06-20  7:04                                                   ` Thorsten Leemhuis
2022-06-20  8:43                                                   ` Tudor.Ambarus
2022-06-20  8:43                                                     ` Tudor.Ambarus
2022-06-20 14:22                                                     ` Tudor.Ambarus
2022-06-20 14:22                                                       ` Tudor.Ambarus
2022-06-21  7:00                                                       ` Peter Rosin
2022-06-21  7:00                                                         ` Peter Rosin
2022-06-21 10:46                                                       ` Peter Rosin [this message]
2022-06-21 10:46                                                         ` Peter Rosin
2022-06-27 12:26                                                         ` Tudor.Ambarus
2022-06-27 12:26                                                           ` Tudor.Ambarus
2022-06-27 16:53                                                           ` Tudor.Ambarus
2022-06-27 16:53                                                             ` Tudor.Ambarus
2022-06-30  5:20                                                             ` Peter Rosin
2022-06-30  5:20                                                               ` Peter Rosin
2022-06-30  9:23                                                               ` Tudor.Ambarus
2022-06-30  9:23                                                                 ` Tudor.Ambarus
2022-06-30 10:20                                                                 ` Tudor.Ambarus
2022-06-30 10:20                                                                   ` Tudor.Ambarus
2022-07-13 16:01                                                             ` Tudor.Ambarus
2022-07-13 16:01                                                               ` Tudor.Ambarus
2022-07-28  7:45                                                               ` Tudor.Ambarus
2022-07-28  7:45                                                                 ` Tudor.Ambarus
2022-07-28  8:39                                                                 ` Tudor.Ambarus
2022-07-28  8:39                                                                   ` Tudor.Ambarus
2022-07-29 20:09                                                                   ` Peter Rosin
2022-07-29 20:09                                                                     ` Peter Rosin
2022-07-30 11:37                                                                     ` Peter Rosin
2022-07-30 11:37                                                                       ` Peter Rosin
2022-07-31  3:44                                                                       ` Tudor.Ambarus
2022-07-31  3:44                                                                         ` Tudor.Ambarus
2022-03-04 20:06           ` Saravana Kannan
2022-03-04 20:06             ` Saravana Kannan
2022-03-04  8:00 ` Thorsten Leemhuis
2022-03-04  8:00   ` Thorsten Leemhuis

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=AM0PR02MB4436C535FDD72EFE422D8B10BCB39@AM0PR02MB4436.eurprd02.prod.outlook.com \
    --to=peda@axentia.se \
    --cc=Cristian.Birsan@microchip.com \
    --cc=Ludovic.Desroches@microchip.com \
    --cc=Nicolas.Ferre@microchip.com \
    --cc=Patrice.Vilchez@microchip.com \
    --cc=Tudor.Ambarus@microchip.com \
    --cc=alexandre.belloni@bootlin.com \
    --cc=du@axentia.se \
    --cc=gregkh@linuxfoundation.org \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=regressions@leemhuis.info \
    --cc=saravanak@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.