All of lore.kernel.org
 help / color / mirror / Atom feed
From: Coiby Xu <coiby.xu@gmail.com>
To: Hans de Goede <hdegoede@redhat.com>,
	Linus Walleij <linus.walleij@linaro.org>
Cc: "open list:GPIO SUBSYSTEM" <linux-gpio@vger.kernel.org>,
	wang jun <availa@outlook.com>,
	Nehal Shah <Nehal-bakulchandra.Shah@amd.com>,
	Shyam Sundar S K <Shyam-sundar.S-k@amd.com>,
	linux-kernel-mentees@lists.linuxfoundation.org
Subject: Re: Any other ways to debug GPIO interrupt controller (pinctrl-amd) for broken touchpads of a new laptop model?
Date: Tue, 6 Oct 2020 12:49:41 +0800	[thread overview]
Message-ID: <20201006044941.fdjsp346kc5thyzy@Rk> (raw)
In-Reply-To: <20201004051644.f3fg2oavbobrwhf6@Rk>

Hi Hans and Linus,

I've found the direct evidence proving the GPIO interrupt controller is
malfunctioning.

I've found a way to let the GPIO chip trigger an interrupt by accident
when playing with the GPIO sysfs interface,

  - export pin130 which is used by the touchad
  - set the direction to be "out"
  - `echo 0 > value` will trigger the GPIO controller's parent irq and
    "echo 1 > value" will make it stop firing

(I'm not sure if this is yet another bug of the GPIO chip. Anyway I can
manually trigger an interrupt now.)

I wrote a C program is to let GPIO controller quickly generate some
interrupts then disable the firing of interrupts by toggling pin#130's
value with an specified time interval, i.e., set the value to 0 first
and then after some time, re-set the value to 1. There is no interrupt
firing unless time internal > 120ms (~7Hz). This explains why we can
only see 7 interrupts for the GPIO controller's parent irq.

My hypothesis is the GPIO doesn't have proper power setting so it stays
in an idle state or its clock frequency is too low by default thus not
quick enough to read interrupt input. Then pinctrl-amd must miss some
code to configure the chip and I need a hardware reference manual of this
GPIO chip (HID: AMDI0030) or reverse-engineer the driver for Windows
since I couldn't find a copy of reference manual online? What would you
suggest?

Thank you!

On Sun, Oct 04, 2020 at 01:16:44PM +0800, Coiby Xu wrote:
>On Sun, Oct 04, 2020 at 07:03:40AM +0800, Coiby Xu wrote:
>>On Sat, Oct 03, 2020 at 03:22:46PM +0200, Hans de Goede wrote:
>>>Hi,
>>>
>>>On 10/3/20 12:45 AM, Coiby Xu wrote:
>>>>On Fri, Oct 02, 2020 at 09:44:54PM +0200, Hans de Goede wrote:
>>>>>Hi,
>>>>>
>>>>>On 10/2/20 4:51 PM, Coiby Xu wrote:
>>>>>>On Fri, Oct 02, 2020 at 03:36:29PM +0200, Hans de Goede wrote:
>>>>>
>>>>><snip>
>>>>>
>>>>>>>>>So are you seeing these 7 interrupts / second for the touchpad irq or for
>>>>>>>>>the GPIO controllers parent irq ?
>>>>>>>>>
>>>>>>>>>Also to these 7 interrupts/sec stop happening when you do not touch the
>>>>>>>>>touchpad ?
>>>>>>>>>
>>>>>>>>I see these 7 interrupts / second for the GPIO controller's parent irq.
>>>>>>>>And they stop happening when I don't touch the touchpad.
>>>>>>>
>>>>>>>Only from the parent irq, or also on the touchpad irq itself ?
>>>>>>>
>>>>>>>If this only happens on the parent irq, then I would start looking at the
>>>>>>>amd-pinctrl code which determines which of its "child" irqs to fire.
>>>>>>
>>>>>>This only happens on the parent irq. The input's pin#130 of the GIPO
>>>>>>chip is low most of the time and pin#130.
>>>>>
>>>>>Right, but it is a low-level triggered IRQ, so when it is low it should
>>>>>be executing the i2c-hid interrupt-handler. If it is not executing that
>>>>>then it is time to look at amd-pinctrl's irq-handler and figure out why
>>>>>that is not triggering the child irq handler for the touchpad.
>>>>>
>>>>I'm not sure if I have some incorrect understandings about GPIO
>>>>interrupt controller because I don't quite follow your reasoning.
>>>>What I actually suspect is there's something wrong with amd-pinctrl
>>>>which makes the GPIO chip fail to assert its common interrupt output
>>>>line connected to one IO-APIC's pin#7 thus IRQ#7 fails to fire. What
>>>>I learn about this low-level triggered IRQ is that the i2c-hid
>>>>interrupt-handler will be woken up by amd-pinctrl's irq-handler which
>>>>is executed when the parent IRQ#7 fires. The code path is as follows,
>>>>
>>>>    <IRQ>
>>>>    dump_stack+0x64/0x88
>>>>    __irq_wake_thread.cold+0x9/0x12
>>>>    __handle_irq_event_percpu+0x80/0x1c0
>>>>    handle_irq_event+0x58/0xb0
>>>>    handle_level_irq+0xb7/0x1a0
>>>>    generic_handle_irq+0x4a/0x60
>>>>    amd_gpio_irq_handler+0x15f/0x1b0 [pinctrl_amd]
>>>>    __handle_irq_event_percpu+0x45/0x1c0
>>>>    handle_irq_event+0x58/0xb0
>>>>    handle_fasteoi_irq+0xa2/0x210
>>>>    do_IRQ+0x70/0x120
>>>>    common_interrupt+0xf/0xf
>>>>    </IRQ>
>>>>
>>>>But the problem is somehow IRQ#7 doesn't even fire when the input's
>>>>pin#130 of the GIPO is low. Without IRQ#7 firing, amd-pinctrl's
>>>>irq-handler wouldn't be executed in the first place, let alonet
>>>>triggering the child irq handler. Btw, amd-pinctrl's irq-handler
>>>>simply iterate over all pins. If there is mapped irq found for this
>>>>hwirq (yes, it won't even check if this pin triggers the interrupt),
>>>>then it will call generic_handle_irq. So there's nothing wrong about
>>>>this part of code.
>>>
>>>Ok, so the i2c-hid irq does fire, but only 7 times a second just
>>>like the GPIO controller's parent irq.
>>>
>>I'm not sure if it's correct to say if hi2c-hid irq fires or not and how
>>frequently it fires since the i2c-hid irq is mapped to pin#130 of the
>>GPIO interrupt controller and the touchpad has another interrupt line
>>connected to pin#130 which fires to indicate new data. All we know is
>>pin#130 of the GPIO chip has low input most of the time when the finger
>>is on the touchpad so we can infer the touchpad has been trying to
>>notify the kernel of new data but somehow GPIO's parent irq only fires 7
>>times / second.
>>
>>>The only thing I can think of then is to add printk-s to check how
>>>long the i2c-hid interrupt handler takes to complete. It could be
>>>there is a subtle bug somewhere causing the i2c transfers to take
>>>longer when run from a (threaded) irq handler. That would be weird
>>>though, so I don't expect this to result in any useful findings.
>>>
>>
>>I also doubted if it takes too much time for the i2c-hid handler to
>>finish reading i2c transfer, processing data and delivering to the input
>>system. After measuring the time internal between the starting of the
>>GPIO irq's parent handler and when pin#130 is unmasked, we can exclude
>>this possibility.
>>
>>I have been wondering if we let make pin#130 have low input thus to
>>trigger a interrupt firing or assert the GPIO's common interrupt output
>>line manually thus we can measure how long does it take for the kernel
>>to receive the signal. But once GPIO's pin is programmed to be a
>>interrupt line we can't write anything to it and it seems other
>>interrupts can only be generated by the hardware. So this idea is not
>>plausible
>>
>
>Btw, there are other users who have the same laptop model but with a
>different touchpad (ELAN). Their touchpads would show in
>/proc/bus/input/devices but are completely dead. hid-recorder which
>will read HID reports from /dev/hidraw gets nothing if they put there
>fingers on the touchpad but the polling mode could also save their
>touchpads. It seems GPIO controller's parent irq for the ELAN touchpad
>doesn't even fire once. And unlike GPIO, IO-APIC has also be used by
>other devices like the keyboard. So maybe it's safe to assert the root
>cause is from the GPIO controller.
>
>>>Other then that I'm all out of ideas I'm afraid.
>>>
>>Thank you for taking time to investigate this issue anyway! Have a nice
>>weekend:)
>>>Regards,
>>>
>>>Hans
>>>
>>
>>--
>>Best regards,
>>Coiby
>
>--
>Best regards,
>Coiby

--
Best regards,
Coiby

WARNING: multiple messages have this Message-ID (diff)
From: Coiby Xu <coiby.xu@gmail.com>
To: Hans de Goede <hdegoede@redhat.com>,
	Linus Walleij <linus.walleij@linaro.org>
Cc: wang jun <availa@outlook.com>,
	"open list:GPIO SUBSYSTEM" <linux-gpio@vger.kernel.org>,
	Shyam Sundar S K <Shyam-sundar.S-k@amd.com>,
	Nehal Shah <Nehal-bakulchandra.Shah@amd.com>,
	linux-kernel-mentees@lists.linuxfoundation.org
Subject: Re: [Linux-kernel-mentees] Any other ways to debug GPIO interrupt controller (pinctrl-amd) for broken touchpads of a new laptop model?
Date: Tue, 6 Oct 2020 12:49:41 +0800	[thread overview]
Message-ID: <20201006044941.fdjsp346kc5thyzy@Rk> (raw)
In-Reply-To: <20201004051644.f3fg2oavbobrwhf6@Rk>

Hi Hans and Linus,

I've found the direct evidence proving the GPIO interrupt controller is
malfunctioning.

I've found a way to let the GPIO chip trigger an interrupt by accident
when playing with the GPIO sysfs interface,

  - export pin130 which is used by the touchad
  - set the direction to be "out"
  - `echo 0 > value` will trigger the GPIO controller's parent irq and
    "echo 1 > value" will make it stop firing

(I'm not sure if this is yet another bug of the GPIO chip. Anyway I can
manually trigger an interrupt now.)

I wrote a C program is to let GPIO controller quickly generate some
interrupts then disable the firing of interrupts by toggling pin#130's
value with an specified time interval, i.e., set the value to 0 first
and then after some time, re-set the value to 1. There is no interrupt
firing unless time internal > 120ms (~7Hz). This explains why we can
only see 7 interrupts for the GPIO controller's parent irq.

My hypothesis is the GPIO doesn't have proper power setting so it stays
in an idle state or its clock frequency is too low by default thus not
quick enough to read interrupt input. Then pinctrl-amd must miss some
code to configure the chip and I need a hardware reference manual of this
GPIO chip (HID: AMDI0030) or reverse-engineer the driver for Windows
since I couldn't find a copy of reference manual online? What would you
suggest?

Thank you!

On Sun, Oct 04, 2020 at 01:16:44PM +0800, Coiby Xu wrote:
>On Sun, Oct 04, 2020 at 07:03:40AM +0800, Coiby Xu wrote:
>>On Sat, Oct 03, 2020 at 03:22:46PM +0200, Hans de Goede wrote:
>>>Hi,
>>>
>>>On 10/3/20 12:45 AM, Coiby Xu wrote:
>>>>On Fri, Oct 02, 2020 at 09:44:54PM +0200, Hans de Goede wrote:
>>>>>Hi,
>>>>>
>>>>>On 10/2/20 4:51 PM, Coiby Xu wrote:
>>>>>>On Fri, Oct 02, 2020 at 03:36:29PM +0200, Hans de Goede wrote:
>>>>>
>>>>><snip>
>>>>>
>>>>>>>>>So are you seeing these 7 interrupts / second for the touchpad irq or for
>>>>>>>>>the GPIO controllers parent irq ?
>>>>>>>>>
>>>>>>>>>Also to these 7 interrupts/sec stop happening when you do not touch the
>>>>>>>>>touchpad ?
>>>>>>>>>
>>>>>>>>I see these 7 interrupts / second for the GPIO controller's parent irq.
>>>>>>>>And they stop happening when I don't touch the touchpad.
>>>>>>>
>>>>>>>Only from the parent irq, or also on the touchpad irq itself ?
>>>>>>>
>>>>>>>If this only happens on the parent irq, then I would start looking at the
>>>>>>>amd-pinctrl code which determines which of its "child" irqs to fire.
>>>>>>
>>>>>>This only happens on the parent irq. The input's pin#130 of the GIPO
>>>>>>chip is low most of the time and pin#130.
>>>>>
>>>>>Right, but it is a low-level triggered IRQ, so when it is low it should
>>>>>be executing the i2c-hid interrupt-handler. If it is not executing that
>>>>>then it is time to look at amd-pinctrl's irq-handler and figure out why
>>>>>that is not triggering the child irq handler for the touchpad.
>>>>>
>>>>I'm not sure if I have some incorrect understandings about GPIO
>>>>interrupt controller because I don't quite follow your reasoning.
>>>>What I actually suspect is there's something wrong with amd-pinctrl
>>>>which makes the GPIO chip fail to assert its common interrupt output
>>>>line connected to one IO-APIC's pin#7 thus IRQ#7 fails to fire. What
>>>>I learn about this low-level triggered IRQ is that the i2c-hid
>>>>interrupt-handler will be woken up by amd-pinctrl's irq-handler which
>>>>is executed when the parent IRQ#7 fires. The code path is as follows,
>>>>
>>>>    <IRQ>
>>>>    dump_stack+0x64/0x88
>>>>    __irq_wake_thread.cold+0x9/0x12
>>>>    __handle_irq_event_percpu+0x80/0x1c0
>>>>    handle_irq_event+0x58/0xb0
>>>>    handle_level_irq+0xb7/0x1a0
>>>>    generic_handle_irq+0x4a/0x60
>>>>    amd_gpio_irq_handler+0x15f/0x1b0 [pinctrl_amd]
>>>>    __handle_irq_event_percpu+0x45/0x1c0
>>>>    handle_irq_event+0x58/0xb0
>>>>    handle_fasteoi_irq+0xa2/0x210
>>>>    do_IRQ+0x70/0x120
>>>>    common_interrupt+0xf/0xf
>>>>    </IRQ>
>>>>
>>>>But the problem is somehow IRQ#7 doesn't even fire when the input's
>>>>pin#130 of the GIPO is low. Without IRQ#7 firing, amd-pinctrl's
>>>>irq-handler wouldn't be executed in the first place, let alonet
>>>>triggering the child irq handler. Btw, amd-pinctrl's irq-handler
>>>>simply iterate over all pins. If there is mapped irq found for this
>>>>hwirq (yes, it won't even check if this pin triggers the interrupt),
>>>>then it will call generic_handle_irq. So there's nothing wrong about
>>>>this part of code.
>>>
>>>Ok, so the i2c-hid irq does fire, but only 7 times a second just
>>>like the GPIO controller's parent irq.
>>>
>>I'm not sure if it's correct to say if hi2c-hid irq fires or not and how
>>frequently it fires since the i2c-hid irq is mapped to pin#130 of the
>>GPIO interrupt controller and the touchpad has another interrupt line
>>connected to pin#130 which fires to indicate new data. All we know is
>>pin#130 of the GPIO chip has low input most of the time when the finger
>>is on the touchpad so we can infer the touchpad has been trying to
>>notify the kernel of new data but somehow GPIO's parent irq only fires 7
>>times / second.
>>
>>>The only thing I can think of then is to add printk-s to check how
>>>long the i2c-hid interrupt handler takes to complete. It could be
>>>there is a subtle bug somewhere causing the i2c transfers to take
>>>longer when run from a (threaded) irq handler. That would be weird
>>>though, so I don't expect this to result in any useful findings.
>>>
>>
>>I also doubted if it takes too much time for the i2c-hid handler to
>>finish reading i2c transfer, processing data and delivering to the input
>>system. After measuring the time internal between the starting of the
>>GPIO irq's parent handler and when pin#130 is unmasked, we can exclude
>>this possibility.
>>
>>I have been wondering if we let make pin#130 have low input thus to
>>trigger a interrupt firing or assert the GPIO's common interrupt output
>>line manually thus we can measure how long does it take for the kernel
>>to receive the signal. But once GPIO's pin is programmed to be a
>>interrupt line we can't write anything to it and it seems other
>>interrupts can only be generated by the hardware. So this idea is not
>>plausible
>>
>
>Btw, there are other users who have the same laptop model but with a
>different touchpad (ELAN). Their touchpads would show in
>/proc/bus/input/devices but are completely dead. hid-recorder which
>will read HID reports from /dev/hidraw gets nothing if they put there
>fingers on the touchpad but the polling mode could also save their
>touchpads. It seems GPIO controller's parent irq for the ELAN touchpad
>doesn't even fire once. And unlike GPIO, IO-APIC has also be used by
>other devices like the keyboard. So maybe it's safe to assert the root
>cause is from the GPIO controller.
>
>>>Other then that I'm all out of ideas I'm afraid.
>>>
>>Thank you for taking time to investigate this issue anyway! Have a nice
>>weekend:)
>>>Regards,
>>>
>>>Hans
>>>
>>
>>--
>>Best regards,
>>Coiby
>
>--
>Best regards,
>Coiby

--
Best regards,
Coiby
_______________________________________________
Linux-kernel-mentees mailing list
Linux-kernel-mentees@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees

  reply	other threads:[~2020-10-06  4:49 UTC|newest]

Thread overview: 84+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-01 13:22 Any other ways to debug GPIO interrupt controller (pinctrl-amd) for broken touchpads of a new laptop model? Coiby Xu
2020-10-01 13:22 ` [Linux-kernel-mentees] " Coiby Xu
2020-10-01 20:57 ` Linus Walleij
2020-10-01 20:57   ` [Linux-kernel-mentees] " Linus Walleij
2020-10-02  9:40   ` Hans de Goede
2020-10-02  9:40     ` [Linux-kernel-mentees] " Hans de Goede
2020-10-02 12:42     ` Coiby Xu
2020-10-02 12:42       ` [Linux-kernel-mentees] " Coiby Xu
2020-10-02 13:36       ` Hans de Goede
2020-10-02 13:36         ` [Linux-kernel-mentees] " Hans de Goede
2020-10-02 14:51         ` Coiby Xu
2020-10-02 14:51           ` [Linux-kernel-mentees] " Coiby Xu
2020-10-02 19:44           ` Hans de Goede
2020-10-02 19:44             ` [Linux-kernel-mentees] " Hans de Goede
2020-10-02 22:45             ` Coiby Xu
2020-10-02 22:45               ` [Linux-kernel-mentees] " Coiby Xu
2020-10-03 13:22               ` Hans de Goede
2020-10-03 13:22                 ` [Linux-kernel-mentees] " Hans de Goede
2020-10-03 23:03                 ` Coiby Xu
2020-10-03 23:03                   ` [Linux-kernel-mentees] " Coiby Xu
2020-10-04  5:16                   ` Coiby Xu
2020-10-04  5:16                     ` [Linux-kernel-mentees] " Coiby Xu
2020-10-06  4:49                     ` Coiby Xu [this message]
2020-10-06  4:49                       ` Coiby Xu
2020-10-06  6:28                       ` Hans de Goede
2020-10-06  6:28                         ` [Linux-kernel-mentees] " Hans de Goede
2020-10-06  8:31                         ` Coiby Xu
2020-10-06  8:31                           ` [Linux-kernel-mentees] " Coiby Xu
2020-10-06  8:55                           ` Hans de Goede
2020-10-06  8:55                             ` [Linux-kernel-mentees] " Hans de Goede
2020-10-06  9:28                             ` Hans de Goede
2020-10-06  9:28                               ` [Linux-kernel-mentees] " Hans de Goede
2020-10-06  9:29                               ` Hans de Goede
2020-10-06  9:29                                 ` [Linux-kernel-mentees] " Hans de Goede
2020-10-08 16:32                                 ` Coiby Xu
2020-10-08 16:32                                   ` [Linux-kernel-mentees] " Coiby Xu
2020-10-14  4:24                                 ` Coiby Xu
2020-10-14  4:24                                   ` [Linux-kernel-mentees] " Coiby Xu
2020-10-14 11:34                                   ` Coiby Xu
2020-10-14 11:34                                     ` [Linux-kernel-mentees] " Coiby Xu
2020-10-14 11:46                                   ` Hans de Goede
2020-10-14 11:46                                     ` [Linux-kernel-mentees] " Hans de Goede
2020-10-15  3:27                                     ` Coiby Xu
2020-10-15  3:27                                       ` [Linux-kernel-mentees] " Coiby Xu
2020-10-15  4:06                                     ` Coiby Xu
2020-10-15  4:06                                       ` [Linux-kernel-mentees] " Coiby Xu
2020-10-26 22:54                                     ` Coiby Xu
2020-10-26 22:54                                       ` [Linux-kernel-mentees] " Coiby Xu
2020-10-27  9:52                                       ` Andy Shevchenko
2020-10-27  9:52                                         ` [Linux-kernel-mentees] " Andy Shevchenko
2020-10-30  4:58                                         ` Coiby Xu
2020-10-30  4:58                                           ` [Linux-kernel-mentees] " Coiby Xu
2020-10-27 10:09                                       ` Hans de Goede
2020-10-27 10:09                                         ` [Linux-kernel-mentees] " Hans de Goede
2020-10-27 15:13                                         ` Andy Shevchenko
2020-10-27 15:13                                           ` [Linux-kernel-mentees] " Andy Shevchenko
2020-10-27 16:00                                           ` Hans de Goede
2020-10-27 16:00                                             ` [Linux-kernel-mentees] " Hans de Goede
2020-10-27 16:09                                             ` Andy Shevchenko
2020-10-27 16:09                                               ` [Linux-kernel-mentees] " Andy Shevchenko
2020-10-29  8:04                                               ` Mika Westerberg
2020-10-29  8:04                                                 ` [Linux-kernel-mentees] " Mika Westerberg
2020-10-30  4:54                                               ` Coiby Xu
2020-10-30  4:54                                                 ` [Linux-kernel-mentees] " Coiby Xu
2020-11-02 19:06                                                 ` Andy Shevchenko
2020-11-02 19:06                                                   ` [Linux-kernel-mentees] " Andy Shevchenko
2020-11-02 22:56                                                   ` Coiby Xu
2020-11-02 22:56                                                     ` [Linux-kernel-mentees] " Coiby Xu
2020-11-03  0:05                                         ` Coiby Xu
2020-11-03  0:05                                           ` [Linux-kernel-mentees] " Coiby Xu
2020-11-03 10:12                                           ` Hans de Goede
2020-11-03 10:12                                             ` [Linux-kernel-mentees] " Hans de Goede
2020-11-03 10:49                                             ` Andy Shevchenko
2020-11-03 10:49                                               ` [Linux-kernel-mentees] " Andy Shevchenko
2020-11-03 11:00                                               ` Hans de Goede
2020-11-03 11:00                                                 ` [Linux-kernel-mentees] " Hans de Goede
2020-10-08 16:26                               ` Coiby Xu
2020-10-08 16:26                                 ` [Linux-kernel-mentees] " Coiby Xu
2020-10-06  9:16                           ` Linus Walleij
2020-10-06  9:16                             ` [Linux-kernel-mentees] " Linus Walleij
2020-10-08 16:40                             ` Coiby Xu
2020-10-08 16:40                               ` [Linux-kernel-mentees] " Coiby Xu
2020-10-02 10:59   ` Coiby Xu
2020-10-02 10:59     ` [Linux-kernel-mentees] " Coiby Xu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201006044941.fdjsp346kc5thyzy@Rk \
    --to=coiby.xu@gmail.com \
    --cc=Nehal-bakulchandra.Shah@amd.com \
    --cc=Shyam-sundar.S-k@amd.com \
    --cc=availa@outlook.com \
    --cc=hdegoede@redhat.com \
    --cc=linus.walleij@linaro.org \
    --cc=linux-gpio@vger.kernel.org \
    --cc=linux-kernel-mentees@lists.linuxfoundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.