From: Viresh Kumar <viresh.kumar@linaro.org> To: Rob Clark <robdclark@gmail.com> Cc: Daniel Vetter <daniel@ffwll.ch>, dri-devel <dri-devel@lists.freedesktop.org>, Rob Clark <robdclark@chromium.org>, Sean Paul <sean@poorly.run>, David Airlie <airlied@linux.ie>, "open list:DRM DRIVER FOR MSM ADRENO GPU" <linux-arm-msm@vger.kernel.org>, "open list:DRM DRIVER FOR MSM ADRENO GPU" <freedreno@lists.freedesktop.org>, open list <linux-kernel@vger.kernel.org>, "Menon, Nishanth" <nm@ti.com> Subject: Re: [PATCH v2 07/22] drm/msm: Do rpm get sooner in the submit path Date: Thu, 19 Nov 2020 11:35:28 +0530 [thread overview] Message-ID: <20201119060528.qscedvc4jlmxakqo@vireshk-i7> (raw) In-Reply-To: <CAF6AEGv=-h7GFj5LR97FkeBBn+gk6TNS5hZkwBwufpE4yO7GyA@mail.gmail.com> On 18-11-20, 08:53, Rob Clark wrote: > On Tue, Nov 17, 2020 at 9:28 PM Viresh Kumar <viresh.kumar@linaro.org> wrote: > > > > On 17-11-20, 09:02, Rob Clark wrote: > > > With that on top of the previous patch, > > > > Don't you still have this ? Which fixed the lockdep in the remove path. > > > > https://lore.kernel.org/lkml/20201022080644.2ck4okrxygmkuatn@vireshk-i7/ > > > > To make it clear you need these patches to fix the OPP stuff: > > > > //From 5.10-rc3 (the one from the above link). > > commit e0df59de670b ("opp: Reduce the size of critical section in _opp_table_kref_release()") This fixes debugfs stuff while the OPP table is removed. > > //Below two from linux-next > > commit ef43f01ac069 ("opp: Always add entries in dev_list with opp_table->lock held") > > commit 27c09484dd3d ("opp: Allocate the OPP table outside of opp_table_lock") This fixes debugfs stuff while the OPP table is added. > > This matches the diff I gave you earlier. > > > > no, I did not have all three, only "opp: Allocate the OPP table > outside of opp_table_lock" plus the fixup. But with all three: And looking at the lockdep you gave now, it looks like we have a problem with OPP table's internal lock (opp_table->lock) as well apart from the global opp_table_lock. I wish there was a way for me to reproduce the lockdep :( I know this is exhausting for both of us and I really want to be over with it as soon as possible, this really should be the last patch here, please try this along with other two. This fixes the debugfs thing while the OPPs in the OPP table are removed (they are already added without a lock around debugfs stuff). AFAIU, there is no further debugfs stuff that happens from within the locks and so this really should be the last patch unless I missed something. -- viresh -------------------------8<------------------------- From: Viresh Kumar <viresh.kumar@linaro.org> Date: Thu, 19 Nov 2020 11:24:32 +0530 Subject: [PATCH] opp: Reduce the size of critical section in _opp_kref_release() There is a lot of stuff here which can be done outside of the opp_table->lock, do that. This helps avoiding a circular dependency lockdeps around debugfs. Reported-by: Rob Clark <robdclark@gmail.com> Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org> --- drivers/opp/core.c | 94 +++++++++++++++++++++++----------------------- 1 file changed, 47 insertions(+), 47 deletions(-) diff --git a/drivers/opp/core.c b/drivers/opp/core.c index 9d145bb99a59..4268eb359915 100644 --- a/drivers/opp/core.c +++ b/drivers/opp/core.c @@ -1251,9 +1251,14 @@ void _opp_free(struct dev_pm_opp *opp) kfree(opp); } -static void _opp_kref_release(struct dev_pm_opp *opp, - struct opp_table *opp_table) +static void _opp_kref_release(struct kref *kref) { + struct dev_pm_opp *opp = container_of(kref, struct dev_pm_opp, kref); + struct opp_table *opp_table = opp->opp_table; + + list_del(&opp->node); + mutex_unlock(&opp_table->lock); + /* * Notify the changes in the availability of the operable * frequency/voltage list. @@ -1261,27 +1266,9 @@ static void _opp_kref_release(struct dev_pm_opp *opp, blocking_notifier_call_chain(&opp_table->head, OPP_EVENT_REMOVE, opp); _of_opp_free_required_opps(opp_table, opp); opp_debug_remove_one(opp); - list_del(&opp->node); kfree(opp); } -static void _opp_kref_release_unlocked(struct kref *kref) -{ - struct dev_pm_opp *opp = container_of(kref, struct dev_pm_opp, kref); - struct opp_table *opp_table = opp->opp_table; - - _opp_kref_release(opp, opp_table); -} - -static void _opp_kref_release_locked(struct kref *kref) -{ - struct dev_pm_opp *opp = container_of(kref, struct dev_pm_opp, kref); - struct opp_table *opp_table = opp->opp_table; - - _opp_kref_release(opp, opp_table); - mutex_unlock(&opp_table->lock); -} - void dev_pm_opp_get(struct dev_pm_opp *opp) { kref_get(&opp->kref); @@ -1289,16 +1276,10 @@ void dev_pm_opp_get(struct dev_pm_opp *opp) void dev_pm_opp_put(struct dev_pm_opp *opp) { - kref_put_mutex(&opp->kref, _opp_kref_release_locked, - &opp->opp_table->lock); + kref_put_mutex(&opp->kref, _opp_kref_release, &opp->opp_table->lock); } EXPORT_SYMBOL_GPL(dev_pm_opp_put); -static void dev_pm_opp_put_unlocked(struct dev_pm_opp *opp) -{ - kref_put(&opp->kref, _opp_kref_release_unlocked); -} - /** * dev_pm_opp_remove() - Remove an OPP from OPP table * @dev: device for which we do this operation @@ -1342,30 +1323,49 @@ void dev_pm_opp_remove(struct device *dev, unsigned long freq) } EXPORT_SYMBOL_GPL(dev_pm_opp_remove); +static struct dev_pm_opp *_opp_get_next(struct opp_table *opp_table, + bool dynamic) +{ + struct dev_pm_opp *opp = NULL, *temp; + + mutex_lock(&opp_table->lock); + list_for_each_entry(temp, &opp_table->opp_list, node) { + if (dynamic == temp->dynamic) { + opp = temp; + break; + } + } + + mutex_unlock(&opp_table->lock); + return opp; +} + bool _opp_remove_all_static(struct opp_table *opp_table) { - struct dev_pm_opp *opp, *tmp; - bool ret = true; + struct dev_pm_opp *opp; mutex_lock(&opp_table->lock); if (!opp_table->parsed_static_opps) { - ret = false; - goto unlock; + mutex_unlock(&opp_table->lock); + return false; } - if (--opp_table->parsed_static_opps) - goto unlock; - - list_for_each_entry_safe(opp, tmp, &opp_table->opp_list, node) { - if (!opp->dynamic) - dev_pm_opp_put_unlocked(opp); + if (--opp_table->parsed_static_opps) { + mutex_unlock(&opp_table->lock); + return true; } -unlock: mutex_unlock(&opp_table->lock); - return ret; + /* + * Can't remove the OPP from under the lock, debugfs removal needs to + * happen lock less to avoid circular dependency issues. + */ + while ((opp = _opp_get_next(opp_table, false))) + dev_pm_opp_put(opp); + + return true; } /** @@ -1377,21 +1377,21 @@ bool _opp_remove_all_static(struct opp_table *opp_table) void dev_pm_opp_remove_all_dynamic(struct device *dev) { struct opp_table *opp_table; - struct dev_pm_opp *opp, *temp; + struct dev_pm_opp *opp; int count = 0; opp_table = _find_opp_table(dev); if (IS_ERR(opp_table)) return; - mutex_lock(&opp_table->lock); - list_for_each_entry_safe(opp, temp, &opp_table->opp_list, node) { - if (opp->dynamic) { - dev_pm_opp_put_unlocked(opp); - count++; - } + /* + * Can't remove the OPP from under the lock, debugfs removal needs to + * happen lock less to avoid circular dependency issues. + */ + while ((opp = _opp_get_next(opp_table, true))) { + dev_pm_opp_put(opp); + count++; } - mutex_unlock(&opp_table->lock); /* Drop the references taken by dev_pm_opp_add() */ while (count--)
WARNING: multiple messages have this Message-ID (diff)
From: Viresh Kumar <viresh.kumar@linaro.org> To: Rob Clark <robdclark@gmail.com> Cc: Rob Clark <robdclark@chromium.org>, "open list:DRM DRIVER FOR MSM ADRENO GPU" <freedreno@lists.freedesktop.org>, "Menon, Nishanth" <nm@ti.com>, David Airlie <airlied@linux.ie>, "open list:DRM DRIVER FOR MSM ADRENO GPU" <linux-arm-msm@vger.kernel.org>, open list <linux-kernel@vger.kernel.org>, dri-devel <dri-devel@lists.freedesktop.org>, Sean Paul <sean@poorly.run> Subject: Re: [PATCH v2 07/22] drm/msm: Do rpm get sooner in the submit path Date: Thu, 19 Nov 2020 11:35:28 +0530 [thread overview] Message-ID: <20201119060528.qscedvc4jlmxakqo@vireshk-i7> (raw) In-Reply-To: <CAF6AEGv=-h7GFj5LR97FkeBBn+gk6TNS5hZkwBwufpE4yO7GyA@mail.gmail.com> On 18-11-20, 08:53, Rob Clark wrote: > On Tue, Nov 17, 2020 at 9:28 PM Viresh Kumar <viresh.kumar@linaro.org> wrote: > > > > On 17-11-20, 09:02, Rob Clark wrote: > > > With that on top of the previous patch, > > > > Don't you still have this ? Which fixed the lockdep in the remove path. > > > > https://lore.kernel.org/lkml/20201022080644.2ck4okrxygmkuatn@vireshk-i7/ > > > > To make it clear you need these patches to fix the OPP stuff: > > > > //From 5.10-rc3 (the one from the above link). > > commit e0df59de670b ("opp: Reduce the size of critical section in _opp_table_kref_release()") This fixes debugfs stuff while the OPP table is removed. > > //Below two from linux-next > > commit ef43f01ac069 ("opp: Always add entries in dev_list with opp_table->lock held") > > commit 27c09484dd3d ("opp: Allocate the OPP table outside of opp_table_lock") This fixes debugfs stuff while the OPP table is added. > > This matches the diff I gave you earlier. > > > > no, I did not have all three, only "opp: Allocate the OPP table > outside of opp_table_lock" plus the fixup. But with all three: And looking at the lockdep you gave now, it looks like we have a problem with OPP table's internal lock (opp_table->lock) as well apart from the global opp_table_lock. I wish there was a way for me to reproduce the lockdep :( I know this is exhausting for both of us and I really want to be over with it as soon as possible, this really should be the last patch here, please try this along with other two. This fixes the debugfs thing while the OPPs in the OPP table are removed (they are already added without a lock around debugfs stuff). AFAIU, there is no further debugfs stuff that happens from within the locks and so this really should be the last patch unless I missed something. -- viresh -------------------------8<------------------------- From: Viresh Kumar <viresh.kumar@linaro.org> Date: Thu, 19 Nov 2020 11:24:32 +0530 Subject: [PATCH] opp: Reduce the size of critical section in _opp_kref_release() There is a lot of stuff here which can be done outside of the opp_table->lock, do that. This helps avoiding a circular dependency lockdeps around debugfs. Reported-by: Rob Clark <robdclark@gmail.com> Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org> --- drivers/opp/core.c | 94 +++++++++++++++++++++++----------------------- 1 file changed, 47 insertions(+), 47 deletions(-) diff --git a/drivers/opp/core.c b/drivers/opp/core.c index 9d145bb99a59..4268eb359915 100644 --- a/drivers/opp/core.c +++ b/drivers/opp/core.c @@ -1251,9 +1251,14 @@ void _opp_free(struct dev_pm_opp *opp) kfree(opp); } -static void _opp_kref_release(struct dev_pm_opp *opp, - struct opp_table *opp_table) +static void _opp_kref_release(struct kref *kref) { + struct dev_pm_opp *opp = container_of(kref, struct dev_pm_opp, kref); + struct opp_table *opp_table = opp->opp_table; + + list_del(&opp->node); + mutex_unlock(&opp_table->lock); + /* * Notify the changes in the availability of the operable * frequency/voltage list. @@ -1261,27 +1266,9 @@ static void _opp_kref_release(struct dev_pm_opp *opp, blocking_notifier_call_chain(&opp_table->head, OPP_EVENT_REMOVE, opp); _of_opp_free_required_opps(opp_table, opp); opp_debug_remove_one(opp); - list_del(&opp->node); kfree(opp); } -static void _opp_kref_release_unlocked(struct kref *kref) -{ - struct dev_pm_opp *opp = container_of(kref, struct dev_pm_opp, kref); - struct opp_table *opp_table = opp->opp_table; - - _opp_kref_release(opp, opp_table); -} - -static void _opp_kref_release_locked(struct kref *kref) -{ - struct dev_pm_opp *opp = container_of(kref, struct dev_pm_opp, kref); - struct opp_table *opp_table = opp->opp_table; - - _opp_kref_release(opp, opp_table); - mutex_unlock(&opp_table->lock); -} - void dev_pm_opp_get(struct dev_pm_opp *opp) { kref_get(&opp->kref); @@ -1289,16 +1276,10 @@ void dev_pm_opp_get(struct dev_pm_opp *opp) void dev_pm_opp_put(struct dev_pm_opp *opp) { - kref_put_mutex(&opp->kref, _opp_kref_release_locked, - &opp->opp_table->lock); + kref_put_mutex(&opp->kref, _opp_kref_release, &opp->opp_table->lock); } EXPORT_SYMBOL_GPL(dev_pm_opp_put); -static void dev_pm_opp_put_unlocked(struct dev_pm_opp *opp) -{ - kref_put(&opp->kref, _opp_kref_release_unlocked); -} - /** * dev_pm_opp_remove() - Remove an OPP from OPP table * @dev: device for which we do this operation @@ -1342,30 +1323,49 @@ void dev_pm_opp_remove(struct device *dev, unsigned long freq) } EXPORT_SYMBOL_GPL(dev_pm_opp_remove); +static struct dev_pm_opp *_opp_get_next(struct opp_table *opp_table, + bool dynamic) +{ + struct dev_pm_opp *opp = NULL, *temp; + + mutex_lock(&opp_table->lock); + list_for_each_entry(temp, &opp_table->opp_list, node) { + if (dynamic == temp->dynamic) { + opp = temp; + break; + } + } + + mutex_unlock(&opp_table->lock); + return opp; +} + bool _opp_remove_all_static(struct opp_table *opp_table) { - struct dev_pm_opp *opp, *tmp; - bool ret = true; + struct dev_pm_opp *opp; mutex_lock(&opp_table->lock); if (!opp_table->parsed_static_opps) { - ret = false; - goto unlock; + mutex_unlock(&opp_table->lock); + return false; } - if (--opp_table->parsed_static_opps) - goto unlock; - - list_for_each_entry_safe(opp, tmp, &opp_table->opp_list, node) { - if (!opp->dynamic) - dev_pm_opp_put_unlocked(opp); + if (--opp_table->parsed_static_opps) { + mutex_unlock(&opp_table->lock); + return true; } -unlock: mutex_unlock(&opp_table->lock); - return ret; + /* + * Can't remove the OPP from under the lock, debugfs removal needs to + * happen lock less to avoid circular dependency issues. + */ + while ((opp = _opp_get_next(opp_table, false))) + dev_pm_opp_put(opp); + + return true; } /** @@ -1377,21 +1377,21 @@ bool _opp_remove_all_static(struct opp_table *opp_table) void dev_pm_opp_remove_all_dynamic(struct device *dev) { struct opp_table *opp_table; - struct dev_pm_opp *opp, *temp; + struct dev_pm_opp *opp; int count = 0; opp_table = _find_opp_table(dev); if (IS_ERR(opp_table)) return; - mutex_lock(&opp_table->lock); - list_for_each_entry_safe(opp, temp, &opp_table->opp_list, node) { - if (opp->dynamic) { - dev_pm_opp_put_unlocked(opp); - count++; - } + /* + * Can't remove the OPP from under the lock, debugfs removal needs to + * happen lock less to avoid circular dependency issues. + */ + while ((opp = _opp_get_next(opp_table, true))) { + dev_pm_opp_put(opp); + count++; } - mutex_unlock(&opp_table->lock); /* Drop the references taken by dev_pm_opp_add() */ while (count--) _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
next prev parent reply other threads:[~2020-11-19 6:05 UTC|newest] Thread overview: 100+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-10-12 2:09 [PATCH 00/14] drm/msm: de-struct_mutex-ification Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 01/22] drm/msm/gem: Add obj->lock wrappers Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 02/22] drm/msm/gem: Rename internal get_iova_locked helper Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 03/22] drm/msm/gem: Move prototypes to msm_gem.h Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 04/22] drm/msm/gem: Add some _locked() helpers Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 05/22] drm/msm/gem: Move locking in shrinker path Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 06/22] drm/msm/submit: Move copy_from_user ahead of locking bos Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 07/22] drm/msm: Do rpm get sooner in the submit path Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 14:35 ` Daniel Vetter 2020-10-12 14:35 ` Daniel Vetter 2020-10-12 15:43 ` Rob Clark 2020-10-12 15:43 ` Rob Clark 2020-10-20 9:07 ` Viresh Kumar 2020-10-20 9:07 ` Viresh Kumar 2020-10-20 10:56 ` Daniel Vetter 2020-10-20 10:56 ` Daniel Vetter 2020-10-20 11:24 ` Viresh Kumar 2020-10-20 11:24 ` Viresh Kumar 2020-10-20 11:42 ` Daniel Vetter 2020-10-20 11:42 ` Daniel Vetter 2020-10-20 14:13 ` Rob Clark 2020-10-20 14:13 ` Rob Clark 2020-10-22 8:06 ` Viresh Kumar 2020-10-22 8:06 ` Viresh Kumar 2020-10-25 17:39 ` Rob Clark 2020-10-25 17:39 ` Rob Clark 2020-10-27 11:35 ` Viresh Kumar 2020-10-27 11:35 ` Viresh Kumar 2020-11-03 5:47 ` Viresh Kumar 2020-11-03 5:47 ` Viresh Kumar 2020-11-03 16:50 ` Rob Clark 2020-11-03 16:50 ` Rob Clark 2020-11-04 3:03 ` Viresh Kumar 2020-11-04 3:03 ` Viresh Kumar 2020-11-05 19:24 ` Rob Clark 2020-11-05 19:24 ` Rob Clark 2020-11-06 7:16 ` Viresh Kumar 2020-11-06 7:16 ` Viresh Kumar 2020-11-17 10:03 ` Viresh Kumar 2020-11-17 10:03 ` Viresh Kumar 2020-11-17 17:02 ` Rob Clark 2020-11-17 17:02 ` Rob Clark 2020-11-18 5:28 ` Viresh Kumar 2020-11-18 5:28 ` Viresh Kumar 2020-11-18 16:53 ` Rob Clark 2020-11-18 16:53 ` Rob Clark 2020-11-19 6:05 ` Viresh Kumar [this message] 2020-11-19 6:05 ` Viresh Kumar 2020-12-07 6:16 ` Viresh Kumar 2020-12-07 6:16 ` Viresh Kumar 2020-12-16 5:22 ` Viresh Kumar 2020-12-16 5:22 ` Viresh Kumar 2020-10-12 2:09 ` [PATCH v2 08/22] drm/msm/gem: Switch over to obj->resv for locking Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 09/22] drm/msm: Use correct drm_gem_object_put() in fail case Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 10/22] drm/msm: Drop chatty trace Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 11/22] drm/msm: Move update_fences() Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 12/22] drm/msm: Add priv->mm_lock to protect active/inactive lists Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 13/22] drm/msm: Document and rename preempt_lock Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 14/22] drm/msm: Protect ring->submits with it's own lock Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 15/22] drm/msm: Refcount submits Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 16/22] drm/msm: Remove obj->gpu Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 17/22] drm/msm: Drop struct_mutex from the retire path Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 18/22] drm/msm: Drop struct_mutex in free_object() path Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 19/22] drm/msm: remove msm_gem_free_work Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 20/22] drm/msm: drop struct_mutex in madvise path Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 21/22] drm/msm: Drop struct_mutex in shrinker path Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 2:09 ` [PATCH v2 22/22] drm/msm: Don't implicit-sync if only a single ring Rob Clark 2020-10-12 2:09 ` Rob Clark 2020-10-12 14:40 ` Daniel Vetter 2020-10-12 14:40 ` Daniel Vetter 2020-10-12 15:07 ` Rob Clark 2020-10-12 15:07 ` Rob Clark 2020-10-13 11:08 ` Daniel Vetter 2020-10-13 11:08 ` Daniel Vetter 2020-10-13 16:15 ` [Freedreno] " Rob Clark 2020-10-13 16:15 ` Rob Clark 2020-10-15 8:22 ` Daniel Vetter 2020-10-15 8:22 ` Daniel Vetter
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20201119060528.qscedvc4jlmxakqo@vireshk-i7 \ --to=viresh.kumar@linaro.org \ --cc=airlied@linux.ie \ --cc=daniel@ffwll.ch \ --cc=dri-devel@lists.freedesktop.org \ --cc=freedreno@lists.freedesktop.org \ --cc=linux-arm-msm@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=nm@ti.com \ --cc=robdclark@chromium.org \ --cc=robdclark@gmail.com \ --cc=sean@poorly.run \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.