* linux-mm@kvack.org - limping on a backup @ 2021-06-22 14:59 Benjamin LaHaise 2021-06-24 12:12 ` David Sterba 2021-06-25 17:00 ` James Bottomley 0 siblings, 2 replies; 12+ messages in thread From: Benjamin LaHaise @ 2021-06-22 14:59 UTC (permalink / raw) To: linux-mm, linux-kernel FYI: the filesystem hosting kanga.kvack.org which hosts linux-mm@kvack.org and a few other assorted things was damaged around 9:17am. The mailing list is back up and running from a March 2nd backup for now. The problem is either a bad SSD or a btrfs bug, however no crash dump was captured to help with debugging. New hardware will be deployed this afternoon after an attempt at data recovery is made. -ben -- "Thought is the essence of where you are now." ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: linux-mm@kvack.org - limping on a backup 2021-06-22 14:59 linux-mm@kvack.org - limping on a backup Benjamin LaHaise @ 2021-06-24 12:12 ` David Sterba 2021-06-25 17:00 ` James Bottomley 1 sibling, 0 replies; 12+ messages in thread From: David Sterba @ 2021-06-24 12:12 UTC (permalink / raw) To: Benjamin LaHaise; +Cc: linux-mm, linux-kernel On Tue, Jun 22, 2021 at 10:59:54AM -0400, Benjamin LaHaise wrote: > FYI: the filesystem hosting kanga.kvack.org which hosts linux-mm@kvack.org > and a few other assorted things was damaged around 9:17am. The mailing > list is back up and running from a March 2nd backup for now. The problem > is either a bad SSD or a btrfs bug, however no crash dump was captured to > help with debugging. Do you have any logs? Also when you have a suspicion that it was caused by hardware, what's the SSD type? If you'd like a more interactive discussion please come to the libera.chat #btrfs channel, there are people with a lot of experience and knowledge about buggy hardware. In case it's a btrfs bug we'd be interested to know at least something that could help to narrow it down. d. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: linux-mm@kvack.org - limping on a backup 2021-06-22 14:59 linux-mm@kvack.org - limping on a backup Benjamin LaHaise 2021-06-24 12:12 ` David Sterba @ 2021-06-25 17:00 ` James Bottomley 2021-06-25 17:12 ` Benjamin LaHaise 1 sibling, 1 reply; 12+ messages in thread From: James Bottomley @ 2021-06-25 17:00 UTC (permalink / raw) To: Benjamin LaHaise, linux-mm, linux-kernel On Tue, 2021-06-22 at 10:59 -0400, Benjamin LaHaise wrote: > FYI: the filesystem hosting kanga.kvack.org which hosts > linux-mm@kvack.org > and a few other assorted things was damaged around 9:17am. The > mailing list is back up and running from a March 2nd backup for > now. The problem is either a bad SSD or a btrfs bug, however no > crash dump was captured to help with debugging. New hardware will be > deployed this afternoon after an attempt at data recovery is made. Perhaps it's time to move this list over to vger or the linux.dev infrastructure now that it's being brought up? We already migrated the containers list without too much pain. Regards, James ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: linux-mm@kvack.org - limping on a backup 2021-06-25 17:00 ` James Bottomley @ 2021-06-25 17:12 ` Benjamin LaHaise 2021-06-25 19:21 ` James Bottomley 0 siblings, 1 reply; 12+ messages in thread From: Benjamin LaHaise @ 2021-06-25 17:12 UTC (permalink / raw) To: James Bottomley; +Cc: linux-mm, linux-kernel On Fri, Jun 25, 2021 at 10:00:15AM -0700, James Bottomley wrote: > Perhaps it's time to move this list over to vger or the linux.dev > infrastructure now that it's being brought up? We already migrated the > containers list without too much pain. Maybe the btrfs bugs should get fixed. -ben -- "Thought is the essence of where you are now." ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: linux-mm@kvack.org - limping on a backup 2021-06-25 17:12 ` Benjamin LaHaise @ 2021-06-25 19:21 ` James Bottomley 2021-06-25 19:26 ` Benjamin LaHaise 0 siblings, 1 reply; 12+ messages in thread From: James Bottomley @ 2021-06-25 19:21 UTC (permalink / raw) To: Benjamin LaHaise; +Cc: linux-mm, linux-kernel On Fri, 2021-06-25 at 13:12 -0400, Benjamin LaHaise wrote: > On Fri, Jun 25, 2021 at 10:00:15AM -0700, James Bottomley wrote: > > Perhaps it's time to move this list over to vger or the linux.dev > > infrastructure now that it's being brought up? We already migrated > > the containers list without too much pain. > > Maybe the btrfs bugs should get fixed. I believe we can do both. James ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: linux-mm@kvack.org - limping on a backup 2021-06-25 19:21 ` James Bottomley @ 2021-06-25 19:26 ` Benjamin LaHaise 2021-06-25 21:52 ` James Bottomley 2021-06-28 13:46 ` Jason Gunthorpe 0 siblings, 2 replies; 12+ messages in thread From: Benjamin LaHaise @ 2021-06-25 19:26 UTC (permalink / raw) To: James Bottomley; +Cc: linux-mm, linux-kernel On Fri, Jun 25, 2021 at 12:21:24PM -0700, James Bottomley wrote: > On Fri, 2021-06-25 at 13:12 -0400, Benjamin LaHaise wrote: > > On Fri, Jun 25, 2021 at 10:00:15AM -0700, James Bottomley wrote: > > > Perhaps it's time to move this list over to vger or the linux.dev > > > infrastructure now that it's being brought up? We already migrated > > > the containers list without too much pain. > > > > Maybe the btrfs bugs should get fixed. > > I believe we can do both. If I were unresponsive at fixing issues, I would understand the need to migrate services, but steps to address the failures have already been taken and additional mitigations are planned. If we migrated services every time a piece of hardware failed or we hit a kernel bug, then we wouldn't have any infrastructure left. -ben -- "Thought is the essence of where you are now." ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: linux-mm@kvack.org - limping on a backup 2021-06-25 19:26 ` Benjamin LaHaise @ 2021-06-25 21:52 ` James Bottomley 2021-06-28 13:46 ` Jason Gunthorpe 1 sibling, 0 replies; 12+ messages in thread From: James Bottomley @ 2021-06-25 21:52 UTC (permalink / raw) To: Benjamin LaHaise; +Cc: linux-mm, linux-kernel On Fri, 2021-06-25 at 15:26 -0400, Benjamin LaHaise wrote: > On Fri, Jun 25, 2021 at 12:21:24PM -0700, James Bottomley wrote: > > On Fri, 2021-06-25 at 13:12 -0400, Benjamin LaHaise wrote: > > > On Fri, Jun 25, 2021 at 10:00:15AM -0700, James Bottomley wrote: > > > > Perhaps it's time to move this list over to vger or the > > > > linux.dev infrastructure now that it's being brought up? We > > > > already migrated the containers list without too much pain. > > > > > > Maybe the btrfs bugs should get fixed. > > > > I believe we can do both. > > If I were unresponsive at fixing issues, I would understand the need > to migrate services, but steps to address the failures have already > been taken and additional mitigations are planned. If we migrated > services every time a piece of hardware failed or we hit a kernel > bug, then we wouldn't have any infrastructure left. It's not about response time, it's about the fact that we finally got kerne.org funded via the LF to pay for someone to run our mailing list infrastructure so we no longer have to do it ourselves. We've already transferred the containers mailman list and vger is going to be migrated to it soon. The new infrastructure comes with HA and a whole host of backend CDN data centres in various geographies and public inbox backing, so it should be quite slick. James ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: linux-mm@kvack.org - limping on a backup 2021-06-25 19:26 ` Benjamin LaHaise 2021-06-25 21:52 ` James Bottomley @ 2021-06-28 13:46 ` Jason Gunthorpe 2021-06-28 13:53 ` Benjamin LaHaise 1 sibling, 1 reply; 12+ messages in thread From: Jason Gunthorpe @ 2021-06-28 13:46 UTC (permalink / raw) To: Benjamin LaHaise; +Cc: James Bottomley, linux-mm, linux-kernel On Fri, Jun 25, 2021 at 03:26:07PM -0400, Benjamin LaHaise wrote: > On Fri, Jun 25, 2021 at 12:21:24PM -0700, James Bottomley wrote: > > On Fri, 2021-06-25 at 13:12 -0400, Benjamin LaHaise wrote: > > > On Fri, Jun 25, 2021 at 10:00:15AM -0700, James Bottomley wrote: > > > > Perhaps it's time to move this list over to vger or the linux.dev > > > > infrastructure now that it's being brought up? We already migrated > > > > the containers list without too much pain. > > > > > > Maybe the btrfs bugs should get fixed. > > > > I believe we can do both. > > If I were unresponsive at fixing issues, I would understand the need to > migrate services, Well, the DKIM issue has been left unresolved for a long time. I saw on the bug conversation there seems to be no clear path to fix it? The LF/vger lists don't have this problem. The amount of email impacted via recipient spam filtering seems to be increasing every month, and tools like b4 don't work as intended. It is not some minor complaint. Jason ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: linux-mm@kvack.org - limping on a backup 2021-06-28 13:46 ` Jason Gunthorpe @ 2021-06-28 13:53 ` Benjamin LaHaise 2021-06-28 14:26 ` Jason Gunthorpe 0 siblings, 1 reply; 12+ messages in thread From: Benjamin LaHaise @ 2021-06-28 13:53 UTC (permalink / raw) To: Jason Gunthorpe; +Cc: James Bottomley, linux-mm, linux-kernel On Mon, Jun 28, 2021 at 10:46:07AM -0300, Jason Gunthorpe wrote: > Well, the DKIM issue has been left unresolved for a long time. > I saw on the bug conversation there seems to be no clear path to fix > it? Nobody had ever bothered to provide or figure out a test case. We have one now and I'm working with Tucows to try to sort out a fix. The fact of the matter is that the DKIM spec is broken and doesn't properly address issues relating to transport of emails containing UTF-8 content over SMTP sessions which are limited to 7 bit transport due to backwards compatibility assumptions. -ben -- "Thought is the essence of where you are now." ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: linux-mm@kvack.org - limping on a backup 2021-06-28 13:53 ` Benjamin LaHaise @ 2021-06-28 14:26 ` Jason Gunthorpe 2021-06-28 14:40 ` Benjamin LaHaise 0 siblings, 1 reply; 12+ messages in thread From: Jason Gunthorpe @ 2021-06-28 14:26 UTC (permalink / raw) To: Benjamin LaHaise; +Cc: James Bottomley, linux-mm, linux-kernel On Mon, Jun 28, 2021 at 09:53:52AM -0400, Benjamin LaHaise wrote: > The fact of the matter is that the DKIM spec is broken and doesn't > properly address issues relating to transport of emails containing UTF-8 > content over SMTP sessions which are limited to 7 bit transport due to > backwards compatibility assumptions. Isn't a 7-bit conversion what I pointed at last time we talked about this? DKIM assumes a "modern" mail system, there should not be 7bit conversions in the mail pipeline. Anyone sending DKIM needs to be 8 bit clean. Jason ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: linux-mm@kvack.org - limping on a backup 2021-06-28 14:26 ` Jason Gunthorpe @ 2021-06-28 14:40 ` Benjamin LaHaise 2021-06-28 16:20 ` Jason Gunthorpe 0 siblings, 1 reply; 12+ messages in thread From: Benjamin LaHaise @ 2021-06-28 14:40 UTC (permalink / raw) To: Jason Gunthorpe; +Cc: James Bottomley, linux-mm, linux-kernel On Mon, Jun 28, 2021 at 11:26:59AM -0300, Jason Gunthorpe wrote: > Isn't a 7-bit conversion what I pointed at last time we talked about > this? I changed several options in postfix last time this was raised, but as nobody ever provided a test case, I had no way of knowing if it worked or not. Personally, I think DKIM provides very little value considering that a good chunk of the spam that goes by has valid DKIM signatures, not to mention that it doesn't help with modern phishing attempts much either. > DKIM assumes a "modern" mail system, there should not be 7bit > conversions in the mail pipeline. Anyone sending DKIM needs to be 8 > bit clean. "Be strict in what you send, and be liberal in what you receive." DKIM makes assumptions about the mail transport layer that are not true. If the signatures had been applied on content *after* the quoted printable conversion, this would never have been an issue. DKIM is a poorly done spec that ignores decades of that philosophy at the IETF. And even if a DKIM signature passes, that's still not enough to trust the resulting email. All it does is ensure that a small subset of valid emails get dropped on the floor. This doesn't seem like an overall win. -ben -- "Thought is the essence of where you are now." ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: linux-mm@kvack.org - limping on a backup 2021-06-28 14:40 ` Benjamin LaHaise @ 2021-06-28 16:20 ` Jason Gunthorpe 0 siblings, 0 replies; 12+ messages in thread From: Jason Gunthorpe @ 2021-06-28 16:20 UTC (permalink / raw) To: Benjamin LaHaise; +Cc: James Bottomley, linux-mm, linux-kernel On Mon, Jun 28, 2021 at 10:40:51AM -0400, Benjamin LaHaise wrote: > On Mon, Jun 28, 2021 at 11:26:59AM -0300, Jason Gunthorpe wrote: > > Isn't a 7-bit conversion what I pointed at last time we talked about > > this? > > I changed several options in postfix last time this was raised, but as > nobody ever provided a test case, I had no way of knowing if it worked or > not. I've been using a script like this against the lore public inbox git repos to monitor my own domain's dkim cleanness and interaction with list serves: #!/usr/bin/python3 import subprocess import collections # Starting points start = XXXXX # git commit id string emails = collections.defaultdict(list) commits = subprocess.check_output(["git","log","master","^" + start,'--pretty=format:%H %aN <%aE>']).decode() for ln in commits.splitlines(): commit,_,email = ln.partition(' ') if "nvidia.com" in email.lower(): emails[email].append(commit) fails = set() not_empty = True; while not_empty: not_empty = False; for email,commits in sorted(emails.items()): if email in fails or not commits: continue commit = commits[-1]; del commits[-1] if commits: not_empty = True; msg = subprocess.check_output(["git","show",commit + ":m"]); try: subprocess.check_output(["dkimverify"], input=msg); #print(email) except: fails.add(email) print("Failed!", email, commit) It has taken a lot of doing, but nvidia.com is now effectively DKIM clean through vger. You could run with with some known-good domains like nvidia.com, facebook.com, google.com, to measure kvack's activity. Failures can often be cross-correlated against a vger list and then you can do A/B comparison to guess what is wrong. > spec that ignores decades of that philosophy at the IETF. And even if a > DKIM signature passes, that's still not enough to trust the resulting > email. All it does is ensure that a small subset of valid emails get > dropped on the floor. This doesn't seem like an overall win. I have no idea. It is here, people beyond us have made this decision, we have to work within it. DMARC is ratcheting this up and is moving to say if DKIM fails then emails should be discared. Jason ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-06-28 16:20 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-06-22 14:59 linux-mm@kvack.org - limping on a backup Benjamin LaHaise 2021-06-24 12:12 ` David Sterba 2021-06-25 17:00 ` James Bottomley 2021-06-25 17:12 ` Benjamin LaHaise 2021-06-25 19:21 ` James Bottomley 2021-06-25 19:26 ` Benjamin LaHaise 2021-06-25 21:52 ` James Bottomley 2021-06-28 13:46 ` Jason Gunthorpe 2021-06-28 13:53 ` Benjamin LaHaise 2021-06-28 14:26 ` Jason Gunthorpe 2021-06-28 14:40 ` Benjamin LaHaise 2021-06-28 16:20 ` Jason Gunthorpe
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).