From: Launchpad Bug Tracker <1793904@bugs.launchpad.net>
To: qemu-devel@nongnu.org
Subject: [Bug 1793904] Re: files are randomly overwritten by Zero Bytes
Date: Fri, 02 Jul 2021 04:17:19 -0000 [thread overview]
Message-ID: <162519943933.31921.16636722506680918513.malone@loganberry.canonical.com> (raw)
In-Reply-To: 153763821634.24720.16203204034487714411.malonedeb@gac.canonical.com
[Expired for QEMU because there has been no activity for 60 days.]
** Changed in: qemu
Status: Incomplete => Expired
--
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1793904
Title:
files are randomly overwritten by Zero Bytes
Status in QEMU:
Expired
Bug description:
Hello together,
I am currently tracking down a "Hard to reproduce" bug on my systems
that I first discovered during gitlab installation:
Here is the Text from the Gitlab Bug https://gitlab.com/gitlab-org/gitlab-ce/issues/51023
----------------------------------------------------------------------------------------------
Steps to reproduce
I still do not have all the steps together to reproduce, so far it is:
apt install gitlab-ce and
gitlab-rake backup:recovery
Then it works for some time before it fails.
What is the current bug behavior?
I have a 12 hour old Installation of gitlab ce 11.2.3-ce.0 for debian
stretch on a fresh debian stretch system together with our imported
data. However it turns out that some gitlab related files contain Zero
bytes instead of actual data.
root@gitlab:~# xxd -l 16 /opt/gitlab/bin/gitlab-ctl
00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
This behaviour is somewhat strange because it was working for a few
minutes/hours. I did write a shell script to find out which files are
affected of this memory loss. It turns out that only files located
under /opt/gitlab are affected, if I rule out files like
/var/log/faillog and some postgresql table files.
What I find even stranger is that it does not seem to affect
Logfiles/databases/git_repositorys but application files, like .rb
scripts. and not all of them. No non gitlab package is affected.
What is the expected correct behavior?
Binarys and .rb files should stay as they are.
Possible fixes
I am still investigating, I hope that it is not an infrastructure problem (libvirt/qemu/glusterfs) it can still be one but the point that files of /opt/gitlab are affected and not any logfile and that we to not have similar problems with any other system leads me to the application for now.
If I would have used docker the same problem might have caused a reboot of the container.
But for the Debian package it is a bit of work to recover. That is all a workaround, however.
---------------------------------------------------------------------------------------------
I do have found 2 more systems having the same problem with different
software:
root@erp:~# xxd -l 16 /usr/share/perl/5.26.2/constant.pm
00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
The Filesize itself is, compared with another machine 00001660 Bytes
for both the corrupted and the intact file. It looks to me from the
outside that if some data in the qcow2 file is written too many bytes
get written so it sometimes overwites data of existing files located
right after the position in memory where the write goes to.
I would like to rule out Linux+Ext4 filesystems because I find it
highly unlikely that such an error keeps undiscovered in that part of
the environment for long. I think the same might go for qemu.
Which leaves qemu, gemu+gluster:// mount, qcow2 volumes, glusterfs,
network. So I am now going to check if I can find any system which
gets its volumes via fusermount instead of gluster:// path if the
error is gone there. This may take a while.
----- some software versions---------------
QEMU emulator version 2.12.0 (Debian 1:2.12+dfsg-3)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
libvirt-daemon-driver-storage-gluster/testing,unstable,now 4.6.0-2
amd64 [installed]
ii glusterfs-client 4.1.3-1 amd64
To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1793904/+subscriptions
prev parent reply other threads:[~2021-07-02 4:37 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <153763821634.24720.16203204034487714411.malonedeb@gac.canonical.com>
2019-04-21 11:05 ` [Qemu-devel] [Bug 1793904] Re: files are randomly overwritten by Zero Bytes Hans
2019-04-23 5:31 ` Hans
2019-04-29 19:58 ` Hans
2021-04-29 9:53 ` Thomas Huth
2021-05-02 5:50 ` Thomas Huth
2021-06-18 15:31 ` Thomas Huth
2021-07-02 4:17 ` Launchpad Bug Tracker [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=162519943933.31921.16636722506680918513.malone@loganberry.canonical.com \
--to=1793904@bugs.launchpad.net \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.