I have four four terabyte hard drives. Each has a partition on it. The
four partitions comprise a RAID 5 array using mdadm. On top of that,
LUKS encryption, then LVM with ext4 logical volumes.
I have four four terabyte hard drives. Each has a partition on it. The
four partitions comprise a RAID 5 array using mdadm. On top of that,
LUKS encryption, then LVM with ext4 logical volumes.
On one LVM partition I have a number of backup files, tarred,
bzipped, and sha256 and sha512 summed. I have a script which will find checksum files, and execute the appropriate program to test the
archives. It puts each program into the background, parallising any
number of checksum tests.
I believe I have found a solution to this problem. I installed the
backports kernel. Since then I have run more than four hours solid of
tests and not found a single error.
I did replace one hard drive. While that resulted in a quieter office,
it did not solve the problem.
Checking voltages from the power supply and the wall with a digital
volt meter did not show any out of spec problems.
I am glad that your storage is working correctly now.
Please run and post the following commands with both the previous
kernel and the backports kernel:
$ cat /etc/debian_version
$ uname -a
I will assume your script spawns a separate, isolated process for
each checksum file.
If you have ruled out the power supply, memory, and disks, another possibility could be a race condition in the kernel and/or I/O stack
that is triggered when multiple processes access storage in parallel.
Are the checksum errors repeatable on another computer with a similar storage architecture and the previous kernel? If so, do they
disappear with the backports kernel?
David
On Mon, 22 Jun 2026 00:32:19 -0700 David Christensen wrote:
Please run and post the following commands with both the previous
kernel and the backports kernel:
$ cat /etc/debian_version
$ uname -a
Backport
root@hawk:~# cat /etc/debian_version
13.5
root@hawk:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"
VERSION_ID="13"
VERSION="13 (trixie)"
VERSION_CODENAME=trixie
DEBIAN_VERSION_FULL=13.5
ID=debian
HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"
root@hawk:~# uname -s
Linux
root@hawk:~# uname -a
Linux hawk 7.0.10+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 7.0.10-1~bpo13+1 (2026-05-28) x86_64 GNU/Linux
root@hawk:~#
Prior kernel is the same except the kernel is
linux-image-6.12.90+deb13.1-amd64 6.12.90-2 amd64
I think that this problem first showed up late April or early May. If
that is correct, the following extracts from my apt logs might help.
These were copied and pasted unwrapped, so unless they were mangled in transit, long lines should be preserved.
...
Apparently, twice in April I had tried the backports kernel(s), and
found them unsatisfactory. So possibly the fix came in between linux-image-6.19.11+deb13-amd64 and the current backports kernel, linux-image-7.0.10+deb13-amd64.
Quite possibly a major version number
might even inadvertently fix something as subtle as this.
I will assume your script spawns a separate, isolated process for
each checksum file.
Correct. It does a find on *.sha256sums, *.sha512sums, and several other suffixes. It then sliced and dices to figure out the appropriate
program to call, sha256sum and sha512sum, respectively.
The files are created using paths relative to the current directory, so
when the script runs, it will pushd to that directory
The key line is
nice "${prog}" "${opts}" -c "${file}" &
Where $prog is the result of the slicing and dicing, opts='--quiet',
and $file the file to be scanned.
I recently added the option to limit the number of background tasks to
the number of processors (nproc --all). That reduces but does not
eliminate the number of errors.
If you have ruled out the power supply, memory, and disks, another
possibility could be a race condition in the kernel and/or I/O stack
that is triggered when multiple processes access storage in parallel.
I wouldn't say those are all ruled out. But the fact that the
backport kernel is a major version number change, and that it appears
to have solved the problem is highly suggestive. With that caveat, I
concur. That is definitely something for the kernel folks to look at.
Are the checksum errors repeatable on another computer with a similar
storage architecture and the previous kernel? If so, do they
disappear with the backports kernel?
I have a much more recent and much faster computer, peregrine, with nvme storage and 12 cores. Hawk has eight cores, and spinning rust. Hawk is
where the problem has shown up. Peregrine does not show the problem.
For 18 gig of data, hawk: 2m18.318s, peregrine 0m24.233s.
On 6/22/26 09:54, Charles Curley wrote:...assuming that there aren't any commits broken for other reasons, or otherwise commits where the codebase can't be built far enough for the
I think that this problem first showed up late April or early May.
If that is correct, the following extracts from my apt logs might
help. These were copied and pasted unwrapped, so unless they were
mangled in transit, long lines should be preserved.
...
Apparently, twice in April I had tried the backports kernel(s),
and found them unsatisfactory. So possibly the fix came in between
linux-image-6.19.11+deb13-amd64 and the current backports kernel,
linux-image-7.0.10+deb13-amd64.
AIUI if someone can come up with a shell script or program whose exit
value reliably indicates the presence or absence of a bug, Git can
do a binary search over a range of commits and locate the commit
where the bug originated.
It looks like there is a newer kernel for Trixie. It is best to file
bug reports against current packages. Can you test it?
https://packages.debian.org/stable/kernel/linux-image-6.12.94+deb13-amd64
On Mon, 22 Jun 2026 15:40:11 -0700
David Christensen <dpchrist@holgerdanske.com> wrote:
It looks like there is a newer kernel for Trixie. It is best to file
bug reports against current packages. Can you test it?
https://packages.debian.org/stable/kernel/linux-image-6.12.94+deb13-amd64
Tested. It came up with all sorts of fails.
charles@hawk:~$ uname -a
Linux hawk 6.12.94+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.94-1 (2026-06-20) x86_64 GNU/Linux
charles@hawk:~$
I gather you are suggesting I file a bug against this kernel, linux-image-6.12.94+deb13-amd64 6.12.94-1.
That would make sense, especially since Linux 6.12 appears to be the
kernel for Debian Stable (Trixie):
https://packages.debian.org/trixie/all/allpackages
On Thu, 25 Jun 2026 18:07:21 -0700 David Christensen wrote:
That would make sense, especially since Linux 6.12 appears to be the
kernel for Debian Stable (Trixie):
https://packages.debian.org/trixie/all/allpackages
OK, will do.
However, things are back up in the air. I rebooted to 7.0.10, and ran
some backups. The prior testing has been all reading: checksum
verification. Nothing on the disk actually changed. This backs from the
SSD to the RAID array. I had several directories fail with the error
message "failed: Bad message (74)", whatever that means.
I immediately fscked the logical volume. No errors. I then diffed the directories against the originals. There were for instances of files
found in the backups and not in the originals. Otherwise the "failed" directories were intact and duplicated the originals. I conjecture that
rsync attempted to delete them and failed to do so. I conjecture that
the next pass will get them.
| Sysop: | Jacob Catayoc |
|---|---|
| Location: | Pasay City, Metro Manila, Philippines |
| Users: | 4 |
| Nodes: | 4 (0 / 4) |
| Uptime: | 497099:28:32 |
| Calls: | 182 |
| Files: | 744 |
| D/L today: |
6 files (5,132K bytes) |
| Messages: | 73,558 |