- Debian Testing
- rsyslog
- Log files stored on an external USB SSD
- The root filesystem should remain as untouched as possible, while the
SSD receives all log writes.
If the USB SSD is unexpectedly disconnected, the USB bus resets, or the filesystem becomes unmounted for any reason, the mount point directory
still exists. As I understand it, applications may continue writing to
that directory, which now belongs to the underlying root filesystem. On systems using flash storage, this could eventually wear out the internal storage without immediately being noticed.
On Mon, Jul 27, 2026 at 08:25:40 -0300, Marcelo Laia wrote:That was my own first thought as well, but it only works if you have
- Debian Testing
- rsyslog
- Log files stored on an external USB SSD
- The root filesystem should remain as untouched as possible, while
the SSD receives all log writes.
If the USB SSD is unexpectedly disconnected, the USB bus resets, or
the filesystem becomes unmounted for any reason, the mount point
directory still exists. As I understand it, applications may
continue writing to that directory, which now belongs to the
underlying root filesystem. On systems using flash storage, this
could eventually wear out the internal storage without immediately
being noticed.
My first thought would be to create a subdirectory on the mounted
external device, and write the logs to files within that
subdirectory.
If/when the external device goes away, that subdirectory will not
exist, so you won't be able to create the log files specified in the configuration.
On Mon, Jul 27, 2026 at 08:25:40 -0300, Marcelo Laia wrote:
- Debian Testing
- rsyslog
- Log files stored on an external USB SSD
- The root filesystem should remain as untouched as possible, while the SSD receives all log writes.
If the USB SSD is unexpectedly disconnected, the USB bus resets, or the filesystem becomes unmounted for any reason, the mount point directory still exists. As I understand it, applications may continue writing to
that directory, which now belongs to the underlying root filesystem. On systems using flash storage, this could eventually wear out the internal storage without immediately being noticed.
My first thought would be to create a subdirectory on the mounted external device, and write the logs to files within that subdirectory.
If/when the external device goes away, that subdirectory will not
exist, so you won't be able to create the log files specified in the configuration.
I'm designing a logging system on Debian Testing and would appreciate
some advice regarding a failure scenario that I recently became aware
of.
My initial idea is quite simple:
- Debian Testing
- rsyslog
- Log files stored on an external USB SSD
- The root filesystem should remain as untouched as possible, while the
SSD receives all log writes.
While discussing this design elsewhere, someone pointed out an issue
that I had not considered.
If the USB SSD is unexpectedly disconnected, the USB bus resets, or the filesystem becomes unmounted for any reason, the mount point directory
still exists. As I understand it, applications may continue writing to
that directory, which now belongs to the underlying root filesystem. On systems using flash storage, this could eventually wear out the internal storage without immediately being noticed.
My first thought was to periodically check /proc/mounts (or /proc/self/mountinfo) with a small script and immediately stop rsyslog
if the filesystem is no longer mounted. This would probably work, but it feels like I might be reinventing something that Linux or Debian already provides.
So my questions are:
1. Is there a standard or recommended Debian/Linux approach to prevent
applications from silently writing to the underlying directory after
a mounted filesystem disappears?
2. Are there existing mechanisms in systemd, mount units, udev, rsyslog, or elsewhere that are intended to handle this situation?
3. How would you implement a robust solution for this type of logging
system?
That was my own first thought as well, but it only works if you have
that much control over the applications that will be writing the files
The only other fallback option that's occurred to me is to mount the
root filesystem read-only
The Wanderer (HE12026-07-27):
That was my own first thought as well, but it only works if you have
that much control over the applications that will be writing the files
According to the original statement, that is syslog. If there are other processes logging without going through rsyslog, it makes the issue that
much difficult.
The only other fallback option that's occurred to me is to mount the
root filesystem read-only
If the OP wants to avoid unwanted writes on the internal devices, I very
much hope that is the very first thing they did.
3. How would you implement a robust solution for this type of logging
system?
You can also mount a read-only ramdisk in between the root filesystem
(if it needs to be mounted read-write) and the usb. Assuming the
directory is /mnt/log. Something similar to those lines in /etc/fstab
may (untested) work.
tmpfs /mnt/log tmpfs defaults,ro 0 0
/dev/usb-block /mnt/log ext4 defaults 0 0
In the same vein, there's also
mount -o bind,ro /emptydir /var/log
The motivation here is a little different. I'm trying to understand how
Linux behaves when the target filesystem unexpectedly disappears and how
to prevent applications from silently falling back to the underlying filesystem.
The motivation here is a little different. I'm trying to understand how
Linux behaves when the target filesystem unexpectedly disappears and how
to prevent applications from silently falling back to the underlying filesystem.
The motivation here is a little different. I'm trying to understand how
Linux behaves when the target filesystem unexpectedly disappears and how
to prevent applications from silently falling back to the underlying filesystem.
Hello everyone,
I'm designing a logging system on Debian Testing and would appreciate
some advice regarding a failure scenario that I recently became aware
of.
My initial idea is quite simple:
- Debian Testing
- rsyslog
- Log files stored on an external USB SSD
- The root filesystem should remain as untouched as possible, while
the SSD receives all log writes.
While discussing this design elsewhere, someone pointed out an issue
that I had not considered.
If the USB SSD is unexpectedly disconnected, the USB bus resets, or
the filesystem becomes unmounted for any reason, the mount point
directory still exists. As I understand it, applications may continue
writing to that directory, which now belongs to the underlying root filesystem. On systems using flash storage, this could eventually
wear out the internal storage without immediately being noticed.
My first thought was to periodically check /proc/mounts (or /proc/self/mountinfo) with a small script and immediately stop rsyslog
if the filesystem is no longer mounted. This would probably work, but
it feels like I might be reinventing something that Linux or Debian
already provides.
So my questions are:
1. Is there a standard or recommended Debian/Linux approach to prevent
applications from silently writing to the underlying directory
after a mounted filesystem disappears?
2. Are there existing mechanisms in systemd, mount units, udev,
rsyslog, or elsewhere that are intended to handle this situation?
3. How would you implement a robust solution for this type of logging
system?
I'm not looking for a centralized log server in this particular
project. The goal is simply to make local logging to an external SSD
as reliable as possible while protecting the root filesystem from
unintended writes.
Any suggestions or references to established best practices would be
greatly appreciated.
Thank you very much.
Hi Andy,
Thank you for your comments.
I completely agree that USB storage is not the most reliable solution,
and if this were a conventional server I would probably choose a
different architecture.
The motivation here is a little different. I'm trying to understand
how Linux behaves when the target filesystem unexpectedly disappears
and how to prevent applications from silently falling back to the
underlying filesystem.
The USB SSD is simply a convenient and inexpensive way to reproduce
that failure mode. The same question could arise with removable SATA
devices, SD cards, or other storage that may become unavailable
unexpectedly.
Several people have suggested placing the log files inside a
subdirectory that exists only on the mounted filesystem, which seems
like a very elegant safeguard. I'm also interested in exploring
whether systemd, udev, or the logging daemon itself offer additional mechanisms for handling these situations gracefully.
This makes local directory 'share' root owned and [nobody else has read/write permissions] for it. Since deja-dup is not running as root,
it now stops with 'backup failed' if the share directory did not mount.
But you can't just pick a random USB stick out of a box of them and
get it mounted where the logging daemon will find it
On Mon, Jul 27, 2026 at 11:06:13AM -0300, Marcelo Laia wrote:
The motivation here is a little different. I'm trying to understand how
Linux behaves when the target filesystem unexpectedly disappears and how
to prevent applications from silently falling back to the underlying
filesystem.
In practice I don't think that is a problem that happens.
| Sysop: | Jacob Catayoc |
|---|---|
| Location: | Pasay City, Metro Manila, Philippines |
| Users: | 5 |
| Nodes: | 4 (0 / 4) |
| Uptime: | 498677:21:02 |
| Calls: | 212 |
| Calls today: | 3 |
| Files: | 910 |
| D/L today: |
31 files (22,593K bytes) |
| Messages: | 69,826 |