Sorry for the lengthy message, but I don't know how to better compress
the facts...

- I have a data aquisition device which delivers data with ~2.8MB/s to
an IPC (fanless Industrial PC, Intel core i7-1185G7E).

- When receiving a START signal, data are stored to SSD, and an analysis
program is started. The analysis program in turn starts several
sub-processes, which in turn start several sub-sub-processes.
All sub-sub-processes read the recorded data files in a "tail -f" fashion.

- When receiving the STOP signal after 5secs, recording is stopped and
the analysis result is collected from the sub-sub-processes.


* Crucial point is the time it takes from STOP to RESULT-RECEIVED:


We started on Windows-11:
- IPC, Windows-11 with all updates
(11th Gen Intel(R) Core(TM) i7-1185G7E @ 2.80GHz, 8 logical processors)

Result-recv[s] Count
----------------------
2.20 2
2.25 82
2.30 328
2.35 767
2.40 857
2.45 632
2.50 372
2.55 136
2.60 40
2.65 16
2.70 1
2.80 1

Approx. 2.4s to "Result-received", with some variation,

but fairly constant. So far so good.

The idea now was that Linux/Debian-13 should perform better on
identical hardware.


- the very same IPC, Debian-13, with all patches
(11th Gen Intel(R) Core(TM) i7-1185G7E @ 2.80GHz, 8 logical processors)
Result-recv[s] Count
----------------------
0.60 1
0.65 75
0.70 137
0.75 109
0.80 21
0.85 8
0.90 29
0.95 18
1.00 5
1.45 1
1.50 1
1.55 2
1.60 10
1.65 17
1.70 27
1.75 177
1.80 532
1.85 384
1.90 92
1.95 39
2.00 15
2.45 1
2.50 1
2.55 4
2.60 6
2.65 102
2.70 365
2.75 395
2.80 604
2.85 562
2.90 115
2.95 19
3.00 7
3.45 3
3.50 9
3.55 12
3.60 17
3.65 15
3.70 21
3.75 31
3.80 39
3.85 21
3.90 7
3.95 1

There are several clusters of answer times, all ~1s apart.
the majority of answers are *later* than on Windows

I don't understand
- why the performance varies that much,
- why the performance is worse than on Windows on the very same PC hardware.

I have tried:
- echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
- no difference
- sudo powerprofilesctl set performance
- no difference
- cpupower-gui: set performance mode, limit min-CPU-Freq to 2500
- no difference
- intel_pstate=disabled at boot
- performance even worse
- intel_pstate=passive at boot, cpu-governor schedutil
- slightly worse performance
- Debian-12, same software
- no difference



For comparison, the very same program on my Desktop PC (with
cpu-governor 'schedutil', since there is a different kernel module at work)
- Desktop PC, Debian-13 with all patches
(Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 8 logical processors)
Result-recv[s] Count
0.40 2
0.45 1
0.50 1
0.55 4
0.60 14
0.65 21
0.70 46
0.75 68
0.80 82
0.85 55
0.90 16
0.95 8
1.00 3
1.05 1

This is what I had expected on the IPC as well.


The problem seems to be that some of the sub-sub-processes are started
very late, not until STOP is received:

local_time[ms] what
----------------------
17817764 68904 MAIN "START" sub-1 started

17817764 69213 sub-1: OPENED sub-1-1
17817764 69215 sub-1: OPENED sub-1-2
17817764 69216 sub-1: OPENED SUB-1-3

17817764 71455 sub-1-2: opened sub-1-2-1
17817764 71455 sub-1-2: opened sub-1-2-2

17817764 73451 sub-1-3: opened sub-1-3-1
17817764 73452 sub-1-3: opened sub-1-3-2

17817764 73905 MAIN "STOP" sub-1 close, wait for result

Note: NO "sub-1-1 OPENED" yet!

17817764 74068 sub-1-3: finished sub-1-3-1
17817764 74072 sub-1-2: finished sub-1-2-1
17817764 74076 sub-1-3: finished sub-1-3-2
17817764 74081 sub-1-2: finished sub-1-3-2

17817764 74165 sub-1: CLOSED sub-1-2
17817764 74166 sub-1: CLOSED sub-1-3

Only now arrive the "sub-1-1: opened..." ! It looks like the
scheduler postponed running sub-1-1 until now...

17817764 74457 sub-1-1: opened sub1-1-1
17817764 74457 sub-1-1: opened ...
17817764 74457 sub-1-1: opened sub1-1-9

17817764 74484 sub-1-1: closed sub1-1-1
17817764 75737 sub-1-1: closed sub1-1-2
...
17817764 75739 sub-1-1: closed sub1-1-9
17817764 76657 sub-1: CLOSED sub-1-1

Which program starts 'late' varies, sometimes it is sub-1-1, sometimes
sub-1-2, sometimes sub-1-3. Sometimes all three start early, this is
when the low response times are achieved.

On my Desktop-PC and on Windows on the same hardware:
- all sub-programs (sub and sub-sub) start immediately after START
- all sub-programs are done quickly after STOP (since processing time
is less than recording time)


So I would be grateful for any hint on how to make the IPC under
Debian-13 as performant (or better) as under Windows, at best as
perfomant as my ancient desktop PC.

TNX
R'

--- PyGate Linux v1.5.18
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

On 03/07/26 at 18:27, Ralf Fassel wrote:

- When receiving a START signal, data are stored to SSD, and an analysis
program is started. The analysis program in turn starts several
sub-processes, which in turn start several sub-sub-processes.
All sub-sub-processes read the recorded data files in a "tail -f" fashion.

I suspect hardware issue with your IPC, maybe not all cores of the I7
cpu are used since the IPC is fan-less. Shooting in the dark, try a
different scheduler, the kernel 6.12.x comes with three scheduler:
deadline (the default), kyber and bfq. To see the available scheduler
simply run:

# cat /sys/block/sda/queue/scheduler

change "sda" according to your SSD device, to try another scheduler run:

# echo bfq >/sys/block/[TheSSD]/queue/scheduler

If you want to know more about Linux kernel scheduler looks at "linux-source-6.12/Documentation/block/" directory of the kernel sources.

Kind regards,
--
Franco Martelli

--- PyGate Linux v1.5.18
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

On 7/3/26 11:27 AM, Ralf Fassel wrote:

I have tried:
- echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
- no difference
- sudo powerprofilesctl set performance
- no difference
- cpupower-gui: set performance mode, limit min-CPU-Freq to 2500
- no difference
- intel_pstate=disabled at boot
- performance even worse
- intel_pstate=passive at boot, cpu-governor schedutil
- slightly worse performance
- Debian-12, same software
- no difference

You might want to check what cpu power configuration is available in the
bios firmware. Your IPC or motherboard manual should have instructions
how to get into that, during boot, before grub starts.

A lot of 'compact' PCs provide a method of setting high performance vs
quiet or energy saving. I'm of the opinion linux generally obeys that
hardware setting but windows sometimes has ways of overriding it and
using max power anyway. This can give the illusion of windows performing better than linux.

If that's not it then I might investigate whether there is some missing firmware. A starting point might be
sudo dmesg | grep -Ei 'fail|error'
to see whether anything jumps out.

--- PyGate Linux v1.5.18
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

Hello,

I am not knowledgeable about data acquisition but you don't mention it:
would not a real-time kernel be more appropriate for the task than a
standard one?
see linux-image-rt-amd64 package.

--- PyGate Linux v1.5.18
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

* Franco Martelli <martellif67@gmail.com>
| On 03/07/26 at 18:27, Ralf Fassel wrote:
| > - When receiving a START signal, data are stored to SSD, and an analysis
| > program is started. The analysis program in turn starts several
| > sub-processes, which in turn start several sub-sub-processes.
| > All sub-sub-processes read the recorded data files in a "tail -f" fashion.

| I suspect hardware issue with your IPC, maybe not all cores of the I7
| cpu are used since the IPC is fan-less.

I can see the usage of *all* 8 CPUs at full throttle when I run a computing-intensive process like multi-channel online FFT. This does
not degrade after several minutes, so I don't think there is physical
damage to individual CPU cores, or a overheating problem.

| Shooting in the dark, try a different scheduler, the kernel 6.12.x
| comes with three scheduler: deadline (the default), kyber and bfq. To
| see the available scheduler simply run:

| # cat /sys/block/sda/queue/scheduler

| change "sda" according to your SSD device, to try another scheduler run:

| # echo bfq >/sys/block/[TheSSD]/queue/scheduler

I had

desktop% cat /sys/block/sda/queue/scheduler
none [mq-deadline]

ipc% cat /sys/block/sda/queue/scheduler
[none] mq-deadline

(note the different position of the [])

So I tried
# echo mq-deadline >/sys/block/[TheSSD]/queue/scheduler
which then resulted in

ipc% cat /sys/block/sda/queue/scheduler
none [mq-deadline]

but did not change the timing of the process.

Same result for 'bfq' and 'kyber'.
# echo bfq >/sys/block/[TheSSD]/queue/scheduler
ipc% cat /sys/block/sda/queue/scheduler
none mq-deadline [bfq]

Still clustering at 1s intervalls 2/3/4s.

The crucial point is always the delayed start of one or two of the subprocesses, which sometimes do not start to run until the recording is stopped (as said, it varies which of the processes is starting late).

I confirmed that it is the subprocess itself which starts late (issue
timestamp right at the start), and not some processing in the subprocess
itself - the various steps in the subprocess happen very quickly once it
is running.

| If you want to know more about Linux kernel scheduler looks at
| "linux-source-6.12/Documentation/block/" directory of the kernel
| sources.

Thanks for the pointer, with that it feels somewhat less
"monkey-see-monkey-do" :-)

R'

--- PyGate Linux v1.5.18
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

* nwe <nwe@gitcoding.net>
| On 7/3/26 11:27 AM, Ralf Fassel wrote:

| > [bad system performance]
| You might want to check what cpu power configuration is available in
| the bios firmware. Your IPC or motherboard manual should have
| instructions how to get into that, during boot, before grub starts.

| A lot of 'compact' PCs provide a method of setting high performance vs
| quiet or energy saving. I'm of the opinion linux generally obeys that
| hardware setting but windows sometimes has ways of overriding it and
| using max power anyway. This can give the illusion of windows
| performing better than linux.

I found an entry
BIOS - Advcanced - CPU Power Management Control
- *disable* CPU C states
(which says something like "allow CPU to go to C states if not used 100%")

So I tried 'disabled' (it was enabled).

I then see the CPUs not throttling down to 'idle' so easily by 'watch'ing
/sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq

Other than that there is "Turbo Mode ON/OFF" (is ON), and
TDP Configurations, where I changed "Power Limit 1" and ...2 from 15000
to 0 (= no custom override).

But all of these changes make no difference for the processing times -

still the start of one or two of the sub-processes is delayed which then results in the large overall response time.

| If that's not it then I might investigate whether there is some
| missing firmware. A starting point might be
| sudo dmesg | grep -Ei 'fail|error'
| to see whether anything jumps out.

Some ACPI unresolved symbols errors "\_SB.UBTC.RUCC AE_NOT_FOUND", but
other than that nothing.

TNX
R'

--- PyGate Linux v1.5.18
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

* didier gaumet <didier.gaumet@gmail.com>
| I am not knowledgeable about data acquisition but you don't mention
| it: would not a real-time kernel be more appropriate for the task than
| a standard one?
| see linux-image-rt-amd64 package.

I installed the rt-kernel, but that made no difference at all.
As I understand it, I would need to recompile my programs to actually
request the realtime features - which I did not do.

Also note that the data aquisition itself is not the problem - I can see
the files on disk grow as expected as the data arrive with 2.8MB/s.
Plus, that data rate should be no problem at all for the SSD.

It is the delayed start of one or two of the subprocesses which ruins
the overall performance.

On my desktop, all subprocesses are started immediately and reliably
100% (same Debian kernel, same binaries of my programs).

Thanks anyway for all responses!
R'

--- PyGate Linux v1.5.18
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

Ralf Fassel (HE12026-07-03):

an IPC (fanless Industrial PC

Stop right there. Nobody cares that your PC is industrial or fanless.

On the other hand, we care that IPC means ?interprocess communication?. Especially in the context of performance issue with concurrent
processes.

Using IPC all over the place will only confuses the people who have the knowledge to help you.

- When receiving a START signal, data are stored to SSD, and an analysis
program is started. The analysis program in turn starts several
sub-processes, which in turn start several sub-sub-processes.
All sub-sub-processes read the recorded data files in a "tail -f" fashion.

That looks inefficiently designed.

local_time[ms] what
17817764 68904 MAIN "START" sub-1 started

The number of digits in your milliseconds is quite strange.

So I would be grateful for any hint on how to make the IPC under
Debian-13 as performant (or better) as under Windows, at best as
perfomant as my ancient desktop PC.

Run your whole system under strace -tttT and see exactly what system
call is taking time in the process that arrives late.

Regards,

--
Nicolas George

--- PyGate Linux v1.5.18
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

On 06/07/26 at 18:21, Ralf Fassel wrote:

Still clustering at 1s intervalls 2/3/4s.

The crucial point is always the delayed start of one or two of the subprocesses, which sometimes do not start to run until the recording is stopped (as said, it varies which of the processes is starting late).

I confirmed that it is the subprocess itself which starts late (issue timestamp right at the start), and not some processing in the subprocess itself - the various steps in the subprocess happen very quickly once it
is running.

I think you need a kernel hacker rather than a Debian-user to suggest to
you some boot parameters to try for that hardware, follows two very
stupid questions:

- is the intel-microcode package installed?
- when Didier recommended the linux-image-rt-amd64 kernel after
installed, have you rebooted the system and verified with the "uname -a" command that the fanless PC is running the real-time kernel?

Kind regards,
--
Franco Martelli

--- PyGate Linux v1.5.18
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

Le 06/07/2026 … 18:56, Ralf Fassel a ‚critÿ:

I installed the rt-kernel, but that made no difference at all.

As Franco Martelli asked: are you sure you booted the real-time kernel
and not the regular one?

Installation of linux-image-rt-amd64 (real-time kernel) package does not remove linux-image-amd64 package (regular kernel).
So after that installation, you have two kernels side-by-side, the
regular one being the default for boot.

Then if you did not remove or purge (apt remove or apt purge followed by
the desired package name), or did not specifically chose the real-time
kernel in the grub menu, chances are that you booted the regular one,
hence the same results.

As I understand it, I would need to recompile my programs to actually
request the realtime features - which I did not do.

[...]
I am really far from knowing much on the subject, but are you not
confusing real-time operating and multithreading?

I would think that real-time on Linux is done by booting a real-time
Linux image,

while multithreading (on Linux, I don't know if Windows has some sorts
of magic tricks) is done either explicitly by writing programs with
detailed multithreading instructions, or implicitly writing generic
global instructions for implicit multithreading.
And only then you have to recompile your code (if it's your code,
because nowadays I suppose a data acquisition program, either commercial
or communautary one, is systematically multithread).

Please don't take my word on all this without verifying by asking to
more competent people.

--- PyGate Linux v1.5.18
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

Problem solved, see my response to Nicolas George elsethread.

For the record, answers to your questions nevertheless.

* Franco Martelli <martellif67@gmail.com>
| - is the intel-microcode package installed?

Yes.

| - when Didier recommended the linux-image-rt-amd64 kernel after
| installed, have you rebooted the system and verified with the "uname
| -a" command that the fanless PC is running the real-time kernel?

Yes, I selected the -rt kernel in GRUB.

Thanks for your help!
R'

--- PyGate Linux v1.5.18
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

Nicolas, if you ever visit G”ttingen in Germany, I owe you a drink of
your choice.

* Nicolas George <george@nsup.org>
| Ralf Fassel (HE12026-07-03):
| > an IPC (fanless Industrial PC

| Stop right there. Nobody cares that your PC is industrial or fanless.

| On the other hand, we care that IPC means ?interprocess communication?.
| Especially in the context of performance issue with concurrent
| processes.

| Using IPC all over the place will only confuses the people who have the
| knowledge to help you.

Point taken.

| > local_time[ms] what
| > 17817764 68904 MAIN "START" sub-1 started

| The number of digits in your milliseconds is quite strange.

For better readability I separated the non-changing part of the
timestamps (milliseconds from epoch) with a space from the changing
part.

| Run your whole system under strace -tttT and see exactly what system
| call is taking time in the process that arrives late.

Spot on, thanks for suggesting this. I would not have dared to run
strace on the subprocess due to the huge amount of output.
But the -T actually helps to see where the process stops and wastes its
time.

In my case it is a hardware license dongle which obviously has problems
when many processes simultaneously try to obtain a license from it
(needless to say: same dongle no problem on Windows). It guess this
also explains the 1sec-clustering which I see in the timings: seems like
there is some internal 1-sec timeout in the library code for the dongle.
If it was due to the PC-Hardware, I would have expected to see even distribution of answering times, but not that clustering.

If I change to a file-based license, I get stable answer times of
0.3secs (yes, it's that fast, and I would have expected exactly this
from this CPU).

Thanks to all who took their time to look into this and their
suggestions.

R'

--- PyGate Linux v1.5.18
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)