• Re: Debian-13 bad performance on IPC

    From Ralf Fassel@3:633/10 to All on Monday, July 06, 2026 18:50:01
    * Franco Martelli <martellif67@gmail.com>
    | On 03/07/26 at 18:27, Ralf Fassel wrote:
    | > - When receiving a START signal, data are stored to SSD, and an analysis
    | > program is started. The analysis program in turn starts several
    | > sub-processes, which in turn start several sub-sub-processes.
    | > All sub-sub-processes read the recorded data files in a "tail -f" fashion.

    | I suspect hardware issue with your IPC, maybe not all cores of the I7
    | cpu are used since the IPC is fan-less.

    I can see the usage of *all* 8 CPUs at full throttle when I run a computing-intensive process like multi-channel online FFT. This does
    not degrade after several minutes, so I don't think there is physical
    damage to individual CPU cores, or a overheating problem.

    | Shooting in the dark, try a different scheduler, the kernel 6.12.x
    | comes with three scheduler: deadline (the default), kyber and bfq. To
    | see the available scheduler simply run:

    | # cat /sys/block/sda/queue/scheduler

    | change "sda" according to your SSD device, to try another scheduler run:

    | # echo bfq >/sys/block/[TheSSD]/queue/scheduler

    I had

    desktop% cat /sys/block/sda/queue/scheduler
    none [mq-deadline]

    ipc% cat /sys/block/sda/queue/scheduler
    [none] mq-deadline

    (note the different position of the [])

    So I tried
    # echo mq-deadline >/sys/block/[TheSSD]/queue/scheduler
    which then resulted in

    ipc% cat /sys/block/sda/queue/scheduler
    none [mq-deadline]

    but did not change the timing of the process.

    Same result for 'bfq' and 'kyber'.
    # echo bfq >/sys/block/[TheSSD]/queue/scheduler
    ipc% cat /sys/block/sda/queue/scheduler
    none mq-deadline [bfq]

    Still clustering at 1s intervalls 2/3/4s.

    The crucial point is always the delayed start of one or two of the subprocesses, which sometimes do not start to run until the recording is stopped (as said, it varies which of the processes is starting late).

    I confirmed that it is the subprocess itself which starts late (issue
    timestamp right at the start), and not some processing in the subprocess
    itself - the various steps in the subprocess happen very quickly once it
    is running.

    | If you want to know more about Linux kernel scheduler looks at
    | "linux-source-6.12/Documentation/block/" directory of the kernel
    | sources.

    Thanks for the pointer, with that it feels somewhat less
    "monkey-see-monkey-do" :-)

    R'

    --- PyGate Linux v1.5.18
    * Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)
  • From Ralf Fassel@3:633/10 to All on Monday, July 06, 2026 19:10:01
    * nwe <nwe@gitcoding.net>
    | On 7/3/26 11:27 AM, Ralf Fassel wrote:

    | > [bad system performance]
    | You might want to check what cpu power configuration is available in
    | the bios firmware. Your IPC or motherboard manual should have
    | instructions how to get into that, during boot, before grub starts.

    | A lot of 'compact' PCs provide a method of setting high performance vs
    | quiet or energy saving. I'm of the opinion linux generally obeys that
    | hardware setting but windows sometimes has ways of overriding it and
    | using max power anyway. This can give the illusion of windows
    | performing better than linux.

    I found an entry
    BIOS - Advcanced - CPU Power Management Control
    - *disable* CPU C states
    (which says something like "allow CPU to go to C states if not used 100%")

    So I tried 'disabled' (it was enabled).

    I then see the CPUs not throttling down to 'idle' so easily by 'watch'ing
    /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq

    Other than that there is "Turbo Mode ON/OFF" (is ON), and
    TDP Configurations, where I changed "Power Limit 1" and ...2 from 15000
    to 0 (= no custom override).


    But all of these changes make no difference for the processing times -
    still the start of one or two of the sub-processes is delayed which then results in the large overall response time.

    | If that's not it then I might investigate whether there is some
    | missing firmware. A starting point might be
    | sudo dmesg | grep -Ei 'fail|error'
    | to see whether anything jumps out.

    Some ACPI unresolved symbols errors "\_SB.UBTC.RUCC AE_NOT_FOUND", but
    other than that nothing.

    TNX
    R'

    --- PyGate Linux v1.5.18
    * Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)
  • From Ralf Fassel@3:633/10 to All on Monday, July 06, 2026 19:20:01
    * didier gaumet <didier.gaumet@gmail.com>
    | I am not knowledgeable about data acquisition but you don't mention
    | it: would not a real-time kernel be more appropriate for the task than
    | a standard one?
    | see linux-image-rt-amd64 package.

    I installed the rt-kernel, but that made no difference at all.
    As I understand it, I would need to recompile my programs to actually
    request the realtime features - which I did not do.

    Also note that the data aquisition itself is not the problem - I can see
    the files on disk grow as expected as the data arrive with 2.8MB/s.
    Plus, that data rate should be no problem at all for the SSD.

    It is the delayed start of one or two of the subprocesses which ruins
    the overall performance.

    On my desktop, all subprocesses are started immediately and reliably
    100% (same Debian kernel, same binaries of my programs).

    Thanks anyway for all responses!
    R'

    --- PyGate Linux v1.5.18
    * Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)
  • From Nicolas George@3:633/10 to All on Monday, July 06, 2026 19:30:01
    Ralf Fassel (HE12026-07-03):
    an IPC (fanless Industrial PC

    Stop right there. Nobody cares that your PC is industrial or fanless.

    On the other hand, we care that IPC means ?interprocess communication?. Especially in the context of performance issue with concurrent
    processes.

    Using IPC all over the place will only confuses the people who have the knowledge to help you.

    - When receiving a START signal, data are stored to SSD, and an analysis
    program is started. The analysis program in turn starts several
    sub-processes, which in turn start several sub-sub-processes.
    All sub-sub-processes read the recorded data files in a "tail -f" fashion.

    That looks inefficiently designed.

    local_time[ms] what
    17817764 68904 MAIN "START" sub-1 started

    The number of digits in your milliseconds is quite strange.

    So I would be grateful for any hint on how to make the IPC under
    Debian-13 as performant (or better) as under Windows, at best as
    perfomant as my ancient desktop PC.

    Run your whole system under strace -tttT and see exactly what system
    call is taking time in the process that arrives late.

    Regards,

    --
    Nicolas George

    --- PyGate Linux v1.5.18
    * Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)
  • From Franco Martelli@3:633/10 to All on Monday, July 06, 2026 21:00:01
    On 06/07/26 at 18:21, Ralf Fassel wrote:
    Still clustering at 1s intervalls 2/3/4s.

    The crucial point is always the delayed start of one or two of the subprocesses, which sometimes do not start to run until the recording is stopped (as said, it varies which of the processes is starting late).

    I confirmed that it is the subprocess itself which starts late (issue timestamp right at the start), and not some processing in the subprocess itself - the various steps in the subprocess happen very quickly once it
    is running.

    I think you need a kernel hacker rather than a Debian-user to suggest to
    you some boot parameters to try for that hardware, follows two very
    stupid questions:

    - is the intel-microcode package installed?
    - when Didier recommended the linux-image-rt-amd64 kernel after
    installed, have you rebooted the system and verified with the "uname -a" command that the fanless PC is running the real-time kernel?

    Kind regards,
    --
    Franco Martelli

    --- PyGate Linux v1.5.18
    * Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)
  • From didier gaumet@3:633/10 to All on Monday, July 06, 2026 21:50:01
    Le 06/07/2026 … 18:56, Ralf Fassel a ‚crit˙:

    I installed the rt-kernel, but that made no difference at all.

    As Franco Martelli asked: are you sure you booted the real-time kernel
    and not the regular one?

    Installation of linux-image-rt-amd64 (real-time kernel) package does not remove linux-image-amd64 package (regular kernel).
    So after that installation, you have two kernels side-by-side, the
    regular one being the default for boot.

    Then if you did not remove or purge (apt remove or apt purge followed by
    the desired package name), or did not specifically chose the real-time
    kernel in the grub menu, chances are that you booted the regular one,
    hence the same results.

    As I understand it, I would need to recompile my programs to actually
    request the realtime features - which I did not do.
    [...]
    I am really far from knowing much on the subject, but are you not
    confusing real-time operating and multithreading?

    I would think that real-time on Linux is done by booting a real-time
    Linux image,

    while multithreading (on Linux, I don't know if Windows has some sorts
    of magic tricks) is done either explicitly by writing programs with
    detailed multithreading instructions, or implicitly writing generic
    global instructions for implicit multithreading.
    And only then you have to recompile your code (if it's your code,
    because nowadays I suppose a data acquisition program, either commercial
    or communautary one, is systematically multithread).

    Please don't take my word on all this without verifying by asking to
    more competent people.

    --- PyGate Linux v1.5.18
    * Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)
  • From Ralf Fassel@3:633/10 to All on Tuesday, July 07, 2026 16:00:01
    Problem solved, see my response to Nicolas George elsethread.

    For the record, answers to your questions nevertheless.

    * Franco Martelli <martellif67@gmail.com>
    | - is the intel-microcode package installed?

    Yes.

    | - when Didier recommended the linux-image-rt-amd64 kernel after
    | installed, have you rebooted the system and verified with the "uname
    | -a" command that the fanless PC is running the real-time kernel?

    Yes, I selected the -rt kernel in GRUB.

    Thanks for your help!
    R'

    --- PyGate Linux v1.5.18
    * Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)
  • From Ralf Fassel@3:633/10 to All on Tuesday, July 07, 2026 16:00:01
    Nicolas, if you ever visit G”ttingen in Germany, I owe you a drink of
    your choice.

    * Nicolas George <george@nsup.org>
    | Ralf Fassel (HE12026-07-03):
    | > an IPC (fanless Industrial PC

    | Stop right there. Nobody cares that your PC is industrial or fanless.

    | On the other hand, we care that IPC means ?interprocess communication?.
    | Especially in the context of performance issue with concurrent
    | processes.

    | Using IPC all over the place will only confuses the people who have the
    | knowledge to help you.

    Point taken.

    | > local_time[ms] what
    | > 17817764 68904 MAIN "START" sub-1 started

    | The number of digits in your milliseconds is quite strange.

    For better readability I separated the non-changing part of the
    timestamps (milliseconds from epoch) with a space from the changing
    part.

    | Run your whole system under strace -tttT and see exactly what system
    | call is taking time in the process that arrives late.

    Spot on, thanks for suggesting this. I would not have dared to run
    strace on the subprocess due to the huge amount of output.
    But the -T actually helps to see where the process stops and wastes its
    time.

    In my case it is a hardware license dongle which obviously has problems
    when many processes simultaneously try to obtain a license from it
    (needless to say: same dongle no problem on Windows). It guess this
    also explains the 1sec-clustering which I see in the timings: seems like
    there is some internal 1-sec timeout in the library code for the dongle.
    If it was due to the PC-Hardware, I would have expected to see even distribution of answering times, but not that clustering.

    If I change to a file-based license, I get stable answer times of
    0.3secs (yes, it's that fast, and I would have expected exactly this
    from this CPU).

    Thanks to all who took their time to look into this and their
    suggestions.

    R'

    --- PyGate Linux v1.5.18
    * Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)