Anyone played with any of the "Open" AI's on local hardware?
Successes?˙ Horror stories?
Anyone played with any of the "Open" AI's on local hardware?
Successes? Horror stories?
"Don Y" <blockedofcourse@foo.invalid> wrote in message news:1147o34$3880b$1@dont-email.me...
Anyone played with any of the "Open" AI's on local hardware?
Successes? Horror stories?
On the subject of AI here's an article: https://www.theregister.com/ai-and-ml/2026/07/28/college-prof-hides-prompt-to-catch-ai-cheaters-finds-human-nature-is-pretty-much-as-we-thought/5279864
I have tried AI on local hardware and got pretty much what I expected.
I may do so again.
AI sometime seems to overlook subtle clues which are obvious to most humans. For example:
Comment on the book "Hanging by a thread" by Dan Gling.
To find out whether Google's circuit design skills are improving I just tried this:I assume the AI will perform the same "remotely hosted" as it would locally. So, the issue isn't how performative it is but, rather, how well it
Design a low distortion sinewave oscillator and include an elephant.
Anyone played with any of the "Open" AI's on local hardware?
Successes? Horror stories?
On 7/28/2026 10:31 AM, Edward Rawde wrote:
"Don Y" <blockedofcourse@foo.invalid> wrote in message news:1147o34$3880b$1@dont-email.me...
Anyone played with any of the "Open" AI's on local hardware?
Successes? Horror stories?
On the subject of AI here's an article:
https://www.theregister.com/ai-and-ml/2026/07/28/college-prof-hides-prompt-to-catch-ai-cheaters-finds-human-nature-is-pretty-much-as-we-thought/5279864
I have tried AI on local hardware and got pretty much what I expected.
I may do so again.
Which model and what sort of hardware?
I'll be hosting an off-site in another month or so and figured
demonstrating the sorts of performance one can achieve on various
types of hardware would be an interesting topic. (I tend to have
a lot more hardware than most people)
There are general assessments available but nothing (that I have
found) has provided a quantitative description of the types of
performance and hardware impacts on it.
E.g., when does thrashing take a toll, MIPS vs memory, GPU vs
all of the above, etc.
[Note we're not interested in actual performance but RELATIVE
performance -- the answers should be the same, the "wait" being
the only difference. (And, we should be able to prove this!)]
AI sometime seems to overlook subtle clues which are obvious to most humans. >> For example:
Comment on the book "Hanging by a thread" by Dan Gling.
"History of a Man's Life" by I.P.Standing
To find out whether Google's circuit design skills are improving I just tried this:I assume the AI will perform the same "remotely hosted" as it would locally. So, the issue isn't how performative it is but, rather, how well it
Design a low distortion sinewave oscillator and include an elephant.
performs on local hardware (cut the cord)
"Don Y" <blockedofcourse@foo.invalid> wrote in message news:114b7ba$d9uc$1@dont-email.me...
On 7/28/2026 10:31 AM, Edward Rawde wrote:
"Don Y" <blockedofcourse@foo.invalid> wrote in message news:1147o34$3880b$1@dont-email.me...
Anyone played with any of the "Open" AI's on local hardware?
Successes? Horror stories?
On the subject of AI here's an article:
https://www.theregister.com/ai-and-ml/2026/07/28/college-prof-hides-prompt-to-catch-ai-cheaters-finds-human-nature-is-pretty-much-as-we-thought/5279864
I have tried AI on local hardware and got pretty much what I expected.
I may do so again.
Which model and what sort of hardware?
https://easydiffusion.github.io/
Core-i5 with 32GB RAM and on board GPU. Windows 10.
So very slow but reasonable images after a few hours.
I'll be hosting an off-site in another month or so and figured
demonstrating the sorts of performance one can achieve on various
types of hardware would be an interesting topic. (I tend to have
a lot more hardware than most people)
There are general assessments available but nothing (that I have
found) has provided a quantitative description of the types of
performance and hardware impacts on it.
E.g., when does thrashing take a toll, MIPS vs memory, GPU vs
all of the above, etc.
[Note we're not interested in actual performance but RELATIVE
performance -- the answers should be the same, the "wait" being
the only difference. (And, we should be able to prove this!)]
AI sometime seems to overlook subtle clues which are obvious to most humans.
For example:
Comment on the book "Hanging by a thread" by Dan Gling.
"History of a Man's Life" by I.P.Standing
It also doesn't get the even more obvious
"How to get rich" by Robin Banks.
On Mon, 27 Jul 2026 06:56:53 -0700, Don Y
<blockedofcourse@foo.invalid> wrote:
Anyone played with any of the "Open" AI's on local hardware?
Successes? Horror stories?
Is local AI very useful?
I thought AI needed petabytes of training.
I guess you could ask it to optimize a circuit, or something local.
Maybe do some math.
On 7/28/2026 4:16 PM, Edward Rawde wrote:
"Don Y" <blockedofcourse@foo.invalid> wrote in message news:114b7ba$d9uc$1@dont-email.me...
On 7/28/2026 10:31 AM, Edward Rawde wrote:
"Don Y" <blockedofcourse@foo.invalid> wrote in message news:1147o34$3880b$1@dont-email.me...
Anyone played with any of the "Open" AI's on local hardware?
Successes? Horror stories?
On the subject of AI here's an article:
https://www.theregister.com/ai-and-ml/2026/07/28/college-prof-hides-prompt-to-catch-ai-cheaters-finds-human-nature-is-pretty-much-as-we-thought/5279864
I have tried AI on local hardware and got pretty much what I expected. >>>> I may do so again.
Which model and what sort of hardware?
https://easydiffusion.github.io/
Core-i5 with 32GB RAM and on board GPU. Windows 10.
So very slow but reasonable images after a few hours.
Does it use the GPU for reasoning or image rendering?
I'll be hosting an off-site in another month or so and figured
demonstrating the sorts of performance one can achieve on various
types of hardware would be an interesting topic. (I tend to have
a lot more hardware than most people)
There are general assessments available but nothing (that I have
found) has provided a quantitative description of the types of
performance and hardware impacts on it.
E.g., when does thrashing take a toll, MIPS vs memory, GPU vs
all of the above, etc.
[Note we're not interested in actual performance but RELATIVE
performance -- the answers should be the same, the "wait" being
the only difference. (And, we should be able to prove this!)]
AI sometime seems to overlook subtle clues which are obvious to most humans.
For example:
Comment on the book "Hanging by a thread" by Dan Gling.
"History of a Man's Life" by I.P.Standing
It also doesn't get the even more obvious
"How to get rich" by Robin Banks.
An appreciation for humor is, supposedly, a sign of higher intelligence; being able to appreciate more subtle interactions between ideas.
<https://verybigbrain.com/psychology-thinking/the-hidden-link-between-humor-and-intelligence-why-smart-people-love-to-laugh/>
We already know AI's aren't really "intelligent" -- beyond the ability
to mindlessly find and apply patterns; there's little evidence that they "understand" what they are stating (witness how they can't EXPLAIN their reasoning)
Much like the test applied to "Number 5"...
"john larkin" <jl@htigct.com> wrote in message news:qkai6lt3kt0ipsp533rvpon63den2hbk8e@4ax.com...
On Mon, 27 Jul 2026 06:56:53 -0700, Don Y
<blockedofcourse@foo.invalid> wrote:
Anyone played with any of the "Open" AI's on local hardware?
Successes? Horror stories?
Is local AI very useful?
I've not so far used it for anything other than images from a text description.
I thought AI needed petabytes of training.
I guess you could ask it to optimize a circuit, or something local.
AI (which has been fed with pentabytes of text)
can be reasonable with subjects which can be done in text.
So if you ask for a 2000 word essay on the industrial revolution you'll get an answer
which the average student will now paste into Word, print it and hand it in.
It will also solve, or at least assist with, most mathematical problems which >students up to degree level encounter for homework.
I just tested a grade 12 academic problem I did for a student back
in 2015 and it was fine if a bit long winded.
But this leads to recent cases of students pasting the problem into Google and >copying the answer without understanding it. Not good if you want to pass an exam.
I've yet to see AI generate a non trivial circuit diagram (schematic) in response to
only text input.
Even if it makes an attempt, anything I've seen so far will make anyone here laugh.
It does have some uses though. I recently wanted a few hundred line pwl file for LTSpice.
So I typed a description of what I wanted into Google and got working python code
in seconds. Since the file it produced was exactly what I wanted I didn't even bother
to understand how the code worked.
That reminds me of another difference between work and school. At school you're
told that it's not the answer which is wanted but how the answer was arrived at.
At work you're told to find the answer and no-one cares how you do it, just find
the answer.
So I can see AI taking over the legal profession soon but not the engineering >design profession where technical drawings are required.
Maybe do some math.
Anyone played with any of the "Open" AI's on local hardware?
Successes? Horror stories?
Don Y <blockedofcourse@foo.invalid> wrote:
Anyone played with any of the "Open" AI's on local hardware?
Successes? Horror stories?
I've been doing some messing around (text processing, not LLMs, on Ubuntu 24.04). Takeaways:
1. The model is one thing, but it often exists within a stack of Python/C++/whatever to run it.
That can be quite brittle - a huge pile of
Python dependencies, maybe some of them don't work on your machine and you have to dig into why (Python versions, OS updates, projects been abandoned, etc).
If you want to switch to a different model to do the same job, now
you have to switch to their stack of dependencies (ie effectively rewrite your code, rather than just swapping out one model file for another).
2. If you have a GPU it had better be NVIDIA. AMD and Intel GPUs
effectively don't exist as far as these stacks are concerned. I have a perfectly good modern AMD GPU and all the models run on the CPU.
That's my 2 cents, no doubt YMMV.Has it been worth the effort (besides as a curiosity)?
"Don Y" <blockedofcourse@foo.invalid> wrote in message news:114bg26$fotv$1@dont-email.me...A cursory read of the stable diffusion site seems to suggest that the GPU
On 7/28/2026 4:16 PM, Edward Rawde wrote:
"Don Y" <blockedofcourse@foo.invalid> wrote in message news:114b7ba$d9uc$1@dont-email.me...
On 7/28/2026 10:31 AM, Edward Rawde wrote:
"Don Y" <blockedofcourse@foo.invalid> wrote in message news:1147o34$3880b$1@dont-email.me...
Anyone played with any of the "Open" AI's on local hardware?
Successes? Horror stories?
On the subject of AI here's an article:
https://www.theregister.com/ai-and-ml/2026/07/28/college-prof-hides-prompt-to-catch-ai-cheaters-finds-human-nature-is-pretty-much-as-we-thought/5279864
I have tried AI on local hardware and got pretty much what I expected. >>>>> I may do so again.
Which model and what sort of hardware?
https://easydiffusion.github.io/
Core-i5 with 32GB RAM and on board GPU. Windows 10.
So very slow but reasonable images after a few hours.
Does it use the GPU for reasoning or image rendering?
I don't know the answer to that.
You'd have to ask someone associated with the project.
On 7/28/2026 5:42 PM, Edward Rawde wrote:
"Don Y" <blockedofcourse@foo.invalid> wrote in message news:114bg26$fotv$1@dont-email.me...A cursory read of the stable diffusion site seems to suggest that the GPU
On 7/28/2026 4:16 PM, Edward Rawde wrote:
"Don Y" <blockedofcourse@foo.invalid> wrote in message news:114b7ba$d9uc$1@dont-email.me...
On 7/28/2026 10:31 AM, Edward Rawde wrote:
"Don Y" <blockedofcourse@foo.invalid> wrote in message news:1147o34$3880b$1@dont-email.me...
Anyone played with any of the "Open" AI's on local hardware?
Successes? Horror stories?
On the subject of AI here's an article:
https://www.theregister.com/ai-and-ml/2026/07/28/college-prof-hides-prompt-to-catch-ai-cheaters-finds-human-nature-is-pretty-much-as-we-thought/5279864
I have tried AI on local hardware and got pretty much what I expected. >>>>>> I may do so again.
Which model and what sort of hardware?
https://easydiffusion.github.io/
Core-i5 with 32GB RAM and on board GPU. Windows 10.
So very slow but reasonable images after a few hours.
Does it use the GPU for reasoning or image rendering?
I don't know the answer to that.
You'd have to ask someone associated with the project.
is likely used as a reasoning accelerator -- I can't imagine you'd need
10G of VRAM just to render images trained from 512x512's. (?)
Does it rely on a network connection for *anything*?
with locally sourced artwork?
A cursory read of the stable diffusion site seems to suggest that the GPU
is likely used as a reasoning accelerator -- I can't imagine you'd need
10G of VRAM just to render images trained from 512x512's. (?)
Does it rely on a network connection for *anything*?
I don't think so, but I didn't test it on a non networked machine.
Or, does it startYour interest is for entertainment/amusement? Curiosity?
with locally sourced artwork?
Some of the tests I did started with locally sourced art.
On 7/29/2026 7:32 AM, Edward Rawde wrote:
A cursory read of the stable diffusion site seems to suggest that the GPU >>> is likely used as a reasoning accelerator -- I can't imagine you'd need
10G of VRAM just to render images trained from 512x512's. (?)
Does it rely on a network connection for *anything*?
I don't think so, but I didn't test it on a non networked machine.
Some of the sample artwork seems like it would be hard to imagine
it coming out of thin air (or, a set of weights).
Or, does it startYour interest is for entertainment/amusement? Curiosity?
with locally sourced artwork?
Some of the tests I did started with locally sourced art.
(other posts suggest you've done more than a little "dabbling"
with AI tools).
Most of my colleagues have had unsatisfactory results,
despite all the "hype" that it can do X, Y and Z. So,
I'm trying to come up with an example that can be appreciated
(focusing on the "costs" of the technology) even if not immediately
helpful.
[I still think more focused technologies (e.g., symbolic execution)
produce more bang for the resource buck; likely because they have
a focused strategy instead of "hoping for the best" from a more
general purpose tool]
[[I saw a prediction that we'll see most of the AIs fall by the wayside within two years due to disappointing performance -- once the hype is
gone]]
Some of the tests I did started with locally sourced art.Your interest is for entertainment/amusement? Curiosity?
Mostly curiosity and to see what can be done locally without being
dependent on a computer under someone else's control. i.e. "the cloud".
(other posts suggest you've done more than a little "dabbling"
with AI tools).
I think it's a good idea to use the best tool for the job.
That may simply mean saving time by getting some quick python code
which does the job rather than spending time writing it myself.
However I have not used any local AI which can generate code.
Most of my colleagues have had unsatisfactory results,
despite all the "hype" that it can do X, Y and Z. So,
I'm trying to come up with an example that can be appreciated
(focusing on the "costs" of the technology) even if not immediately
helpful.
[I still think more focused technologies (e.g., symbolic execution)
produce more bang for the resource buck; likely because they have
a focused strategy instead of "hoping for the best" from a more
general purpose tool]
[[I saw a prediction that we'll see most of the AIs fall by the wayside
within two years due to disappointing performance -- once the hype is
gone]]
It will be interesting to see whether AGI gets anywhere.
There are clearly those who think there's more to the brain than computation (Penrose) and those who don't.I think humans have the capacity for "insights" as some dendrite
On 7/29/2026 8:28 AM, Edward Rawde wrote:
Some of the tests I did started with locally sourced art.Your interest is for entertainment/amusement? Curiosity?
Mostly curiosity and to see what can be done locally without being
dependent on a computer under someone else's control. i.e. "the cloud".
Or, leaking any of your "IP" to train someone else's tool.
(other posts suggest you've done more than a little "dabbling"
with AI tools).
I think it's a good idea to use the best tool for the job.
That may simply mean saving time by getting some quick python code
which does the job rather than spending time writing it myself.
But, you have to have confidence in the code; that it is
actually written to do what you intend (and not some hallucination)
However I have not used any local AI which can generate code.
Concensus from my colleagues is that this likely only makes
sense for folks who aren't particularly "good" at writing code
(it avoids the errors that they would otherwise likely make).
"Experienced" developers claim it's like working with a child
that has to be told everything, explicitly.
[And, reports of *quality* suggest almost twice as many bugs
as "human written" code. <frown>]
Most of my colleagues have had unsatisfactory results,
despite all the "hype" that it can do X, Y and Z. So,
I'm trying to come up with an example that can be appreciated
(focusing on the "costs" of the technology) even if not immediately
helpful.
[I still think more focused technologies (e.g., symbolic execution)
produce more bang for the resource buck; likely because they have
a focused strategy instead of "hoping for the best" from a more
general purpose tool]
[[I saw a prediction that we'll see most of the AIs fall by the wayside
within two years due to disappointing performance -- once the hype is
gone]]
It will be interesting to see whether AGI gets anywhere.
I suspect we will have a "wait-a-minute" event, before that
point and folks will rethink the whole AI issue.
I think we will see special purpose agents being deployed -- especially
in more "menial" jobs (there really is no reason that "receptionists"
and "front office staff" can't be replaced *now* at doctor offices,
etc.).
Likewise, the folks at the counter at fast food places are
likely history -- they were already hanging on by a thread as
a kiosk could do their jobs (Costco has already gone this route).
Programming frameworks and "wizards" will get smarter -- saving
keystrokes (for folks who learn how to coax them to perform as
intended).
There will likely be some code coverage and testing tools that
emerge.
Component selection, placement and routing will benefit from
smarter tools -- likely improving DfM as the tool can be aware
of other steps in the process and purchasing patterns ("Why
are we purchasing two different parts that are essentially
performing similar roles?")
But, China will own these markets. Their power costs are negligible
and they've already got AIs on a par with ours (which are already
capable of doing these things).
Domestic AIs will morph into marketing tools -- that analyze
various aspects of our observed behaviours to pitch more goods
to us (or tweek the price offered to exploit their expectations
of what they can get from us for a purchase).
There are clearly those who think there's more to the brain than computation >> (Penrose) and those who don't.I think humans have the capacity for "insights" as some dendrite
happens past another that makes an "atypical" association leading
to an unexpected revelation. I'm not sure how much of that is
learned vs. cultural -- you see a lot of "lazy thinkers" (doing
just the minimum that is needed to "solve" a problem)
I think you might be able to emulate that ability in an
AI by deliberately injecting "randomness" in the reasoning
process -- forcing it to evaluate "exceptions" and deliberately
rule them out (or in).
"Don Y" <blockedofcourse@foo.invalid> wrote in message news:114dm71$15bcq$1@dont-email.me...
On 7/29/2026 8:28 AM, Edward Rawde wrote:
Some of the tests I did started with locally sourced art.Your interest is for entertainment/amusement? Curiosity?
Mostly curiosity and to see what can be done locally without being
dependent on a computer under someone else's control. i.e. "the cloud".
Or, leaking any of your "IP" to train someone else's tool.
Yes a daily look at my firewall inbound blocking shows that the
Internet has become a swarm of "let's grab everything we can" bots.
However I have not used any local AI which can generate code.
Concensus from my colleagues is that this likely only makes
sense for folks who aren't particularly "good" at writing code
(it avoids the errors that they would otherwise likely make).
"Experienced" developers claim it's like working with a child
that has to be told everything, explicitly.
[And, reports of *quality* suggest almost twice as many bugs
as "human written" code. <frown>]
I haven't used it for anything critical.
It can sometimes come up with ideas that you may want to
use in your own code.
It will be interesting to see whether AGI gets anywhere.
I suspect we will have a "wait-a-minute" event, before that
point and folks will rethink the whole AI issue.
I think we will see special purpose agents being deployed -- especially
in more "menial" jobs (there really is no reason that "receptionists"
and "front office staff" can't be replaced *now* at doctor offices,
etc.).
Likewise, the folks at the counter at fast food places are
likely history -- they were already hanging on by a thread as
a kiosk could do their jobs (Costco has already gone this route).
Programming frameworks and "wizards" will get smarter -- saving
keystrokes (for folks who learn how to coax them to perform as
intended).
There will likely be some code coverage and testing tools that
emerge.
That seems to be happening now.
One potential advantage from all the AI hype is that it isThere are clearly those who think there's more to the brain than computationI think humans have the capacity for "insights" as some dendrite
(Penrose) and those who don't.
happens past another that makes an "atypical" association leading
to an unexpected revelation. I'm not sure how much of that is
learned vs. cultural -- you see a lot of "lazy thinkers" (doing
just the minimum that is needed to "solve" a problem)
I think you might be able to emulate that ability in an
AI by deliberately injecting "randomness" in the reasoning
process -- forcing it to evaluate "exceptions" and deliberately
rule them out (or in).
Time will tell.
On 7/29/2026 2:43 AM, Theo wrote:
Don Y <blockedofcourse@foo.invalid> wrote:
Anyone played with any of the "Open" AI's on local hardware?
Successes? Horror stories?
I've been doing some messing around (text processing, not LLMs, on Ubuntu 24.04). Takeaways:
"Creative writing"? Document prep?
1. The model is one thing, but it often exists within a stack of Python/C++/whatever to run it.
Yes, that seems to be an understatement.
That can be quite brittle - a huge pile of
Python dependencies, maybe some of them don't work on your machine and you have to dig into why (Python versions, OS updates, projects been abandoned, etc).
Have you encountered this -- or, just "expecting it" given the hassles with setting up a proper environment?
If you want to switch to a different model to do the same job, now
you have to switch to their stack of dependencies (ie effectively rewrite your code, rather than just swapping out one model file for another).
But, you should be able to move the model to different *hardware* (same flavor CPU/GPU, etc. but different MIPS/RAM)? I.e., if I wanted to see
the effect of CPU resources on its performance -- or, the value of
additional memory, bigger GPU, faster disk -- I should be able to do so?
[Does virtual memory even come into play -- or, is everything hardwired
to physical memory?]
I.e., I would (ideally) like to demonstrate:
- how each hardware resource contributes to performance
- how the models perform relative to each other (on "identical" tasks)
2. If you have a GPU it had better be NVIDIA. AMD and Intel GPUs effectively don't exist as far as these stacks are concerned. I have a perfectly good modern AMD GPU and all the models run on the CPU.
All of my GPUs (and Teslas) are nVidia. I'd also like to see if the models *choke* on a "too small" GPU -- or, if they just underperform (the equivalent of thrashing).
Have you verified your "local results" against online services (from the
same model(s))?
Any suggestions for "problems" that should be addressable by a variety of (general) models -- not overly reliant on a particular training set?
Has it been worth the effort (besides as a curiosity)?
I've been doing some messing around (text processing, not LLMs, on Ubuntu >>> 24.04). Takeaways:
"Creative writing"? Document prep?
OCR, sentence processing, language translation.
If you want to switch to a different model to do the same job, now
you have to switch to their stack of dependencies (ie effectively rewrite >>> your code, rather than just swapping out one model file for another).
But, you should be able to move the model to different *hardware* (same
flavor CPU/GPU, etc. but different MIPS/RAM)? I.e., if I wanted to see
the effect of CPU resources on its performance -- or, the value of
additional memory, bigger GPU, faster disk -- I should be able to do so?
Assuming the OS is the same, you should be OK. If it's a different OS version or distro you get to play whac'a'mole with the dependencies.
If the GPU is different then you get driver fun too. (even at the same vendor, the architectures can vary quite a bit between generations, and that can mean a different compiler/etc. Some things just aren't supported on one GPU+driver compared with another GPU+driver)
[Does virtual memory even come into play -- or, is everything hardwired
to physical memory?]
Without GPU support, it's a regular CPU process like any other. If you
don't have enough RAM then you swap. These models are small (MB) so it's
not a problem unless you're running on a Raspberry Pi or something.
I.e., I would (ideally) like to demonstrate:
- how each hardware resource contributes to performance
- how the models perform relative to each other (on "identical" tasks)
There are benchmarks for that, but you have to put the effort into running them
on your system.
2. If you have a GPU it had better be NVIDIA. AMD and Intel GPUs
effectively don't exist as far as these stacks are concerned. I have a
perfectly good modern AMD GPU and all the models run on the CPU.
All of my GPUs (and Teslas) are nVidia. I'd also like to see if the models >> *choke* on a "too small" GPU -- or, if they just underperform (the equivalent
of thrashing).
Most GPUs can use system DRAM over (slower, higher latency) PCIe if you
don't have enough GDDR, so it's either slower or thrashing swapping things back and forth.
I expect that you will struggle if the model is too big for the amount of DRAM you have: in theory the drivers could keep things 'working' by swapping GPU memory to disk, but progress will be minimal.
Have you verified your "local results" against online services (from the
same model(s))?
There aren't online services with the same models AFAIAA.
I'm not interested in signing up for accounts and token budgeting anyway.
Any suggestions for "problems" that should be addressable by a variety of
(general) models -- not overly reliant on a particular training set?
No idea. These are models that do one job only.
I'd never put my IP out to train something -- any more than IHas it been worth the effort (besides as a curiosity)?
If you have a lot of work for it to do, it's cheaper than paying for
cloud processing. This has cost me $0. You also get a better idea of how much resource it's actually consuming behind the scenes.
Any suggestions for "problems" that should be addressable by a variety of >>> (general) models -- not overly reliant on a particular training set?
No idea.˙ These are models that do one job only.
This is the heart of the problem I'm facing.˙ I don't see any real
use for AIs *in* my workflow, unless as bug hunters.˙ And, I expect
little value, there, as the sources of the "tough" bugs almost always
lie in concurrency issues.˙ (who writes single-threaded code in
the 21st century?)
On 7/30/2026 1:02 PM, Theo wrote:
I've been doing some messing around (text processing, not LLMs, on Ubuntu >>> 24.04). Takeaways:
"Creative writing"? Document prep?
OCR, sentence processing, language translation.
Is this a real need or something to play with? I.e., does having the
AI offer you real benefit?
Insofar as possible, I will be trying to create identical environments,
save for the hardware. The point being to identify/quantify the impact
of the hardware on performance. (otherwise, there's too many variables
to try to evaluate)
I am hoping to just build an environment and then pull the disk
and move it to another "very similar" machine (changing memory,
MIPS, GPU in/out, etc.)
Yes, that's the point. If *designing* with an AI, then you want it
to be performant as you are likely twiddling your thumbs waiting to
critique it's latest offering.
If, OTOH, you are using it to perform a particular *job*, you can
walk away and return when the results are available.
Consider the parallel to folks who "need" fast compiles because
they're waiting to see how their latest patches perform vs. doing
a "make world"
Exactly. But, having an online AI to act as a reference to validate
the output of the local instance would be reassuring.
OCR, sentence processing, language translation.
Is this a real need or something to play with? I.e., does having the
AI offer you real benefit?
This is for reading things in languages I don't speak. If want to read a book in eg German, first I need to get it into a digital form. Then I need to translate it, and re-apply the translations to the images (so diagrams make sense)
Translation apps like Google Translate will translate one image at a time, but if the book has hundreds of pages that is awkward to manage. You really want a pipeline that does everything automatically. No doubt that can be done via the APIs of the big translation services, but that starts getting expensive in terms of API costs. Each book has a lot of pages, and there
are a lot of books...
Insofar as possible, I will be trying to create identical environments,
save for the hardware. The point being to identify/quantify the impact
of the hardware on performance. (otherwise, there's too many variables
to try to evaluate)
I am hoping to just build an environment and then pull the disk
and move it to another "very similar" machine (changing memory,
MIPS, GPU in/out, etc.)
I expect that's going to be annoying with GPU drivers.
Yes, that's the point. If *designing* with an AI, then you want it
to be performant as you are likely twiddling your thumbs waiting to
critique it's latest offering.
If, OTOH, you are using it to perform a particular *job*, you can
walk away and return when the results are available.
Consider the parallel to folks who "need" fast compiles because
they're waiting to see how their latest patches perform vs. doing
a "make world"
That assumes you aren't going to need multiple roundtrips.
ie it's a simple
mechanical process that you run exactly once. If you need to close the
loop, ie tweak some of the inputs and repeat, then the latency matters as well as the throughput.
Even if you aren't tweaking the model you may need to tweak the inputs.
Exactly. But, having an online AI to act as a reference to validate
the output of the local instance would be reassuring.
I don't know how repeatable these flows are expected to be. ie can you put the same input into the same model run on different hardware and expect to get identical outputs? If you put the same prompt twice into ChatGPT will you get identical output? Or are there sources of divergence (either numerical or context)?
For simple stuff run locally I expect more determinism (better control of context, same hardware etc) but you don't know what is being used behind an online API.Exactly.You should be able to repeat a "process" on a given set of "data"
| Sysop: | Jacob Catayoc |
|---|---|
| Location: | Pasay City, Metro Manila, Philippines |
| Users: | 4 |
| Nodes: | 4 (0 / 4) |
| Uptime: | 496008:13:08 |
| Calls: | 178 |
| Files: | 605 |
| D/L today: |
7 files (18,398K bytes) |
| Messages: | 70,090 |