(Yeah, it's gonna be a long one!)


### INTRO ###

So I decided to play around with head tracking. That's software that
tracks the movement of your head and then maps it into a game, so that
when you turn your head left, the view turns left; look up, and the
screen pans up, etc. It's sort of a poor-man's VR. (Actually, most VR
headsets use head tracking too, to move the viewpoint about as you
look around in real life. But the stuff I was using didn't need a VR
headset).

Early head-tracking needed specific hardware to work; either some sort
of IR signaling device (and paired detector) or a special camera.
Nowadays, there is software (and sufficient CPU cycles) available that
you can do the same thing with an off-the-shelf webcam. This latter
method isn't quite as accurate but it's generally cheaper since you
can re-use existing hardware. And I like cheap... especially for a
product I'm not sure I'll really use or want.


### THE SOFTWARE ###

There are a number of software products available, both free and
retail. Amongst the former are Trackhat and Opentrack; with the
latter, Lookpilot and BeamTracker (both on Steam). There are probably
more options, but those were the four I focused on. I ultimately
picked "LookPilot". First, because it was the cheaper of the two paid solutions, and second, because it promised the easiest
'drop-in-and-use' experience. I didn't want to spend a lot of time
tinkering with settings or compiling or begging the software gods to
let the app work with my webcam, and Lookpilot seemed the best suited
for that need.

And in this regard, "LookPilot" succeeded very admirably. I installed
the app, plugged in the webcam, and that was pretty much all I needed
to do to get it to work.

[Well, not really. I had to fiddle around with Windows
settings because I had ruthlessly disabled all access
to webcams in Win11's privacy settings, and had to un-do
all that work to let LookPilot work. But that's a ME issue,
and nothing to fault the software with. Once I cleaned up
my mess, LookPilot had no issues.]



### THE HARDWARE ###

My hardware set-up is not optimal. While my PC was more than
sufficient to run the software, there were other issues. My webcam is
a 15+ year old device that wasn't high-end even when I purchased it.
It supports a maximum resolution of 640x480 at 30fps. It is very
purposefully a bad webcam (if you're going to make me endure a
video-call, you're going to suffer for it with a low-resolution image
on my end ;-), but I also wanted to see how low-end I could go with
the software. I figured if the software worked even with such shitty
hardware, it might convince me to upgrade the webcam at some point.

Another issue was that my primary gaming monitor -a 50" HDTV set three
feet away from me-- was offset to my left. This means whenever I play
a video game, I'm always looking slightly to my left. I wasn't sure if
this would cause issues with the software, but it was an added level
of complexity it would have to deal with.

With that out of the way...



### THE GAMES ###

I wanted to try the head-tracking software out with three specific
programs: "Microsoft Flight Simulator" (2020 edition); "American Truck Simulator", and "Call of the Wild: The Hunter". All three games, I
knew, supported natively head-tracking to one degree or another so I
figured they were good contenders. Also, I happened to have all three
installed on my computer.


* American Truck Simulator
The first game I tried was, of course, that damned truck sim. ATS has
you sitting in the cockpit of a truck, and while you drive with a
wheel (or the keyboard, or a gamepad, or whatever) you can use the
mouse to swivel your head about and look around your truck interior.
And while it's no real hardship to do so, it does require you move
your hand off your normal controls to fiddle about with the mouse,
which means most of the time you just leave the head of your in-game
avatar staring straight ahead. But of course, when you're driving, you
really want to be looking around; at your mirrors, through the side
windows when turning, down at your coffee cup to make sure there's
still something to drink... so the idea of having head-tracking seemed
a perfect match for the game.

The game instantly recognized I had head-tracking available. I didn't
have to change any settings for it to work. I started driving and when
I moved my head, the viewpoint moved too. Initially, the settings were
a bit too sensitive; unless I locked my IRL head to perfect
immobility, the most minor shudder (or you know, just breathing) made
the screen jiggle and twitch. I don't entirely blame "LookPilot" for
this. As I said, the resolution of my web-cam is very low. For
another, the lighting in the room was also very poor (it didn't help
that I was testing at night). Illuminating the surroundings helped
some. But even more useful, the app came with a number of
user-submitted tracking presets for ATS/ETS, and using those I was
able to dial-in settings that made the experience far less nauseating.
Even at its best, there was still some minor twitchiness... but it was
mostly unnoticeable.

The driving experience was... different. It was neat to be able to
look down at the GPS only when I needed to, rather than having it as
an always-in-your-face HUD element. For some reason, using the
head-tracking made me much more aware of the speed limit signs; I just
seemed to be able to notice (and focus) on them much more clearly and
often. There was a bit of lag and slowness to the tracking; nothing
game breaking or even that bothersome, but it did sometimes feel like
my head was moving through molasses.

[note: this probably can be changed in settings but I
wanted to get a baseline experience before fucking
around with stuff]

Slightly more annoyingly, I only had about a 200 degree radius to turn
my head; about 100 degrees left or right. This meant I couldn't really
crane my head backwards to see what was behind me, and that limitation
felt somewhat unnatural. I could look out my windows, but because of
the limited radius, it didn't really help me much.

[note: this too can be changed in settings... but it's
really unnerving to move your head slightly to the left
and suddenly be looking directly behind you.]

An odd effect is that --with this head-tracking, smart cruise control,
and lane-keeping enabled, I found I could drive hundreds of miles on
the highway with my hands completely off the control. Lane-keeping
kept me from drifting out of my lane, so I didn't need to steer. Smart
cruise control kept me at speed (and slowed down if there was traffic
ahead) so I didn't need to accelerate. And with head tracking, I could
look around without using the mouse. It felt really odd to go minutes
without touching a control!

Still, overall I found the experience really neat, if a bit gimmicky.
If I were to tweak the settings a bit (and maybe upgrade the webcam) I
can imagine using the headtracking as a regular feature.



* Microsoft Flight Simulator 2020.
This game worked pretty much the same. I launched the game (and then
waited for the inevitable 5GB update to complete), and as soon as I
was in the plane's cockpit, moving my head resulted in the on-screen
image shifting to the same degree. I particularly liked that if I
lifted myself up a bit, it was as if I were peering out the side of my
cockpit and looking down past the wings at the ground. Other than
that, the experience was mostly the same as with American Truck
Simulator. There was the same sluggish response and limited ability to
see around you (both of which, again, I'm fairly sure can be fixed by
tinkering with the settings).

These limitations were a bit more obvious, though, in MS Flight Sim.
For one thing, I had almost no ability to look straight up (which is
sort of necessary when moving in a 360 degree environment), and for
another my virtual head kept clipping through the canopy of the
aerobatic plane I was flying. Once more, I am sure these are solvable
issues but with MSFS you'll probably have to tune-in the head-tracking
for each different plane you fly to avoid the clipping issues.



* The Hunter: Call of the Wild
This one is a first-person hunting game. It's quite pretty and I
almost never play it to hunt; rather, I just like walking through its
expansive maps and pretending that I'm hiking through the woods. I
have, in fact, actually played this game while on a treadmill! So the
idea of walking in one direction while I swing my head about
to-and-fro looking at the sights seemed quite intriguing.

Unfortunately, the game only supports a very specific type of
head-tracking called TOBI, and I was unable to get it to work with
LookPilot. So that pretty much killed this experiment before it got
started.



### MY CONCLUSIONS ###

This definitely was an interesting experience. I'm not sure I'm
entirely convinced of its utility... but neither am I of the belief
it's useless. It does feel very much like a gimmick... but a fun one.

There are some very obvious problems with the tech, though, and I
don't mean LookPilot's specific implementation. It's a problem common
to any web-cam based head-tracking implementation. Mainly, that it
relies on you turning your head as a way to indicate that you want to
look elsewhere. While this is a very natural thing to do, it does
cause an issue because --when you're playing a video game-- you sort
of need to keep looking straight ahead at the monitor in front of you.
So whenever you turn your head to look left, you are taking the screen
out of your direct line of sight.

Now, in my case, with a giant 50" monitor stuck only a few feet away
from my face, this wasn't quite so traumatic. Even if I turned my head
90 degrees to the right, I could still see most of my screen from the
corner of my eye. But if I had a smaller screen (or was playing on a
laptop) turning my head that much would have left most of my screen
outside my range of sight. That's a bit of an issue if you're racing
down the highway at 80mph!

To some degree, your brain automatically tries to compensate for this.
When you turn your head left, and (in order to keep the screen in
focus) your eyes shift right. It mostly worked in my case (again, that
giant screen was really helpful) but there was a weird feeling of
disconnect that I was always looking in two directions at once. It
wasn't nauseating or anything, but it definitely tired out my eyes
faster than I'd normally get from playing.

To some degree, I think tweaking the settings might help a bit with
this... as might centering my screen so it's not offset to the left,
or getting a more accurate web-camera. Projecting the game onto
multiple displays might help too. Still, I think overall this is a
limitation of the technology. In VR, the monitor is attached to your
face so it always is centered on where you are looking. But with fixed monitors, head tracking means you need to look away from them to turn
your view.

Another issue is that this tech relies on tracking your face. That
means the camera must always be able to see your face to determine
which way you are looking. That means that if you lean to look over
your shoulder, the camera can't see your face anymore. Similarly, you
can't really look straight up without it losing its ability to make
out your facial feature. This limits the tech to about 200 degrees of
rotation, and ~120 degrees up/down. You can't really look at your toes
or the stars with this tech. Maybe if you had multiple cameras (which
LookPilot doesn't support) but with a single-camera? It's just not
possible to get more range.

I rather enjoyed using the head-tracking in "American Truck
Simulator", even if it did tire me out faster. Checking my mirrors
felt a lot more natural, as did using the in-dash GPS rather than
through a video-game HUD element. But at the same time, some of the
turn-radius limitations did sometimes get in the way. It didn't make
for a worse experience but it did make for a different one. It did
feel very gimmicky. I might keep using it; I might not. I'm not sure
either way is better.

"LookPilot" also seems fairly limited with how it interacts with
games. You aren't going to just be able to use it with, say, Quake or
most FPS games, for instance... unless somebody has built a mod that
adds head-tracking functionality into the game. Even games which have
native head-tracking built in, you'll still need to tweak the settings
to get the best experience. And even-then, most of the pre-configured
settings in "LookPilot" are user-submitted, with varying levels of
quality. So maybe technically you can get it working with "Crysis"...
but it might not be worth the effort.

With extra monitors the experience might be better... but the aim of
my experiment was to see how well the head-tracking software would
work without too much effort and without adding expensive extra
hardware. And I have to admit, in the two games were it worked, the
experience was fairly easy to set-up. I liked it enough that
--although I started with the 14-day trial "Lookpilot" offered, I
quickly upgraded to the paid version. It's worth investigating, if
you've an interest (the free open-source versions will, from
everything I've read, require quite a bit more tinkering though).

But I don't really think this will ever advance part the 'neat
gimmick' phase.





--- PyGate Linux v1.5.14
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)