Announce: BAM + iPhone-based Head Tracking

[peterept]

Head tracking in Future Pinball is pretty awesome (even without 3D glasses)!

But it takes a lot of effort to obtain a Kinect (or other cameras) and set it up.

So I have created a new BAM head tracking plugin which makes use of your iPhone camera instead!

I have created an app called it Phonemote (a bit of a play on Wiimote), which uses the Front Facing TrueDepth camera and tracks your head at 60 FPS and sends it to the Phonemote BAM plugin running on Windows.

The only requirement is you need an iPhoneX+. (Well actually you can use it on other iPhones, but if they don't have a TrueDepth front facing camera, then you need to use the rear world-tracking camera and attach the phone to your head LOL).

Would love for feedback on what you think.

Setup Instructions

iPhone

1. On your iPhone, Download PhoneMote from the AppStore, and launch it. If it is tracking your face you will see a green circle.
2. Position your Phone so that the front camera can track your head for a typical pinball game.
I recommend a magnet phone holder stuck to the side of the pinball with the phone in portrait mode, or, place it at the end of the pinball table (against backboard) in landscape mode. Move your head around to make sure it tracks you well.

BAM (Windows)

1.. On your Pinball PC, download the PhoneMoteBAM.DLL and place it in your BAM plugins folder: C:\Games\Future Pinball\BAM\plugins
2.. Launch your Pinball, open BAM menu, and select the tracker as "PhoneMote"
3. Verify you see the Tracking data update (position, FPS, latency).
4. Select "Calibrate" from the menu, and it will display a calibration image which you scan with your iPhone rear camera.

Recommendation for Single Eye use

If you use BAM without 3D glasses, then I recommend trying the single eye view. Basically you cover one eye and set Phonemote to use the other eye (eg: cover left eye and set Phonemote to use Right eye). One easy way to do this if you wear glasses, is just cut out a small post-it note and cover the center of your vision by sticking it to your glasses. If you don't wear glasses, then a

That's it. Enjoy! Please share any feedback!

Special Thanks to @ravarcade for sharing the BAM Plugin sourcecode.

 

[GeorgeH]

You don't need Kinect at all now? I think Rav's 3D code was added to Visual Pinball a while back (not sure if it is still there). Will this work on VP also? I think the BAM code used to work on Pinball Arcade also. You should post this on VP Forums and VP Universe if you haven't already.

 

[peterept]

You don't need Kinect at all now? I think Rav's 3D code was added to Visual Pinball a while back (not sure if it is still there). Will this work on VP also? I think the BAM code used to work on Pinball Arcade also. You should post this on VP Forums and VP Universe if you haven't already.

Yes, no Kinect required! Tracking as good (if not better) from the iPhone TrueDepth camera.

I’m new to all this so I’ve never used Visual Pinball. But if it uses BAM then this should work (calibration step would either still occur in FP or using a printed PDF!). I’ll try it out! Thanks.

 

[GeorgeH]

Yes, no Kinect required! Tracking as good (if not better) from the iPhone TrueDepth camera.

I’m new to all this so I’ve never used Visual Pinball. But if it uses BAM then this should work (calibration step would either still occur in FP or using a printed PDF!). I’ll try it out! Thanks.

Thank you so much for this! I have never used 3D but it is nice of you to create this. Now folks don't have to spend over a hundred just to try it.

All I know is Rav's code for 3D was added to it in some way. Of course, you could ask Rav. I found this:

Visual Pinball X 3D – VPX + BAM Face tracking – Monte ton Cab

montetoncab.fr

I also found this video with some directions below the video:

 

[peterept]

So based on that tutorial the Phonemote tracker plugin should just work as-is!

 

[GeorgeH]

I found this on the following on the TrueDepth camera. Apparently, it is used for facial recognition:

"The TrueDepth camera system first debuted alongside the iPhone X in 2017 and can be found in every iPhone that has launched since then, as well as in the 3rd gen iPad Pro (2018) and later. This includes the latest iPhone 13, 13 Mini, 13 Pro, and 13 Pro Max, as well as the iPad Pro (2021)."

I would think you can use the app on your table without buying a new phone. Just park it on the table, then unpark it when done and bring it with you.

 

[peterept]

I found this on the following on the TrueDepth camera. Apparently, it is used for facial recognition:

"The TrueDepth camera system first debuted alongside the iPhone X in 2017 and can be found in every iPhone that has launched since then, as well as in the 3rd gen iPad Pro (2018) and later. This includes the latest iPhone 13, 13 Mini, 13 Pro, and 13 Pro Max, as well as the iPad Pro (2021)."

I would think you can use the app on your table without buying a new phone. Just park it on the table, then unpark it when done and bring it with you.

Yep! That’s how I suggest using it - use your phone (don’t buy another one)

I suction capped on a $10 magnetic phone holder to the side of my pinball (see original post video) and just run the app and drop the phone there when i want to play pinball. Easy!

 

[Carny_Priest]

Thanks for this. I will give this a try for FP, but maybe will also work for a lazy man's way to set a good POV for static VPX. It looks like VPVR was merged into the main VPinball development branch and an advance release candidate is available. All this is coming together right when I am looking for a solution. My system is rather old, so I don't know how well it will play as a standard way for me to play with VPX.

 

[Carny_Priest]

Had to manually enter the Host IP address for the cabinet to receive tracking data. Calibrate selection does not display an image for me to scan. I'm stuck here. Normally, I have the playfield screen as Full Screen but I also tried Windowed.

PS - I am using most current BAM.

 

[peterept]

Had to manually enter the Host IP address for the cabinet to receive tracking data. Calibrate selection does not display an image for me to scan. I'm stuck here. Normally, I have the playfield screen as Full Screen but I also tried Windowed.

PS - I am using most current BAM.

Sorry that it didn’t work - what did it display when you pressed calibrate?

 

[Carny_Priest]

Playfield screen is just a black screen.

I do run the playfield with NVidia DSR - it renders at 4K and downscales to HD. It's the way I run all my pinball software. I have not yet tried to run at native HD or test it out on my laptop, but that will be my next steps.

 

[Carny_Priest]

I tried with playfield running in native HD and I did not get a calibration image. As recommended for x64 OS, I had patched the FPLoader and Future Pinball executables with NT Core 4GB patcher. I tested with the unpatched versions and I confirmed through SHA256 hashes that the Future Pinball executable is identical to the one distributed with the BAM 2-1 installer. No calibration image.

My development laptop is an old laptop from work with security restrictions intact. I can connect it to Wi-Fi but I can't connect to other devices on the same network. Security on my current work laptop is even more restrictive, so I don't have any other systems to do further testing.

I am using current BAM v1.5-360; Windows 10 22H2; iPhone 11 w/ Phonemote v2.0.1, plugin is whatever is current on the project website, NVidia 1070TI, Intel i5 3570k; 16Gb RAM

In short, I can connect and receive telemetry from the phone but I don't get a calibration image and I don't see any head tracking happening on the playfield.

 

[peterept]

Hey, are you running in FP in Arcade mode? I think if it's not in arcade mode then BAM isn't passing the OpenGL mode to the plugin.

 

[Carny_Priest]

I am running with Arcade mode checked. Force Arcade Mode was not enabled in the BAM Settings, but enabling it did not change anything. I did not get the calibration image.

Edit - Is there a debug or log that I should look for?

 

[Printz]

Hi. I am also experiencing the same symptoms as Carny_Priest, did you ever find a solution?

 

[Carny_Priest]

Nope. I was interested in defining POV for VPX-GL, but the POV system has now been overhauled. It would still be nice to enable head tracking for FP or VPX. My iPhone is equipment provided by my employer with security features enabled. Are you using a personal iPhone?

 

[Printz]

I am using a personal iPhone without restrictions. Movement registers in FP but not in BAM_tracker (but counter is running when green circle is visible). Configuration screen goes black and screen on iPhone remains unchanged when hitting calibrate.

 

[Carny_Priest]

If auto broadcasting fails to work on your network then you can enter the Host IP address in BAM so that it may receive telemetry from the phone. If you have tried this already then you are stuck at the same place I am.

Could be that there is some BAM security feature that can be changed. I think it is one that allows COM. This may have nothing to do with the tracker driver, but it's the one thing I have not tried yet.

I was interested in this to help establish better POV for VPX tables, but VPXdev has entirely overhauled the POV system with 10.8, so I haven't pursued this further. It would be nice to see more players getting this working though.

 

[Printz]

Auto broadcasting did not work (using Google Mesh). Entering host IP works, but only within FP. In FP i can see the raw data from the iphone, but only get a black screen when calibrating and no movement of the playfield.

If i start "BAM-Tracker.exe" there is a running counter while head visible for iphone and the counter stops when head not visible, so there must be a connection between iphone and BAM.

The key is probably to get Auto Broadcast to work. I tried to disable firewall entirely, without any luck. I have also tried two different networks. Maybe it has to do with mixed 2.4 ghx / 5ghz networks?

 

[Carny_Priest]

I am stuck at the same place you are, then. I don't know what else to try.

 

[mysty]

Head tracking in Future Pinball is pretty awesome (even without 3D glasses)!

But it takes a lot of effort to obtain a Kinect (or other cameras) and set it up.

So I have created a new BAM head tracking plugin which makes use of your iPhone camera instead!

I have created an app called it Phonemote (a bit of a play on Wiimote), which uses the Front Facing TrueDepth camera and tracks your head at 60 FPS and sends it to the Phonemote BAM plugin running on Windows.

The only requirement is you need an iPhoneX+. (Well actually you can use it on other iPhones, but if they don't have a TrueDepth front facing camera, then you need to use the rear world-tracking camera and attach the phone to your head LOL).

Would love for feedback on what you think.

Setup Instructions

iPhone

1. On your iPhone, Download PhoneMote from the AppStore, and launch it. If it is tracking your face you will see a green circle.
2. Position your Phone so that the front camera can track your head for a typical pinball game.
I recommend a magnet phone holder stuck to the side of the pinball with the phone in portrait mode, or, place it at the end of the pinball table (against backboard) in landscape mode. Move your head around to make sure it tracks you well.

BAM (Windows)

1.. On your Pinball PC, download the PhoneMoteBAM.DLL and place it in your BAM plugins folder: C:\Games\Future Pinball\BAM\plugins
2.. Launch your Pinball, open BAM menu, and select the tracker as "PhoneMote"
3. Verify you see the Tracking data update (position, FPS, latency).
4. Select "Calibrate" from the menu, and it will display a calibration image which you scan with your iPhone rear camera.

Recommendation for Single Eye use

If you use BAM without 3D glasses, then I recommend trying the single eye view. Basically you cover one eye and set Phonemote to use the other eye (eg: cover left eye and set Phonemote to use Right eye). One easy way to do this if you wear glasses, is just cut out a small post-it note and cover the center of your vision by sticking it to your glasses. If you don't wear glasses, then a

That's it. Enjoy! Please share any feedback!

Special Thanks to @ravarcade for sharing the BAM Plugin sourcecode.

Hope there will be an "Android" version as well of this app ........ Great app, so it looks :D !!

 

[cleody1981]

Anyone get the calibration image to show? Would like to know if any progress or solution for this?

 

[Anthias]

Hey this is neat, well done and thank you. If I ever succeed in getting my cab built this is a must have.

But if we can get it for Android (if anybody has the necessary knowledge to make it) not only can those of us not in the appleverse benefit, but I recently bought a face recognising android phone for under a hundred bucks at the supermarket. I'd pay that to just bolt it and leave it on the machine! (An iPhone would be a bit expensive for that.)

Whoever does it has time to spare lol, no rush for me. I finally got the controls hooked up in my cab only to have psu and mobo fry, and also no working screens. The reduced income caused by the silly season means the project is scuppered for a few months. Unhappy man I am.

 

[trippingale]

I was looking at replacement front camera for I phones, about $5-10 on eBay, is it possible to use stand alone camera? I do know once you go thru the apple chipset it becomes mated pair, would be interesting if anyone has hacked and made a stand alone version.

 

[derpyder]

Same issue here with calibration returning a black screen.