FlexPoser: Real-time face tracking and animation [Public testing phase]
102 replies, posted
I'm glad to finally release this addon that I have been working on since this February: FlexPoser allows players to connect their webcam and map their own facial motions directly onto their characters. This works online (players can see each other's facial expressions in real-time) and in combination with any game mode and other addons. I'm hoping this will improve your machinima, RP and just provide fun altogether.
Video: [URL="https://www.youtube.com/watch?v=P4EoH7lz5sg"]YouTube link[/URL]
To use the addon, the following steps are needed:
- FlexPoser must be installed on the server
- Players that want to use FlexPoser must download a binary module (for other players: no additional steps needed, you can still join the server and see the expressions)
[B]Instructions for server hosts (also for single player and listen servers):[/B]
Subscribe to the addon in the GMOD Workshop: [url]http://steamcommunity.com/sharedfiles/filedetails/?id=282498239[/url]
[B]Instructions for players:[/B]
Follow the steps on this page: [url]http://flexposer.com/download.php[/url]
[B]Privacy notice: statistics gathering (enabled in multiplayer only)[/B]
I developed this addon as part of my uni graduation project for a master's in game technology. In the current state of the addon, some anonymous statistics are tracked about servers and users:
A summary per game session: the game mode, the map, the number of players, and the number of face tracked players
A summary of a face tracked player's participation in the session: for all 21 facial animation channels (such as left eyebrow raised, etc), the average and variance of the activation level during the session
All data is stored anonymous: server IPs and player IDs are encrypted before they are stored in the database, i.e. if a player participated in multiple sessions those sessions can be linked, but there is no (feasible) way of finding out who that was. [B]In case you really don't want any stats tracked from your server[/B], you can run "fp_trackingenabled 0" once in console after starting your server (dedicated or listen). Statistics will be disabled directly for that session and all subsequent sessions. You can turn it on again with "fp_trackingenabled 1". Statistics will be removed in its entirety on August 8th.
Also related to my research, the in-game GUI has a tab labeled "Questionnaire". In it you can find (gasp) a questionnaire. :) It contains some questions on how you used the addon. My intention with all this data is to study how face tracking can enhance the multiplayer gaming experience. Some of my main questions are: in what types of games do players like to use this feature? In what types of games are they more expressive? etc.
Research stuff aside, I also want your feedback on this addon. Please tell me if you can get it to work, and also if it doesn't work! I have tested this on a handful of machines, but since I'm inexperienced with releasing stuff, please be patient if the addon (esp. the binary module) doesn't work for you at once. I'll try to fix it ASAP. And let me know what you think!
[B]Face tracking library[/B]
I used the open-source library [URL="https://github.com/kylemcdonald/FaceTracker"]FaceTracker[/URL] developed by [URL="http://jsaragih.org/"]Jason Saragih[/URL] and maintained by [URL="http://kylemcdonald.net/"]Kyle McDonald[/URL], available for non-profit and academic use (see its license).
[B]Update July 11th, 2014: [/B]
I am aware of a lot of issues related to webcam compatibility and will release an update next Thursday which will hopefully address most issues. At the very least, many crashes will be prevented and useful errors will be shown instead.
[B]Update July 13th, 2014: [/B]
I just updated the server addon and the binary module. Server addon should update automatically via Workshop. Please download the latest binary module at [url]www.flexposer.com/download.php[/url].
- Improved the instructions included with the binary module download. Most important change: you must download the 32-bit VC++ redistributables, even when you have an 64-bit OS.
- Fixed crash when face2.tracker is missing, now displays in-game error instead. File is also renamed to fp_face2.tracker to emphasize its relation to this addon.
- Fixed crash when webcam was pulled out while tracking, now displays in-game error instead.
- Fixed crash on Enable/Load profile when webcam is incompatible.
- Fixed the white rectangle in the "white rectangle represents your face" screen sometimes being of incorrect scale
- Added webcam selection: module now checks how many video capture devices are connected and allows you to choose which device to use. An option to remember a default webcam is available, so that you do not have to select it every time.
- Added console command fp_openmenu to open FlexPoser menu, as alternative to the !flexposer chat command.
- Added fault recovery mechanism: module periodically performs an asynchronous check using a different face detection algorithm (Viola-Jones) and resets tracking if the results are too different. The mechanism requires a file fp_haar.xml, now included with the binary module download.
- Limited head rotation to 45 degrees for each axis (pitch/yaw/roll), because anything further than 45 degrees cannot be accurately tracked.
- Made Questionnaire panel more compact in size so that people with lower Y-resolutions can also complete it.
[B]Update July 14th, 2014: [/B]
Just updated the server addon and DLL again to resolve issues that were introduced by last night's update. Special thanks to [B]erkor[/B] for trying out 10 test builds, helping me a lot with localizing the problems.
- Fixed crash when Enabling the addon, caused by retrieving device list in an unsafe way. Now uses recommended method for Windows for listing devices.
- Fixed camera turning on and turning off right after. If your camera did this, then it probably [B]is[/B] supported, so please try again!
- Device listing now lists names instead of number, i.e. "Logitech Webcam C210" instead of "Device 1"
- White rectangle in the face detection screen now moves the same way as you horizontally.
Would players without the binary module still see the expressions?
[QUOTE=pkhzor;45317312]Would players without the binary module still see the expressions?[/QUOTE]
Module is only to track and broadcast your expressions.
I've been looking forward to this for a while. Great Work!
[QUOTE=vexx21322;45317332]Module is only to track and broadcast your expressions.[/QUOTE]
Correct, thx. (updated post)
Can anyone report if the binary module is working? I tested it on some different machines but they were all guys with the same tools as me installed, so I'm kinda worried about that.
This is simply wonderful. Great work!
I can see people making very creepy faces with this, I may add this to a DarkRP server I develop for and scare the fuck out of some kids.
doesn't HAC have this anyway? (henry's animation tool)
This is a pretty cool addon.
Do you plan to add any further expressions into this mod? Such as the ability for it to mirror vowels correctly?
I was stupid and placed it in the actual bin folder instead of lua/bin. Whoops.
[QUOTE=Dancingonpie;45321663]Do you plan to add any further expressions into this mod? Such as the ability for it to mirror vowels correctly?[/QUOTE]
Hi, with the way Source models are facially rigged its hard to track and recreate more complex mouth shapes than this. I already use 3 of the 4 types of lip movements (the 4 being: pucker, funnel, part, stretch), the rest is mostly up to the calibration and lighting conditions of your camera image. If you tried before during the day, try again during the night, you might get better results. Sunlight is quite a big problem, even when filtered through curtains.
The other factor that can create an "effect" of better vowel mirroring is the tracking frequency. In case you haven't found the option, go to the preferences tab in-game and set Tracking rate to high. Note that altough it says 30 FPS, it will only track as fast as it can, i.e. some webcam frames will probably be skipped because you need a monster PC to run the game and track at 30 FPS. Lowering image scale might help as well, as it gives more priority to speed then. Here is a demo of what the resulting animation can be with decent calibration and if no webcam frames are skipped: [URL="https://www.youtube.com/watch?v=qZbUANKoPhg"]YouTube link[/URL]. This one was not tracked real-time but it does use the exact same tracking algorithm, just without skipping frames.
You must download a binary module (dll file) for Garry's Mod in order to track and broadcast your facial expressions. This module allows the game client to communicate with your webcam and track your face. The binary module is only available for Windows at the moment.
To get the binary module, please follow these instructions:
Download and install Visual C++ redistributables for Visual Studio 2012.
Download the binary module: gmcl_flexposer_win32.zip (99 downloads).
Extract gmcl_flexposer_win32.dll and face2.tracker to your GarrysMod\garrysmod\lua\bin folder.
Join a server with the FlexPoser server addon installed, type !flexposer in chat to open the FlexPoser GUI and follow the instructions.
Note: If you joined a FlexPoser enabled server before you downloaded the binary module, you can open the FlexPoser GUI in-game and press Enable or Retry to detect the module. You do not have to restart Garry's Mod or rejoin the server after extracting the module.
I will not be VAC-banned for this?
Add the answer in the F.A.Q. too
oh man. my head angles keep flickering though, probably thanks to laptop.[/QUOTE]
Head rotation tracking doesn't work when you tilt your head too much (like near 90 degrees), sadly. You can turn head rotation tracking off in Preferences if you want.
[editline]8th July 2014[/editline]
[QUOTE=Pizhik;45325420]I will not be VAC-banned for this?
Add the answer in the F.A.Q. too[/QUOTE]
Good suggestion. The DLL is [URL="http://wiki.garrysmod.com/page/Creating_Binary_Modules"]proper binary module[/URL] that is only loaded when required by LUA and only interacts with the game when called from LUA, so it will not get you VAC banned.
Thats so freaking amazing O_O
Am i doing it right tho?
This binary module only works on Windows platforms.
aw. If you need help porting it to Linux, you can give me a shout, I have plenty of experience making cross-platform code with C/C++
imagine the machinimas
God dayum. Nice work! This should be default with gmod.
could this potentially be expanded to TF2 HWM faces? they are much more flexible (nice meme) in regards to facial animation, and could produce much more lifelike results. it wouldn't be very viable for playermodels (unless you use PAC) but it could be greatly useful for posing and machinima
Once I start flexposer gmod crashes, I've already made sure that the library and the tracker file are in lua/bin, along with installing the c++ redists, I dunno what it could be.
Forgive me if this is planned, but any chance of a tool so you can put it on a ragdoll?
I got a problem.
I got the VC Redist for 64 bit, but it crashes after a few seconds of enabling the module.
It's frustrating me ;-;
That's amazing - I literally thought of this idea myself roughly an hour and a half before I saw this thread appear!
Awesome that you have done it! I would have never got around to doing it myself - it looks excellent, good work!
Adult, thanks! Indeed I could use some help if I want to port it to Linux (and Mac). I'm still thinking about it and I don't have the time right now to start porting, but I'll keep you in mind and your help is appreciated.
Vipes and Darkomni, there can be many uses for this but because I have to focus on my studies I'm gonna leave the features as is. Are you any good with LUA? If so I don't mind you unpacking and editing the LUA for private use, as long as you don't redistribute it. I can give you some pointers.
ph:lxyz, thanks! I thought the idea was cool too and was lucky that I found a supervisor at my uni that allowed me to combine it with studies.
a-cookie and erkor (and others with crashes), sorry this is my first release so I expected some troubles. Can you try the following to narrow it down? Start a game and run the following in console line by line:
lua_run_cs print( __GetFlexPoserVersion() )
It will likely crash at step 2 or 3, can you tell me what happens?
NisshokuZK, you have one hell of a portfolio for your age my friend. Awesome work!!
I entered the commands and I didn't crash at all and __GetFlexPoserVersion( ) returned 1 in the console
~Yeah, I get the same, pretty epic though!
This is exceptionally cool. I was getting crashes so I tried the console commands, then no crash.
My housemate is severely creeped out right now, so that's a plus. Not sure if it's the twisted Magnuson expressions or my gurning at my webcam.
Bad news: my beard seems to prevent proper expression detection but since my fiancée also has the same problem the fact that an addon can do this well is pretty amazing!
Sorry, you need to Log In to post a reply to this thread.