PDA

View Full Version : Nuicapture markerless REALTIME motioncapture kinect system



geo_n
11-02-2012, 12:08 PM
http://www.youtube.com/watch?v=J5NiVWnl_vM&list=UUDsZWfELx-oE3C-qfqD4SbQ&index=2&feature=plpp_video
Saw this while googling kinect mocap.


Meanwhile brekel will have pro versions for mocap and point cloud. Blender is going to support facial mocap from brekel, too.
http://www.brekel.com/

Titus
11-04-2012, 01:03 PM
I'll test Brekel. I've been considering Ipi but its pipeline is slow.

silviotoledo
11-04-2012, 05:23 PM
http://youtu.be/yZEamKAcEjQ

This is amazing.

The result from a a point cloud animation data is translated to some morph targets.

My question is. How?


Didn't understand how.

Is there some specific ways to model morphs and some specific names/shapes for morphs so it will work correctly?



Silvio Toledo

silviotoledo
11-04-2012, 05:44 PM
ok. Got it! 11 pre defined sets that are connected to 11 predifined morphs.

Same as Jimmy Rig, but Jimmy Rig uses 6 morphs only, actually.

The good about Jimmy Rig is that is translates data to Lightwave.

Brekel sends data throught FBX but I'm not sure Lightwave will read a group of points data as only one envelop as endomorph. So we need retarget.
Maybe FBX files give us a only one graphic that represents a group. Not sure.

Hope Jimmy Rig advances in this area.

Greenlaw
11-04-2012, 07:35 PM
I'll test Brekel. I've been considering Ipi but its pipeline is slow.
This is just my personal opinion but I can't say using iPi DMC is in anyway slow. A year ago, my wife and I captured all the motions for Happy Box (http://bit.ly/NKsLrX) in a couple of hours on a Saturday afternoon using iPi DMC with two Kinects. In this single session, we captured motions for three characters and eighteen shots, which included a couple of 360 degree motions. Tracking and retargeting was done over a couple of evenings after that. We were extremely pleased with how quickly this came together because we had an insanely tight deadline for this project.

By comparison, for big-budget productions at work we employ a professional mocap studio to capture our motion data, and a typical capture session requires a large crew and can take an entire day or longer, and we don't usually see the mocap retargeted and cleaned up on our rigged characters for several days. Of course there is a big difference in quality between what a professional motion capture studio does and what I do in my living room using two Kinects but there is also an enormous difference in cost and speed.

If you're going to compare iPi DMC with the handful of realtime single-Kinect systems currently available, you may want to consider other things besides the apparent 'speed' of these systems. Realtime single-Kinect systems are pretty neat because they provide instant mocap data but in the end you should also compare the quality of the data. Depending on the type of motions you want to capture, the time you thought you saved in capture may be lost once you begin cleaning up the motions for production use.

You also need to consider that iPi DMC offers more options and features: besides single and dual Kinect, you can optionally use 4 to 6 PS3 Eye cameras which allow you to capture a performance space as large as 20 x 20 feet. In the next few weeks, you will also be able to track wrist rotations and handheld props--today, they released a version that lets you capture rotational data using two Wii Motion Plus or two PS3 Move controllers, and the update for tracking this data is expected to come out in two to three weeks. iPi recently adding head tracking--currently, you can track H and P rotation, and Y rotation is arriving soon. Next month, iPi intends to add support for tracking two interacting performers in the same take. I don't think you're going to see all this in a realtime system any time soon.

To be fair, iPi DMC does not offer face capture like Nuicapture can, and the iPi devs have stated that if they ever offer face capture it will most likely be sold as a separate dedicated program.

That Nuicapture can already do both simultaneously is quite remarkable. It should be interesting to see how people use this capability in their 'home brew' games and movie productions.

G.

geo_n
11-04-2012, 07:36 PM
Ipi is slow but its the only one that supported ps3 cams to give accurate mocap.
Brekel looks very interesting though

http://www.youtube.com/watch?v=yZEamKAcEjQ

geo_n
11-06-2012, 08:11 PM
If anyone is interested to get lightwave supported for brekel kpf, please contact the developer.
http://www.brekel.com/forum/viewtopic.php?f=31&t=1490&sid=c38d076266b8fe02e211abf96e317d55

I've started a thread there and it seems Brekel was a lw user before and he's open to help any scripter to have his face mocap software work with lightwave. I showed him papagayo script from Dodgy and it seems brekel kpf can work on morphs, too. Some people already ported a translator for c4d, blender. Hope we can have lw support.
For 99usd facial mocap system, can't be beat. Results with the monkey look acceptable.

Greenlaw
11-06-2012, 10:49 PM
Thanks for the info! Brekel Face does look very interesting. I've been following the development of this and other Kinect based face capture systems for a while. FaceShift was one system that was looking really good but when it was released a few days ago, I felt it was a little expensive for me and my little film projects. Brekel Face looks like it will hit the sweet spot between price, capability and performance. If it gets direct Lightwave support, I'm definitely in. I'm still keeping my eye on Jimmy|Rig's face capture development for the same reason.

2013 is going to be a fun year for indie mocap makers! :)

G.

geo_n
11-07-2012, 02:03 AM
For indie films and low budget projects I think brekel kpf is a good system. Ipi express plus brekel kpf is very affordable.
Faceshift looks more hidef scanning from what I've seen. Not sure of the price range.

Greenlaw
11-07-2012, 09:28 AM
Faceshift is Kinect based, so it's technically 'lo-def' like the others. The quality of the results appears to be quite high considering the hardware limitations but the cost is high-ish too (IMO): $1500 a year or $800 a year for the 'indie' license. To be fair, they do offer a $150 non-commercial license, which seems like a very good deal if it works as advertised. I might have to look into that as soon as we're finished with our latest Brudders movie. (Little Green Dog is a non-commercial operation of course--at least for now.)

Regarding other systems: Brekel Kinect Face Pro is on my watch list for face capture but I'm mostly waiting to see somebody use it with Lightwave first. As much as I'm tempted to try it, I personally don't have to bandwidth to do more R&D right now. I don't know what to think about Nuicapture yet because so far all I've seen is raw capture data--I am curious to see what early adopters do with this system. I'm also keeping an eye on Jimmy|Rig because it will very likely support Lightwave and, since I already own a license, I can try its face capture system immediately when it's ready.

For me, I predict the ideal low-cost 'indie production' combo will be iPi DMC 2.0 Basic or Standard for body capture (both versions feature dual Kinect, Express is single Kinect only) and Brekel Kinect Face Pro for face capture. In my experience, iPi DMC has been easy to use and currently produces the best results for a Kinect-based system, and BKFP appears to offer a lot of features (11 morphs vs. the usual 6) and flexible third-party support for a low price. For example, it's not just Motion Builder ready, it also records directly into DAZ and Poser! (Somebody please add Lightwave to this list!) :p

This is strictly my opinion of course and I'm still waiting to see what others do with BKFP first.

G.

geo_n
11-07-2012, 07:07 PM
Regarding other systems: Brekel Kinect Face Pro is on my watch list for face capture but I'm mostly waiting to see somebody use it with Lightwave first.

BKFP appears to offer a lot of features (11 morphs vs. the usual 6) and flexible third-party support for a low price. For example, it's not just Motion Builder ready, it also records directly into DAZ and Poser! (Somebody please add Lightwave to this list!) :p

This is strictly my opinion of course and I'm still waiting to see what others do with BKFP first.

G.

Shame, blender and c4d translators are being created and already works partially by morphs. Again maybe newtek needs to reach out to these developers. They won't come to lw.
Another app that will not have lightwave support like render engines out there.

Greenlaw
11-07-2012, 10:41 PM
Has anybody fogbugzed a request for this? Otherwise, the developers may not realize there is an interest.

G.

geo_n
11-08-2012, 12:00 AM
Doubt it. There's less than 20 views at brekel forum for lw support.
http://www.brekel.com/forum/viewforum.php?f=31&sid=29f6ef2279fb1822e6e2f22f94f7e1d4

Greenlaw
11-08-2012, 12:15 AM
I would put in the fogbugz request except I'm not actually using Brekel Kinect Face Pro yet so it wouldn't be an informed request. I'll look at the trial version later this week--if it looks worthwhile, I'll submit the request for a LightWave link.

G.

silviotoledo
11-08-2012, 01:25 PM
Does faceshift work with Lightwave?

Greenlaw
11-08-2012, 03:32 PM
I don't know. I signed up for FaceShift beta a while back but I simply did not have time to test it out. (Sigh! Too many big projects right now.)

Except for Jimmy|Rig, I haven't seen anybody using Kinect based face capture with LightWave. I'm looking forward to trying J|R's take on face capture but the last time I checked, it was still in early development.

G.

geo_n
11-13-2012, 11:01 PM
Greenlaw I fogged it. Case 52946.
https://fogbugz.newtek.com/default.asp?52946_ri5184c3couo1eil
Brekel said he can make an output from his software but he can't understand the code from motionmixer. Nobody understands it I've already asked somep people. Maybe you have resources that could help, a direct line to Newtek. :D

silviotoledo
11-19-2012, 07:09 PM
Just to answer myself. yes, FACESHIFT does work pretty with lightwave. I did my own tests using FBX. 48 morphs is exported and we are able to edit the graph editor of each one, if necessary.

geo_n
11-19-2012, 07:55 PM
So the fbx from faceshift contains model data with morphs? Can we use our own head model in faceshift?

Greenlaw
11-20-2012, 10:05 AM
I think Brekel Face Pro exports morph data via FBX too--just not sure how well it works with Lightwave yet. If I have time this afternoon today, I'll install it and try some simple tests. In reading the docs, it apparently produces higher quality when using Kinect for Windows. I only have Kinect for XBox but I imagine that will be enough to help me decide if I need to get the Windows version.

Later, we might look into trying the non-commercial license for Faceshift. Personally, I think the commercial license is too expensive but it will be good to see how it compares with Brekel Face Pro.

We're not planning to use face capture on our current movie project (we're using an updated version of the mouth system we set up for Happy Box,) but if any of these FC systems works well, maybe we'll use one on the next movie.

G.

Greenlaw
11-20-2012, 10:22 AM
Just to answer myself. yes, FACESHIFT does work pretty with lightwave.
That sounds very promising! Is it be possible to show an example?

geo_n
11-21-2012, 03:34 AM
I tested the sample files from brekel. they're only targetting points. No mesh no morph. The dev doesnt know how to support lw so thats the end of the story. Faceshift is asking 1500usd annually for the software I think. Hehe.

Greenlaw
11-21-2012, 04:30 AM
That's curious because the Brekel Face Pro manual and software specs do mention the option to include morph data in FBX. According the manual, it supports 11 morph/blend targets which are saved in values from 0 to 1. (Just to be clear, the data can apparently be saved as point cloud, joints, bones, or morph data--right now I'm only talking about morph data in FBX or course.)

It's my guess that FBX morph data probably does not translate directly to MorphMixer, at least not in the current Lightwave. I can't confirm this though because I don't have any experience with transferring morph data from FBX, only mocap animation data.

Maybe Silvio can answer this? Silvio, when you import the morph data from the FaceShift FBX, does the data translate to Morph Mixer or do you have to manually link or move the data to Morph Mixer? Thanks in advance for any info.

Maybe Brekel is talking about problems directly exporting to LightWave Morph Mixer from his software? Now that would be Lightwave specific. However I don't imagine this is a difficult format because other face animation and lipsyncing programs can export it (Magpie Pro, Mimic Pro for Lightwave, AnnoSoft Lipsync, TAFA, etc.,)--I think this is similar to the morph data that Brekel saves directly to the FBX.

Anyway, I obviously didn't get to test Brekel Face Pro today...maybe tomorrow.

FYI, FaceShift is $150 for a non-commercial license, which would be suitable for our current LGD productions. Naturally, if we ever decide to go commercial, we'd have to upgrade the license.

G.

Greenlaw
11-21-2012, 04:54 AM
Ah, okay...I looked at the docs again a little more closely this time. Apparently the software can output 'animation units', which sound like morph data but it doesn't embed this in its FBX export. However, some other formats supported by Brekel, like .PZ2 for Poser and DAZ, can include animation units for morphs. To do the same for Lightwave, I'm guessing he needs to add support for MorphMixer.

I'll try the software tomorrow to familiarize myself with it, and submit a Fogbugz feature request to NewTek. After that, if we're lucky, maybe somebody from NewTek will contact Brekel about this. :)

G.

Greenlaw
11-21-2012, 04:57 AM
Oh, it looks like you already submitted a fogbugz request. I'll submit my own in support of yours then. :)

geo_n
11-21-2012, 05:31 AM
Thanks Greenlaw. The more people interested the more chances newtek might do something.
I've asked a few people about morphmixer docs and no one seems to know if there is one. Sensei said there is none actually.
I've also suggested to Brekel to look at papagayo script from Mike Green that exports txt files from papagayo to morphmixer directly but he can't understand it I guess its too lightwave specific.
I'm sure newtek assisted in some way to devs from magpie, tafa, mimic to understand morphmixer so its entirely on newteks hands.

Greenlaw
11-21-2012, 12:25 PM
Yes, since LW 10.x, NewTek has proven they want to make Lightwave play nice with third-party tools (hardware and software)--if they weren't, I probably wouldn't be making these homebrew mocap productions with it. I'm sure if enough users requested support (via Fogbugz,) for these entry level face capture systems it will certainly get their attention.

G.

Greenlaw
11-21-2012, 12:48 PM
BTW, we installed Brekel Face Pro late last night and our Kinect for Windows arrived a few minutes ago. We decided to add the Windows version to our pipeline because it sounds like all of these programs can use Near Mode for higher resolution face capture, and Near Mode isn't available for the XBox version.

We also ordered a couple of PS3 Move devices last night to test the wrist and props tracking with iPi DMC--these should arrive Friday or Monday. iPi Recorder can currently record Wii Motion Plus or PS3 Move, and tracking in for these devices in iPi Studio is going to be available any day now. When we get around to testing, we'll compare the two. iPi Soft has already stated that the Move device is more accurate but I'd like to see the difference anyway.

As I said earlier, we don't need these features for current movie project but production is actually moving along quickly now and we're beginning to think about the R&D for our next film. In fact, I pretty sure I can make time for a little Face Pro dabbling this evening and will report what (little) I learn tonight. :)

G.

Greenlaw
11-21-2012, 06:59 PM
My six year old daughter and I played around with Brekel Pro Face this afternoon and I have to admit, it's pretty slick. All you do is plug in the Kinect and launch the software--the software then automatically finds a face and starts following it. Brekel Pro Face picked out my daughter's face first probably because she was more front and center and also because she's cuter than I am. But when she put her hand to her face, the system immediately switched to tracking my face. Pretty smart.

However, if we sat too close to each other, it would give up. My guess is that in this situation the merged 3D volume probably looks like a two headed beast to Brekel Pro Face, which just confuses the tracker.

The quality of the realtime track is a bit jittery, which is expected because Kinect data can be pretty crude--the raw first pass of iPi DMC data can look pretty jittery too. I imagine that at some stage there is a post process which smooths the motion, probably like how iPi DMC's Configurable Jitter Removal and Trajectory Filter can stabilize the actor and lock its feet. At a glance, I did see two Filter controls--one slider controls reduces the motion of the entire head and the other reduces the sensitivity of the face 'muscles', but I don't think this control is meant for jitter removal. Need to look at the manual again when I get the chance.

Playing with Brekel Pro Face was a lot of fun but that's as far as we got this afternoon. My daughter tried to get it to track a stuffed toy's face, which was an amusing idea but, not surprisingly, Brekel Pro Face didn't even try.

I haven't tried using my own geometry or anything yet--this test was simply to see the software run. To be honest, I'm not one hundred percent sure how I'm going to use this data yet but I'll try doing something a little more productive with it soon. When I have time, I'll also try the face capture in FaceShift, Nuicapture, and J|R (when it's available) and see how they compare.

G.