iPi Studio Desktop Motion Capture Tests

[Greenlaw]

Hi,

Here are some motion capture tests we did using the iPi Studio (http://www.ipisoft.com/) markerless motion capture system and four Playstation 3 Eye cameras.

Deform Test (http://www.youtube.com/user/LGDTestTube#p/a/u/0/G2KLtGsl-L0). This test was done earlier this week using the latest version of the software. The performance was designed to help test character deformations for the Brudders short film my wife and I are making. (Shown here is the iPi default character and rig, not a Brudders character. We'll show the Brudders rigs and models in a later video.)

What's new in the software is shoulder tracking and the ability to use six synchronized Playstation cameras. We're only using four cameras right now. We'll probably add two more cameras after we finish the film. Using the PS3 cameras we're able to get four synchronized streams of 640 x 480 video at 60 fps; fast enough to prevent motion blur issues.

There are a few small problems with this mocap test which are described in the YouTube text. A new issue is that the feet get unfixed briefly during the body twist. The feet seem to track well during the circle walk though. There is some jittering in the joints but the programmers are working on that. Also, the system currently doesn't track the head or wrists, but I'm told they're planning to add head tracking in a future update. (If they can do shoulders now, I'm sure the other stuff is coming soon.)

Sword Test (http://www.youtube.com/user/LGDTestTube#p/a/u/1/qo4bz4ucCM4). This is a test I did last July using an older build of the software. The purpose of this test was to see if I could use a prop while recording motions and iPi DMC seemed to see past the prop with no problems. This version did not track shoulders but it did a fairly decent job even without it.

By the way, the PS3 cameras are cheap: $30 each! By contrast, a single Kinect camera is about $150. But if you already have a Kinect, iPi intends to support it in the 'express' version of the software.

Okay, so what does any of this have to do with Lightwave? Well, we've been able to successfully retarget the iPi Studio mocap to Lightwave rigs and characters using Animeeple (http://www.animeeple.com/). We'll show some of the tests in a future video, and of course we'll eventually release the short film using this system.

If you have any questions, feel free to ask here in this thread. :)

Cheers!

Greenlaw

[Greenlaw]

I should have added that the developer told me you can import Lightwave rigged characters directly into iPi DMC via Collada, and have DMC retarget the data there. I haven't tried that myself yet but if anybody out there does it, let me know.

FYI, using Animeeple to get the iPi DMC mocap data into Lightwave has worked well for me but I had to purchase the program's FBX exporter to do it. I describe my current LW-to-Animeeple-to-LW workflow in the Animeeple thread elsewhere in these forums. It starts on page 25 here:

Animeeple Thread Page 25 (http://newtek.com/forums/showthread.php?t=101917&page=25)

Feel free to ask questions here or there, or at my blog Little Green Blog (http://littlegreendog.blogspot.com/).

G.

[rezman]

The results seem really, really good. No real jitter that I can see and the motion seems dead on. What is your overall impression...is it worth the cash for a production environment, or is a costlier solution in order?

[Greenlaw]

The results seem really, really good. No real jitter that I can see and the motion seems dead on. What is your overall impression...is it worth the cash for a production environment, or is a costlier solution in order?
Thanks! It's been a lot of work getting to this point. I first bought into iPi Studio about a year and a half ago when it was in its early beta stage. The system has come a long way since then, and the biggest improvements have come about in the last six months or so. It's also been fun; from the beginning, the developers have been very helpful and open to ideas from their users. So far, they've put in almost everything I've asked for, so I'm one very happy customer.

Now, to answer your question, is it worth it? It really depends on your expectations. If you dream of making 'Avatar 2' with this system, then obviously you're high. :)

When I first heard about iPi Studio, I had the crazy idea that it might be good enough to create weekly 'machinima-style' Brudders episodes with. And at the time, it was an insanely crazy idea. But these days I'm starting to think the idea might actually be doable. The short film my wife and I are working on is a 'proof of concept' for this idea. However, I have to admit, the project has grown into something more ambitious than what we originally intended. The final movie is going to be much bigger than just an animated episode of the comic strip.

Anyway, our total investment in the system has been relatively low. When I 'bought in' back in early 2009, iPi Software had a fantastic introductory offer, but we have also gone through a few different cameras and computers since then, so our 'savings' was offset by a lot of R&D and experimentation.

Nowadays, the system costs between $595 and $995 for the software and $120 (4 cameras) to $180 (6 cameras) for the Playstation 3 Eye cameras. Throw in a set of active USB repeater cables (about $10-$15 each,) and you have a motion capture system that costs between $760 and $1240 or so.

Well, to start off anyway. You'll also need a pretty decent desktop system (quad or six core) with a GTX 260 card or better. Other costs include clothing (basically a red or green long sleeve shirt, a black T-shirt, and jeans--all available at Old Navy,) a set of tripods, c-stands or security camera mounts, and a Helicoil kit for tapping the base of the cameras. FYI, I found a vendor in Texas that sells the perfect lightweight c-stands for about $12.50 each, and I purchased the Helicoil kit from Amazon for about $15. (Modifying the cameras with the Helicoil kit was easy. I made a video showing how to mod the cameras, which I'll post it in the near future. By the way, it's not necessary to mod the cameras, but it does make them more stable and a lot easier to mount.)

You'll also need an adequate shoot space. We have our garage set up for small green screen shoots, which seems to work well for this mocap system too. But in theory, you can use any space that doesn't have a lot of reflective surfaces; some users shoot in gymnasiums or auditoriums, some in their living room, others shoot outdoors on overcast days. At the iPi website, you can see examples shot in office spaces and community centers, and the results are pretty good.

Overall, the current iPi Studio is a good 'entry level' system for the total investment. It's still a little rough around the edges, but having 'worked' with these guys this past year, I have a lot of confidence in their ability to make it a great system. They're quite brilliant, and they're nice people too.

I guess the big question is, is iPi Studio good enough for studio production work? In its current state, I think it has potential for previs work, and in limited ways, you could use the data for actual production shots (i.e., background characters.) With a few more improvements, like selective Jitter Removal, head and wrist tracking, FBX support, and some better retargeting and motion editing tools, I think it could be ready for prime time. (At that point, it should probably be more expensive too. Um, I didn't just say that did I?)

But for my own personal projects, hell yeah, it's worth it. :)

Stay tuned; I'm going to keep posting my experiences with this system in this thread.

G.

[rezman]

Thanks for the great reply. It makes me think a little of 3D Coat...they started rather small and now have a very mature program.

The alternatives to this system are a whole heck of a lot more, so it really seems like a no-brainer.

[geo_n]

Looks like your further along your mini mocap studio, lightwave workflow.
Would be cool to see pics of your garage setup. The size, how and where the cameras are mounted, optimal angles, lighting, etc. Like moviemagic stuff. :D
The basic version has 4 camera support. Is it good enough and should just go the standard?

[Greenlaw]

Looks like your further along your mini mocap studio, lightwave workflow.
Would be cool to see pics of your garage setup. The size, how and where the cameras are mounted, optimal angles, lighting, etc. Like moviemagic stuff. :D
The basic version has 4 camera support. Is it good enough and should just go the standard?

I'll do that. We've been preparing a 'production log' that has info like that. For now, you can see a pic of the stage on our blog here:

Acting! (http://littlegreendog.blogspot.com/2010/11/acting.html)

The space looks HUGE in this pic...but it's not really. My useable performance space is only about 6 square feet. The miracle of wide angle lens. :)

I'm using four cameras, but today the developer told me that more cameras will reduce the jittering. I'm not sure my system can handle six cameras but I'm ordering a fifth camera tonight. If it works out well, I'll think about adding a sixth. I'm skeptical because I have a quad core system and they recommend a six core system for six cameras.

He also said that they're planning to add 'Configurable Jitter Removal' in mid-December. This feature will let you you apply Jitter Removal to selected joints. The way it works now, Jitter Removal is applied to the entire rig, which isn't always desireable. That could solve some issues for me because I've observed that jittering is more noticeable in some joints than others.

G.

[geo_n]

So greenbackground is some sort of matt cloth?
The mocap data I got from you before was good enough when I cleaned it in max. 4 cameras seem ok. Looks like this will be the way to do mocap soon rather than renting a mocap studio for a day.
Can IP capture more than one person? We did a project for a boxing anime project and used 100% mocap. Full body animation takes so much time to animate.

[Greenlaw]

So greenbackground is some sort of matt cloth?
The mocap data I got from you before was good enough when I cleaned it in max. 4 cameras seem ok. Looks like this will be the way to do mocap soon rather than renting a mocap studio for a day.
Can IP capture more than one person? We did a project for a boxing anime project and used 100% mocap. Full body animation takes so much time to animate.

The backdrop is a standard foam-back green screen. I don't remember who I bought it from but it came from a vendor on eBay. If you get a green screen, I highly recommend the foam-back type; it's less likely to wrinkle and it has a nice matte surface. My green screen in the pic is hanging from two standard photographer's backdrop stands.

Cowboy Studio (http://www.cowboystudio.com/), by the way, is an awesome store for 'mini stage' gear. I purchased my lightweight c-stands from them for about $12 each, and today I bought two more and found that the price had dropped to about $10. These stands are surprisingly high quality for the price and perfect for the PS3 cameras after modding the base. Other items I purchased from them previously: a tabletop kit for shooting small products which comes with display stands, two lights, and a small tripod, all for about $60; a second backdrop stand for $60. The quality of everything has been excellent for the price; I wouldn't call it 'heavy duty' but it's just the right weight for a small stage with a tiny crew (one to two people, in my case.) Cowboy Studio has their own web store, but do a search for their products on Amazon because sometimes they have better deals there. If I buy more stage lights for our studio, I will probably go with these guys.

The floor is covered with interlocking foam tiles I purchased from Wonder Mat (http://www.wondermat.com/). The tiles are designed for kids play areas, so the thick padding is nice for falling down on. The color is almost ideally suited for green screen; in real life it's a little off from my backdrop, but on video it looks the same. (Just like in the photo.) I have enough tiles to cover a 10 x 10 square foot area, but as you can see here I only have space for 8 x 8, of which only 6 x 6 is actually useable for the distance of my cameras. (I'll have to see if I can get permission to shoot at work sometime; we have much more space at the new facility.)

I should mention again that a green screen is NOT required for this mocap system. I think it may help get cleaner results, but the reason I have the green screen is because I also do green screen shoots. It's just convenient that it also works well for the mocap.

As for multiple actors, the answer is mostly no. Being a markerless sytem, the tracker needs to see as much of the performer as possible; a second person may cause too much occlusion for accurate tracking. In theory, the system may be able to track each actor in separate passes, but in practice I'm not sure you can really do this.

For our short film, we have three cg characters performing in sync. To do this, we use music and video playback on a laptop just off-screen, which the performer uses to time his actions to. By design, we don't have any direct interaction between the characters, but if interaction was required you may need to adjust the motions using the mocap editing tools in Animeeple or Motion Builder; both let you bring in multiple characters and apply an animation layer to the mocap data. (I don't own MB, so I can't answer any questions about it.) I've also been using LW's built-in FK-IK to edit mocap (easier now with the new joints options,) and lately I've been trying out IK Booster to edit the mocap.

G.

[robertoortiz]

We need this for a project here that I am pitching
Thanks for keeping us in the loop, this thread is pure gold.

-R

[Greenlaw]

Glad you're finding this interesting. I like writing about this project partly because I'm excited about it, but also because writing things down helps me remember them. :)

[DragonFist]

I'm very excited about this technology and your work to get a good workflow with lightwave and have been following your post on the subject.

Thanks for the thread! :thumbsup:

[erikals]

wonder have you tried 4 versus 6 cameras, is there a big difference?
i guess it must be?... as the price difference between Standard and Basic is so big...

never mind :]
http://www.ipisoft.com/forum/viewtopic.php?f=2&t=4799&sid=a409c6b135d62599d12d37d3b8a641bc
http://www.ipisoft.com/forum/viewtopic.php?f=7&t=14&sid=a409c6b135d62599d12d37d3b8a641bc

 

[Greenlaw]

Cameras 5 and 6 arrived last Saturday and two new c-stands from Cowboy Studios arrived today. I'll probably get around to testing them later this week. I suspect my system may be able to handle five cameras at 60 fps, but I'll probably have to reduce the frame rate for six cameras.

The advantage of having six cameras, I'm told, is higher accuracy and reduced jitter. I'll let you know to what degree later this week.

Hope this works with my current setup. I think the only meaningful upgrade I can do on this computer is replacing the SATA 3 with an SSD, but at this stage I really don't want to spend more money on this film, I just want to get it done. But if it doesn't work, then four cameras it is and that's that. :)

G.

[erikals]

maybe consider a 10,000 rpm HDD... ?

[Greenlaw]

Had a breakthrough today with using IK Booster to edit my iPi Studio mocap data on a grossly cartoon proportioned character. The tool is actually very easy to use once I got the hang of it, and the more I use it the better it seems to work for me. It was my hope that the combo of Animeeple and IK Booster would help me avoid purchasing MotionBuilder, and now it looks like this might actually work.

So what's useful about it? I'm finding it very easy to offset specific bone chains globally throughout an animation or for a specific range of frame. This is going to be helpful in getting the Brudders characters to hold their musical instruments properly while preserving the original mocap animation.

Now I'm working on stabilizing the feet and legs with IK Booster. I had pretty good luck with this using the traditional LW methods (IK and Goal Heading,) but I'm hoping IK Booster will reduce the set up time and simplify things further.

(I will explain all in detail when I get back to that tutorial I promised after this project is done.)

BTW, I found Splinegod's Importing and Using MoCap Data in LightWave 3D webinar videos very helpful in figuring this stuff out. If you don't have MotionBuilder or don't want to use it, these videos are well worth viewing. While I don't agree with all of the techniques presented, the information contained explained so many Lightwave mocap mysteries for me.

G.

[erikals]

nice, very interesting to read about the process, keep us updated, and hopefully, hopefully, you'll put up a DVD for sale in the future... :]

[Greenlaw]

Wow! The iPi programmers just added Kinect support to iPi Studio:

iPi Desktop Motion Capture: Kinect Test 2 (http://www.youtube.com/watch?v=GCu8KTrC4sc)

Those guys are amazing.

Back to my project, I've been trying to get things going with LW 10's FBX the past couple of weeks, but it's looking like my mocap workflow was actually working better under LW 9.6.1. There are a number of things about LW 10's FBX support that are improvements but one of the fixes is messing us up. I'd go into more detail but I still have to test a few things to be sure. In the meantime, it's back to 9.6.1 for us...at least for the FBX mocap import and transfer to the final rig.

G.

[Greenlaw]

Lots of updates in the past two days.

First, we added an SSD to the 'mocap computer' which should allow us to capture from six PS3 cameras even with a quad core system. Official iPi specs say you need a six core system for six cameras but a user at the iPi forums says the SSD makes up the difference on his quad core as far as bandwidth goes. I'll be testing with six cameras in a day or two.

And today iPi Software released version 1.1.0.112 of iPi DMC with these features:

* Improved accuracy. Fixed bugs in jitter removal algorithm.
* Configurable jitter removal options for body parts.
* New export option for MotionBuilder-friendly BVH.

Just what I've been waiting for. :)

I've been having trouble getting Animeeple and LW 10 to play nice but Cageman, Rebel Hill and other artists in the Hard CORE forums helped me figure out that there is a bug with Animeeple's FBX format. (Motion Builder FBX of the same data transfers to native LW rigs without nearly as many workarounds.) Of course I'll send the data to Animeeple and hopefully they can get this problem solved.

G.

[erikals]

it probably has many similarities to grabbing video, so yeah, a fast HDD or SSD should help alot i think :]

very curious about the new jitter removal... \:]

[Greenlaw]

it probably has many similarities to grabbing video, so yeah, a fast HDD or SSD should help alot i think :]

very curious about the new jitter removal... \:]

Using Build 112 right now with the iPi test file I posted last month. Should give a good comparison. The performance is about 2500 frames long, and it looks like it's going to take one more hour to complete a full re-track of the video. (About two hours total.)

A couple of months ago I thought this was fast but now I already want to upgrade my video card. The hunger never ends. :)

I'll do a similar session using six cameras for another comparison later this week. The iPi devs tell me that six cameras should reduce jitters even more.

G.

[erikals]

it's a good thing you can't rent CPU power, i'd be broke (or even more broke...) :]

hm, looks like SLI is not supported yet...?

[Greenlaw]

I don't think it does yet. But I'm using a GTX 260, and guys who are using GTX 480 with iPi DMC are reporting significantly faster tracking speed. (2X and better!) It's tempting but I don't think I can justify upgrading the video card in this computer yet because it's fairly new and for everything else I do here, it's actually quite good. Also, the Brudders film is an experimental non-commercial project that's already gone way over its budget and schedule.

Maybe we'll upgrade when we come up with a project that can pay for itself. (Okay, that's really my wife talking. And you're absolutely right dear.) :)

G.

[erikals]

maybe do what i did, buy a used GTX card... :]

LOL,... hehe, you have a good wife http://erikalstad.com/backup/anims.php_files/smile.gif

[Greenlaw]

I just finished running two tests using the new Configurable Jitter Removal in iPi DMC Build 112. I was wondering how they would implement this feature and their solution is very nice. I've attached a screen cap of the new panel. I think how it's meant to be used is fairly self-explanatory.

Anyway, my first test using 'best guess' Jitter Removal settings looked pretty good, and the second test using tweaked settings was nearly perfect. I think my second settings may be just slightly aggressive on the arms but at least I now have the option to back off or increase the process exactly where I need it.

I want to take some time finding the best settings for this performance, and then I'll post a comparison video.

Of course the real test will be when I apply this tool to the actual Brudders perfomance. The last build's Jitter Removal hammered out too much detail from the performances, so I'm very eager to see how much better this new build will handle these motions.

G.

[UnCommonGrafx]

I'm curious...
How accurate can you make these systems?
I'm doing ASL and would like to capture down to finger motion. I need to look at a more complex system? Any recommendations?
Appreciate it.

[erikals]

for fingers i've currently come to that hand animating them using a pose saver is the fastest /best way to do it.
http://www.jacobobarreiro.com/jweb/scripts/

for walks etc i would use iPi

not sure about the feet, maybe redo the animation for the feet manually...
as the iPi data looks a bit inaccurate...

 

[UnCommonGrafx]

It's more about capturing the language on the body as someone signs than it is for me or us to create animation, of which we are slowly beginning to do.
Optitrack, cyber glove... more things to look at. Thanks.

[erikals]

http://www.acceleglove.com/
http://www.5dt.com/hardware.html#glove

looks very good compared to the price, here is what i wonder though,
can it record the movement of fingers spreading rolling the thumb?

if not... is it worth the effort if it can't?

 

[Greenlaw]

Hi Robert,

iPi Studio currently doesn't track hands, not even wrist rotations. I know that head tracking is coming soon (they recently implemented shoulder tracking which looks pretty good; also, one iPi user recently figured out how to use iPi DMC to quickly rotocap the head right into the iPi mocap,) and maybe they'll add wrist tracking, but fingers...that's very complicated and I wouldn't expect it anytime soon.

That said, I'm sure they could configure their system to track fingers as a separate pass that could be merged with the body tracking. I've asked about this in the past but right now I think they're more focused on perfecting their body capture system.

So the short answer is what others have already described here: use a specific 'hand capture' device, or create a library of hand poses that you can mix using Motion Mixer or one of the other pose saving/blending plug-ins, or if you're 'old school' you can put the hand poses in sub-zero frames and use Create Key At Current Item And Decendents on the wrist joints/bones.

G.

[UnCommonGrafx]

This conversation is MOST appreciated. Thanks.

[Greenlaw]

Hi Robert,

I just read your post again and see that I misunderstood what you were asking. For body capture, I think iPi Studio has taken a big leap with this latest build. The precision seems to be quite a bit better even when tracking only four PS3 cameras.

The issue with four cameras, however, is that it can introduce a bit of jitter to the body and limbs, which is why the new Configurable Jitter Removal is so important. That said, the drawback with CJR is that if you use it too aggressively, you may lose some of that precision. To be clear, the motion will still look good and be reasonably accurate, just not as accurate to the original performance because of the smoothing involved.

With six cameras, the capture quality is supposed to get much more precise, and with more data to track, the jitters should be greatly reduced. I haven't used six camera yet, but I'm planning to do a six camera session this weekend. I'll post comparison results next week.

@Erikalis. Yeah, in the raw capure video the feet are especially off when I make that full body twisting motion. I think this is because iPi does not twist any bones through the thighs, so sideways rotation can get over-constrained in certain poses. In other words, the position of the feet seems accurate but the rotation sometimes gets constrained by the ik system. The developers say they limited the rotations in the legs to improve overall rig stability during tracking. They're looking into improving the tracking system for the legs but I got the impression that it was going to take some time to figure out.

FWIW, when you don't see the performance video superimposed over the rigged actor, you don't really notice the slight feet misalignment during the circle walk segment, and where you do see it, it should be easy to fix (use a rig that allows twisting through the thighs or use Lightwave's Joint Twist at the knee.)

I haven't tried fixing the feet during the full body twist but I don't think this would be difficult either, even if you use Lightwave to fix this (meaning, without Motion Builder.) I'll try to spend a little time editing these problem areas just to be sure of what's involved.

G.

[erikals]

this plugin might be interesting for locking feet to the ground...
http://www.youtube.com/watch?v=yWJzkyWpQgA

 

[erikals]

Originally Posted by Megalodon
http://www.acceleglove.com/
http://www.5dt.com/hardware.html#glove

looks very good compared to the price, here is what i wonder though,
can it record the movement of fingers spreading rolling the thumb?

if not... is it worth the effort if it can't?
http://www.newtek.com/forums/attachment.php?attachmentid=91563&d=1294284001

 

umf,... sent them a mail, they never got back to me :[
guess it can't be done /is too advanced...

[erikals]

ok, just a bit late...

The gloves come in two variants. The 5-sensor glove has sensors that only measure the flexion (curving) of the 5 fingers. The 14-sensor glove measures the flexure at 2 points for each finger as well as the abduction (spreading) of the fingers.


The rolling of the thumb (also called opposition) is something that we unfortunately do not measure.

...so, no rolling thumb, and prices for 14 sensors is quite high...

 

[Greenlaw]

For anybody using the latest iPi Studio (build 112), here's a good Configurable Jitter Removal tip from Michael, one of the developers.

I was using the new Configurable Jitter Removal and I found myself wanting to use different levels of CJR at different points in the Take. Unfortunately, I was seeing a 'bump' in the mocap every where I changed the CJR settings. I wasn't sure if this was a bug or if it meant I could only use one CJR setting throughout a Take, so I asked.

Michael got back to me immediately. He essentially advised that if I saw a 'bump', I should set the Trajectory Filter to 0 and then set it back to 1 (or whatever setting I was using.) This should immediately smooth out any bumps in between the different CJR applications. He also told me that they will try to automate this workaround in the next release.

G.

[erikals]

got a bit info from AcceleGlove,...

Thanks for your e-mail. The AcceleGlove can track position of fingers (relative to the hand) for all movements except those that are purely in a plane parallel to the ground.

Take a look at the User Manual, Section 4 (How the AcceleGlove Works). It talks a bit about how the sensors work.
http://acceleglove.com/AcceleGloveUserGuide.pdf

so, maybe...

[UnCommonGrafx]

Greenlaw,
I get to buy one of these systems. You wouldn't happen to have a cost sheet or anything all together as to the price and purchase locations?
This is gonna be fun.

What is the other software I need other than LW to work with this?

[Greenlaw]

Hi Robert,

Sorry for responding late. I'll post a total cost breakdown here later tonight. If you need some info now, this subject was briefly discussed here a few days ago:

http://newtek.com/forums/showthread.php?t=114123&highlight=ipi+cost&page=4

Scroll down to post #49. I think I covered all the 'accessories' you will need, but I'll look again to see if I missed anything. (Actually, I just thought of a few things.)

G.

[Greenlaw]

BTW, sorry for not posting much info lately. Had to put the project on hold because of work. Hope to get back to it soon though.

In the meantime, I've been working on a new section for the website which will include the production log for this project (finally!) It should go online this weekend.

G.

[Greenlaw]

The following info might come in handy to other iPi to LW users.

In recent weeks I've been learning Motion Builder 2011 for the project, and yesterday I started using my own iPi mocap data in that program.

The current release of iPi DMC has a Motion Builder .bvh export option which saves out a rig with MB-friendly hierarchy and joint names. The height of the MB .bvh rig exported from iPi DMC seems to be around 1.7m (which is based on my own height plus a little extra.) This height imports correctly into Animeeple (and probably Lightwave.) However, if you import this file into Motion Builder, the rig is very tiny...so small, in fact, that you cannot even see it. (All you see is a cluster of nulls at zero.)

I tried importing the data to bvhacker (http://www.bvhacker.com/), a free .bvh editing tool, and even though it reports the height as 1.7 (meters I presume,) the rig looks very tiny in that program too. Fortunately, it's trivial to scale the rig up in bvhacker. Just press Ctrl+ + to scale the rig by a factor of 2 and re-save file. For the purpose of using the .bvh file as a reference for Motion Builder, it looks like using the scale up command five times gives you a .bvh reference rig that comes up to the mid-thigh to an average adult character in MB, which seems typical based on various tutorial MB videos I've watched.

If you use the .bvh in Animeeple, just leave the scale alone. (Animeeple allows you to set the correct import scale factor for .bvh files and remembers that setting from session to session.)

I put in a request to iPi Software to allow the user to set the scale factor when exporting a .bvh file. If they implement this feature, it will eliminate to the scaling step in bvhacker required for MB.

G.

[Greenlaw]

Robert, here's the cost breakdown for the iPi Studio system:

iPi Studio: $595 for Basic (4 cameras) and $995 for Standard (6 cameras). (Cameras not included.)

PS3 Eye camera: $30 to $40. I paid $30 on Amazon as recently as last November (2010), but I noticed that the price went up to $40 during the holiday season. Perhaps the price will come down again.

16 ft USB Repeater cable: $11 each from Newegg. These are active cables, meaning there is a box at the end that extends the USB signal. You can supposedly daisy-chain up to five of these cables and still have a good signal. I have at least one cable for each camera, and the longest I've needed is a daisy-chain of two cables for the farthest cameras.

Helicoil kit: $18. This is a kit that allows you to tap the base of the cameras so you can mount them to a c-stand or tripod. The metal PS3 base already has a hole in it, so drilling it slightly larger to use this kit is easy. Alternatively, you can use Velcro, but screw mounting it obviously more stable. I purchased my Helicoil kit from Amazon but I think you can buy this kit at camera or auto parts store.

c-stands: $12 from Cowboy Studios. This product is surprisingly high quality for the price. There is no head for panning and tilting like with a tripod, but you don't need that because the PS3 camera base rotates and tilts. Cowboy Studios has a store on Amazon; sometimes the prices for their products are lower there than at their own website.

Finally, you will need appropriate clothing. The iPi Studio may be a markerless system but there are some wardrobe considerations.

You'll want blue jeans ($30 or so.) They don't have to be skin tight but they should not be baggy either. I would get something that fits closely but gives you plenty of freedom to move.

You'll also need a long-sleeved shirt, red or green color recommended, and a black short sleeved T-shirt. This combination helps iPi DMC track your arms, and the black shirt reduces cast shadow on your torso that can confuse the tracker. I bought these from Old Navy at a local mall for $10 or so each.

iPi recommends dark shoes to reduce shadow interference. That said, I've been wearing white socks. The socks are bright enough under my stage lights that shadows are not an issue. I'm not 100% sure there is any advantage in wearing white socks though. I should test this next time.

Of course you'll need a computer to record and track the mocap data. The computer should have an available USB controller for every two cameras. Be advised that multiple USB ports can share a controller, so you want to be sure you actually have enough controllers for the number of cameras you intend to use. I understand that most modern computers have at two onboard controllers. If you need more controllers, you will have to add USB expansion cards. It's recommended that you get a USB 3.0 controller card; these are backwards compatible with USB 2.0, plus forward compatible with faster USB 3.0 drives. (BTW, some USB 3.0 controller cards also work as SATA 6GB cards.) Prices vary, and one of my addition USB 3.0 cards came with the purchase of a Western Digital USB 3.0 drive. (About $120 for 1TB drive and USB 3.0 adapter card.)

If you intend to use six cameras you will probably want a dedicated SSD drive. These are finally coming down in price. A Corsair Force 60GB SSD is $129 at Newegg; I bought the 120GB model for about $230 recently. These drives are not as fast as rated but they are still wicked fast, and fast enough to capture video streams from six PS3 cameras at 640 x 480 and 60fps. iPi suggests a six core computer for six cameras, but I'm finding that an i7 quad core works fine if you have an SSD for video capture. (BTW, a fast SATA drive (7200 rpm) is fine for capturing full res and speed from four cameras.)

Finally, the most important item for tracking is the video card. You'll want a Nvidia Geforce GTX 260 or better for good results. With a GTX 260, I'm able to track at about 2 seconds per frame, but users with a GTX 480 card are reporting multiple frames per second!

Well, that should cover what I consider the 'essentials'. Since your own package requirement may vary, I'll leave the final math to you.

Hope this helps.

G.

[UnCommonGrafx]

You rock!
(Almost missed this!)
Thanks for the info.

[3dWannabe]

I use this for mounting the PS3 camera to a tripod. Cardboard strips combined with clear 3" packing tape is used to position the camera's up/down and hold this position firmly.

cost $8.50 each plus shipping.

http://www.bhphotovideo.com/c/search?Ntt=SMMSC&N=0&InitialSearch=yes

[antsj]

Hi.

I have been tracking the conversations on this thread and learning alot.

On another site there was a posting of sale @ Daz.com I have vopied in the info to help you go to the exact page and not have to move throught the site to finding items:


iPi Desktop MoCap Standard Edition
http://www.daz3d.com/i/shop/itemdetails/?item=11817 (http://www.daz3d.com/i/shop/itemdetails/?item=11817) Sale Price: $682.50 Save 30% through 01/27/2011
iPi Desktop MoCap Basic Edition http://www.daz3d.com/i/3d-models/-/ipi-desktop-mocap?item=11816&spmeta=rl&_m=d
http://www.daz3d.com/i/3d-models/-/ipi-desktop-mocap?item=11816&spmeta=rl&_m=d
Sale Price: $402.50 Save 30% through 01/27/2011


Hope this info helps others looking to buy before Jan 27, 2011.

aj

[Gillissie]

FYI - I started a thread that details the steps required to get BVH data generated from iPi into LW 10 on a character.

http://www.newtek.com/forums/showthread.php?t=116664

Note, this is from my own experimentation, so if there is an easier or better way to do this, please comment.

[Greenlaw]

Good news today for iPi users: I ran the auto update for iPi DMC today and found that I now have the much aniticipated Kinect support. Unfortunately, I don't own a Kinect so I can't test this out, but I'm tempted to get one. I don't actually need a Kinect because six PS3 Eye cameras will actually give you much higher quality mocap data, but the Kinect setup takes less space and should be easier to set up, making it ideal for a smallish living room or apartment space. (Or for those nights when it's too late and too cold to lug my computer out to 'the stage'.)

This week saw another big upgrade for me: I upgraded my GTX 260 to a GTX 460. Normally I wouldn't have bothered because the 260 is an excellent card and I had no complaints about it. Even intensive tasks like voxel sculpting in 3D Coat is fully interactive with this card. However, I read that the 460 can process iPi motion capture data 4 times faster or better.

I just did my first test, and it's absolutely true! The mocap test at the beginning of this thread runs about 1500 frames and took about 2 hours to process before; with the new card, the same motion now takes about 25 minutes! I'm stunned.

Not bad for the price. Last year the GTX 4x series cost about $500, but the GTX 460 I just bought cost $170 after rebate. The model I got is this one:

MSI N460GTX CYCLONE 1GD5/OC GeForce GTX 460 (Fermi) 1GB 256-bit GDDR5 PCI Express 2.0 x16 HDCP Ready SLI Support Video Card (http://www.newegg.com/Product/Product.aspx?Item=N82E16814127510)

I chose this card because it's smaller, quieter, and requires less power than the other cards. (This card requires a 450W power supply and my system has a 460W power supply; many of the other GTX 4x cards require a min of 500W.) It does have less memory than my 260 did, but I think this card's 1GB will be fine for what I do here.

Tomorrow or Monday I will do my first six camera session for the Brudders movie. Will post another update then. Stay tuned!

G.

[Greenlaw]

Hmm. Build 113's tracking is faster but also less accurate in the arms. Reported the problem with examples and iPi Soft is looking into it. Hopefully a new build is coming soon. In the meantime, back to trusty old Build 112. :)

BTW, I was told that the Kinect enabled Recorder should be out any day now.

G.

[erikals]

ps, sort of related, for cleaning up frames in LW, i found this "Keystrainer" plugin interesting,
might be of use,...
http://www.newtek.com/forums/showthread.php?t=55154

 

[Greenlaw]

Hmm. Build 113's tracking is faster but also less accurate in the arms.
Just ran a new track in 113 using an 'actual size' Actor rig into the project. (iPi Soft previously recommended a rig scaled slightly taller than actual size to improve stability.) This time the new results look very accurate in the arms and overall jitter is greatly reduced. The new results compare favorably to 112, maybe even better.

So all is apparently good with 113. Waiting to hear from iPi since they mentioned they were able to reproduce my errors. Hopefully, making sure you set proper actor scale now is all it takes.

Running the Configurable Jitter Removal and will compare results soon. I'm expecting the mocap to be even closer to 'spot on' now.

Looking forward to the six-camera shoot this afternoon. Stay tuned.

G.

Erikals: thanks for the tip. Keystrainer sounds useful. I'll check it out.

[Greenlaw]

A few people have asked to see an example of an iPi Studio calibration video shot on our stage so here it is:

iPi Studio Calibration Video (http://www.youtube.com/user/LGDTestTube?feature=mhum)

The video first shows the data in RAW Bayer format, and then switches to the final result in iPi DMC. Some users may be surprised by the simple 'performance' but this action gives us accurate calibration results every time.

What isn't shown in the video are the steps for using the calibration video in iPi DMC but we don't do anything out of the ordinary. Basically, we set the BG segment to about 30 frames of the clean video at the footage head, and then set the ROI (Region of Interest) from the frame when the LED is first touched to the floor for the first corner of the 'cage' to last frame when the LED touches to floor for the tip of the 'spiral' motion. Then in DMC, under the Calibration tab, we make sure to enable Auto Detect Initial Camera Positions and set the FOV. (For PS3 Eye cameras, use the default setting of 75 degrees.) Setting the lighting angle is optional for calibration but we set it to 'head on' to match our stage lighting anyway; we think it might help a little for the difference matte. To define a ground plane, we just choose the five contact points of the 'cage' (you need a minimum of three ground points.) Next we click the Calibrate button and wait for the process to complete. With a modest gaming card like the Nvidia Geforce GTX 220, it may take about 10 minutes. We recently upgraded to a GTX 460 so I'll post a time for that soon.

When the processing is done there is one step left to complete the calibration: set the known height of one camera. We usually use camera #1 but any camera will do. Remember to use the Ground Height Fine-tuning slider for extra precision. Finally, save your scene under the Scene tab. Save your Project file too just in case you need to run the calibration again.

Now it's time to track some performances from this session. Just load an action video and the Calibration scene file you just saved and you're ready to go.

Please feel free to ask any questions about the process here.

Greenlaw

[Greenlaw]

By the way, we ordered a Kinect Sensor camera. I'm sure our current six camera PS3 Eye setup is more accurate and flexible than a single Kinect but it occured to me that nearly all of the actions fo we captured for act I of 'Don't Fear the Pooper' probably could have been done using the Kinect system. We still need mocap data for this rest of the short film so it's a good time for us to test the Kinect version.

I'll write more about it after it arrives.

G.

[Greenlaw]

Ah! The Kinect Sensor arrived today! Hope I can find time to do some tests soon.

G.

[erikals]

sounds like mucho fun...! :]

[geo_n]

Ah! The Kinect Sensor arrived today! Hope I can find time to do some tests soon.

G.

Cool!
Hopefully its good enough for upperbody. The only space I have is in front of my desktop. :D So this will be a simpler setup? PC and kinect sensor only.

[erikals]

yup! :]

hope the price is nice too :]

[Greenlaw]

Sigh! I think I'm too tired to try setting up the Kinect tonight. Hopefully I can do it tomorrow evening.

I do have good news to report: Michael (one of the iPi developers) informed me in the iPi forums that as soon as they finalize the Kinect feature they will be working on head tracking. Their target for finalizing Kinect support is sometime in the next month.

After that, they want to work on getting iPi Studio to work with two Kinects at the same time. Very interesting!

G.

[Greenlaw]

I didn't know when I'd have time at home to try it out so I brought my gear into work today. (Thankfully mocap tech is actually relevant to my job!)

I'm using my old Dell e1705 Dual Core Laptop with an aging Nvidia card, running XP Professional x32. The laptop has an external eSATA drive for video capture. (It also has an external eSATA RAID but I forgot to bring that in.) FYI, this is the computer I did my very first capture tests with a year or two ago.

I installed the latest version of Recorder, plugged in the Kinect, waited a few minutes for Windows to install all the necessary drivers. Then I launched Recorder and selected the Kinect as an input device. And that was it...I immediately see a depth video in realtime.

The video res is 640 x 480, but the frame rate seems to be limited to 30fps. Is this the actual limit or is this just my system?

I'm amazed by how little space is required to do a full body capture. This will be perfect in our living room at home.

Anyway, I haven't done any actual capture yet but maybe after lunch and after I get a chance to go through the docs. I will try to track the results at home tonight.

G.

[3dWannabe]

After replacing the Logitech cameras with the PlayStation Eye's and then upgrading to 6 .... I'm almost hoping your test goes badly. Ahhhh!

But, if it's a better solution for working in small areas, that would be worth the money.

Thanks for testing this out!!!

I just hope he'll get head motion working soon, as for me that would really bring the mocap to life.

[Greenlaw]

Just got back from lunch and skimmed the docs. Using the system seems pretty simple. Unlike the full PS3 Eye cam setup, you don't need to worry about lighting or the color of your clothes. Oh, and 30 fps is apparently the normal rate. I'm still not sure if this is a Kinect limit or the current built-in iPi Studio limit for Kinect.

BTW, there's supposed to be an update soon with new drivers so these details are bound to change. But for a first beta release, it seems like reasonably solid system.

Bad news: I forgot to bring in an eSATA cable so I can't actually capture a performance yet. I can connect my video drive by USB but it's just too slow for this data. Because setup is so easy, I'll continue this test later tonight at home.

Hopefully my eSATA will be adequate for the laptop. I also have an eSATA RAID which will probably be fast enough if the standard 7200 rpm eSATA drive doesn't cut it. The best option most likely is an SSD, which thankfully have come down in price a lot. On my workstation at home I'm using an 'inexpensive' 120GB SSD for capture. The drive cost about $175, but you can get a 40GB version for a lot cheaper and 40GB will actually hold a lot of iPi video data, at least from the PS3 cameras. When you're done with a session, just off-load the video to a regular hard drive.

If the laptop isn't fast enough, I'll switch to my workstation at home. Since the software is already installed there, I should be able to just plug in the Kinect and go. The cable should be plenty long enough to go from our computer room to the living room.

More later.

G.

[Greenlaw]

Sigh! Sorry, I still haven't completed a real test for the system. I brought the camera back into work though and installed the new software on the laptop; also remembered to bring in the eSATA cable, a standard eSATA drive and a miniature eSATA RAID. The new drivers are from Primesense, the company that developed the Kinect for Microsoft, so I guess these are the official Kinect drivers as opposed to the open source ones. Very interesting.

I'll try to squeeze in some tests today but my real work will have to get done first of course. If I can find the time, I'll test both drives with the laptop. If the bandwidth is no good with the laptop and these drives, then I'll try running the capture from my workstation.

Stay tuned. :)

G.

[geo_n]

Greenlaw do you think its possible to run ipisoft with kinect on my hardware decently?
q6600 quad with 8gig ram and nvidia 8800gts. A bit old hardware I know.

[Greenlaw]

Maybe. If I have time, I'll try my workstation here at work later. It's a Xeon quad with a GT 220 and XP Pro x64. It's a good machine but it's fairly 'old' too. I'm not sure what the specs for the GT 220 are but I think the card may be average by today's standards.

I did manage to do a quick bandwidth test for the laptop. With the eSATA mini-RAID, the system seems to be able to capture just fine from the Kinect but with the non-RAID eSATA drive it could only capture for about 15 seconds before choking. This drive has a lot of data on it already and it's not optimized so that may be part of the issue. I'll try another test with a cleaner drive later.

A little info about the drives:

The eSATA RAID is two notebook drives in a small enclosure that I put together a few years ago. Nothing fancy. Hope to be able do some real capture tests this afternoon.

The other eSATA drive is an old 7200 rpm Seagate external drive. I've used this drive with the laptop to capture some early tests with the four PS3 cameras but that might have been 320 x 240 at 30fps. (That was quite a while ago.)

BTW, these are not new SATA 3 (6GB) drives, just 'old' SATA 2.

Anyway, it seems like even this old duo core laptop (Dell e1705) can capture from a Kinect so long as the capture drive is fast enough.

G.

[Greenlaw]

Finally got to shoot a motion test with the Kinect today at work. When I got home I tried tracking it. The first video failed but the second one tracked surprisingly well. I think the first one failed because my feet were partially outside the bottom edge of the video.

There was one error in the results of the second test caused by one arm occluding the other. I think this will be easily correctable in iPi Studio. But first we must eat dinner and watch the latest episode of Naruto Shippuuden. (Thank god the fillers are finally over!) :D

More later.

G.

[Greenlaw]

Earlier I described a mini-RAID I use with the laptop. This was a drive I built a while back to help me edit HD footage with the laptop. The RAID works fine for capturing mocap data with the Kinect but these days I would suggest getting an SSD instead. SSDs are actually affordable now, cheaper than building a RAID, and you will get better performance too. I have an SSD in my workstation and it works great with our multi-camera system, and it definitely outperforms the RAID.

Anyway back to the Kinect testing. I tried correcting the occluded arm. I could fix some of it in iPi Studio but it's very tricky. Finally, I gave up and felt the error could be more easily fixed in Motion Builder and probably using Animeeple or IK Boost in LW. However, it would be even easier to reshoot the motion in a manner that makes it easier to track.

I think that's the beauty of the Kinect system. Yes, compared to the PS3 Eye system, the Kinect is clearly very limited in the types of motions you can capture with it, but much easier to use and reshooting a motion is simply a matter of launching the Recorder app and doing another performing another take. Can't get much easier than that!

I'll try to post some results later this weekend.

G.

[Greenlaw]

I finally got around to using iPi Desktop Motion Capture Build 114 today. So far this is an excellent update! The tracking results are superior to what I was seeing in previous builds. More importantly, I'm not seeing any of the errors I reported with the last one. Also, the 'pre-jitter removal' jitters have been greatly reduced and tracking speed is blazing fast. (I'm using a GTX 460 and iPi DMC reports that I'm averaging about 0.67 second per frame!)

On closer examination of the new Deform Test track, I found three pops in the rig. Fixing the pops was unbelievably easy though: scrub a few frames past the pop, track backwards through the pop (maybe 10-20 frames,) set the Trajectory Filtering slider to 0 and then back to the setting you want, pop disappears!

One surprise was that the arms folding bit looks quite good even without wrist motion. I expected this action to be fairly difficult for this system.

Needless to say, I'm very pleased with this build. Still no head tracking in iPi DMC but this feature is apparently coming very soon (after the final release of the Kinect feature.) If the head tracking works well, I may consider re-tracking all the capture footage.

I haven't tried the latest iPi Recorder with the six PS3 cameras yet, but will do so soon. I will also do some more tests using Kinect, maybe tomorrow.

Last week we solved some rigging/weighting problems in Motion Builder using Maya, which makes is much easier to edit the data now. After editing in MB, the mocap data for all is coming into Lightwave 10 perfectly now. I finally, I feel we can move ahead and finish this thing!

I'll try to post a comparison video of the Deform Test tonight or tomorrow night.

G.

[erikals]

hm, 0.67 per frame, not bad... thought it'd be more... :]

[Greenlaw]

hm, 0.67 per frame, not bad... thought it'd be more... :]
Yes, 0.67 second per frame is a big speed boost for me!

Part of the speed improvement comes from the new graphics card (the Geforce GTX 460 replaces our GTX 250 in this computer) but additional performance comes from tracking optimizations in the last two builds. To compare: using the GTX 250 I was averaging about 2.0 to 3.0 seconds per frame, and on my old computer using a GeForce 9600 GS it was more like 4.0 to 6.0 seconds per frame. At the time I thought that was fast. Two years ago using the original beta release of iPi DMC, I think it took me about 10.0 to 15.0 seconds to track a frame. (Back then, the software used the CPU rather than the GPU for processing.)

Right now with build 114 I'm working my way through re-tracking all of our mocap files for 'Pooper', and so far every one of them is looking so much better! And it's been fast enough that the batch processor I've been asking for might not seem necessary anymore. (I'd still like to have this feature though.)

BTW, it's a good time to buy a new graphics card. The GTX 460 cost about $500 last year but I found it recently for $170 after rebate. I haven't tried it with 3D-Coat yet but I imagine it's going to be wonderful for voxel sculpting.

G.

[Greenlaw]

A few days I ago, I asked my 4-year old to be a guinea pig in a Kinect test. The results were mostly a fail but pretty hilarious. I think she was standing too far from the Kinect when she did her 'kitty' dance because the tracker kept losing track of her arms but what surprised me was that it seemed to track her running towards the camera fairly well.

I'll try a re-track of this Kinect test using build 114 tomorrow, and see if it picks up the arms any better. If the test results are good, I'll post a video.

Part of the problem was her height: when I try to scale the iPi rig down to kid size, it doesn't quite get small enough. A workaround might be to position the Kinect a lot closer and trick the Kinect into thinking she's adult size. Proportions will be different but I think the iPi DMC could still do it.

G.

[cresshead]

what's your thoughts on the kinect version...could i get stuff done with it regards typical conversation type motions/gestures and people sitting as well as standing, i'd be outputting to biped bvh and would data thin the mocap on import, then exctract footsteps etc over in 3dsmax.

i'm guessing with the small capture volume walking and major action moves will not be supported but i have a capture lib for that...i'm hoping i can use my own mo cap for coversation scenes mainly.

[Greenlaw]

Hi Cresshead,

I need to capture and track a few 'action' shots that should be ideal for the Kinect. I'm planning to record in our living room this weekend which might be just big enough for Kinect's maximum capture space, so I should have a complete answer for you soon.

In general, I think the Kinect works best for motions that mainly involve 'z-moves' and it's possible to create bigger motions if used in clever ways.

For example, if you need to record a long walk, record your walk from as far back as possible and walk towards the camera. You can use programs like Motion Builder or Animeeple to loop the walk and modify it's path. Then, you can record separate upper body performances and target this data to the upper heirarchy to make the motion less 'loopy'. (Be advised, this is just theory...I haven't done this with Kinect footage yet.) I guess you could do some of the same things editing .bvh in motion mixer too.

One 'action' shot I'm planning for Kinect is the shot where Sgt leaps on a tombstone from behind and topples it. (The shot is in the act I animatic.) At first glance, this seems impossible with Kinect because the tombstone will occlude the performer, but...then I realized I can avoid the occlusion by turning the shoot around 180 degrees, and have DMC track the performers 'backside'. Again, this is just a theory at this point, but I think it will work.

The point I'm trying to make is that there may be ways to work around the Kinect's limits for certain types of 'big' moves.

Let you know how this goes.

G.

[cresshead]

awesome thanks for replying so fast and i really do look forward to how your tests turn out for the kinect :thumbsup:

[Greenlaw]

If you understand the how the system works, it may help you improve your peformance and get predictable results. I know that understanding how the PS3 Eye system works has helped me get perfect callibration and consistently good mocap data.

I don't know exactly what iPi Kinect does to track motions but based on my observation here's what I think it's doing:

First it measures the depth of the capture space and finds the ground plane. It does this automatically so, unlike the PS3 setup, a 'calibration' video (http://www.youtube.com/user/LGDTestTube#p/a/u/0/MlG1_65s2_c) is not necessary. Then, when the performer steps into frame, the system isolates the actor from the background and creates a point cloud or mesh with the actor displaced against it. Finally, it presses a 'rag-doll' with body parts that collide against the displaced surface; this is how I think it tracks the rig. There's other stuff going on that tries to help the body from making unnatural poses and for keeping the feet steady on the ground, and it tries its best to interpolate motions when there is occlusion. (I think this is a very simple description of what it does; I'm sure there's a lot of crazy math involved that I will never understand.)

More later this weekend.

G.

[cresshead]

any news on your kinect session?

[Greenlaw]

Sorry, I got sidetracked this weekend because Lightwave 10 appeared to stopped reading the animation from my Motion Builder files. Rebel Hill came to the rescue though; it turns out that if there is more than one animation Take in an FBX file, Lightwave doesn't know which to choose. I'm guessing that it defaulted to Take 1, which I know was an 'empty' Take.

I'll say it again, Valkyrie needs to give users more options in the Import/Export panels. Using Load from Scene with FBX does allow you to pick and choose items, but it still leaves out some useful things out like animation Takes.

Anyway, the animation import issue has been solved so I will get back to Kinect testing soon. Let you know how it goes. :)

G.

[Greenlaw]

Just downloaded build 115 of iPi Studio, which includes April Fools 'enhancements'.

One of the features they added was support for people as short as 0.3 meters tall. This was probably done in response to the silly 'kitty dance' test I did a couple of weeks ago with the assistance of my four-year old daughter. The experiment was only intended as idle fun but now that they've officially added 'little people' compatibility, I think I'll have to give it some serious thought.

Those guys! :)

G.

[cresshead]

you have to hand it to them...they listen and respond :thumbsup:

maybe we'll get 'pet' mocap soon then!:D

[erikals]

that´d be nice, and far from impossible i would guess... :]

[Greenlaw]

I worked with a bunch of horse mocap on a commercial here and saw the footage from the original session. I guess if they can mocap horses...maybe?

Then again, I've worked with cats on a short and they don't necessarily take directions, so the technology might actually be the easy part. :)

G.

[cresshead]

any news on the kinect tests you were hoping to do?
i see the reduced cost offer ends on22nd april...just 2 days away!!!!

anyhow anyoe tried out the kinect with brekel?
http://www.brekel.com/?page_id=155

this is a freeby an makes bvh files [biped friendly apparently]

[Greenlaw]

Sigh! Sorry, workload has been pretty heavy these days so I've had to put personal stuff off.

When I get home tonight I was planning on working on my comics (unless I fall asleep,) but I can do those Kinect tests instead. I'm just as eager to work on 'Pooper' again and I want to recreate a couple of shots that were originally done with the PS3 that I felt needed a retake anyway.

I'll let you know later tonight or tomorrow morning how that goes.

G

[cresshead]

that would be fantastic!

just been playing with the sample bvh from brekel in 3dsmax with biped...
has some issues with the spine bend and the hips seem off...been fixing up the keys somewhat in 3dsmax...no head movement...so been adding some random stuff in there..must say the lack of head movement really drops the quality of motion capture for realism but adding some keys back in on the head really helps sell the motion as more life like and realistic.

seems best to do some auto key reduction importing into 3dsmax then manual key reduction and re editing the feet.

mo cap is never gonna get -mocap record>plot>render in 5 mins!
plenty of hands on editing i still think to clean up the capture data.
..given this sample anyhow!

if there's a chance of a sample bvh [biped export friendly if poss!] i'd love to take a look!

also noticed that a kinect is now just £80 ..down from £120!!

i think i'll get a kinect and see how good this brekel thing is for a start and then maybe get a new g/card and ipisoft if they appear to offer better capture quality and clean up tools.

it's certainly an exciting time to have a basic mocap setup for just £80!

link
http://www.brekel.com/?page_id=394

also worth a read for people just wanting bvh output for use with other apps ie second life, lw etc.
http://rock-vacirca.blogspot.com/2011/02/creating-sl-animations-using-kinect.html

[Greenlaw]

Sure, I can post a test an iPi Kinect .bvh file. I'll try to post one here tomorrow.

As for 'clean up' within iPi Studio, it's pretty limited. You can correct problems during tracking of course but sometimes you'll run into issues that have to be fixed elsewhere. The rotating feet during the 'twist' in my deform test is a good example. The iPi guys explained that this particular action really tests the limits of the iPi Studio rig. To correct this problem in Lightwave you can apply IK on legs and feet and then kill the keyframes for the range of the error; when you're done, you can bake new keys if you want. You can use 'traditional' IK or even better you can use IK Boost for this and let it rebake the keys.

Motion Builder gives you a few more options. For example, you can create aux effectors for the feet that are locked in position and rotation and then constrain the original feet effectors on and off the aux effectors in the graph editor. Once the constraints are off, the mocap keys takes over the feet motion again. Pretty nifty.

G.

[cresshead]

this might help you over in motionbuilder>>

http://www.youtube.com/watch?v=iRCszmNoDyM&feature=related

[Greenlaw]

Hi Cresshead,

Sorry, didn't get to it last night either. Work and small child really kicks my @$$. I'll see if I can get to it tonight.

If all you're looking for are examples of how well the system works, check out the 3D World/iPi Contest winners, which were announced today:

3D World/iPi Software Contest Winners (http://www.ipisoft.com/3dwcomp.html)

Danse Kabyle is pretty awesome, though the artist used two high end cameras and a PS3 Eye to capture, not a Kinect. CG was done in Max. The artist mentions he's looking into adding a Kinect to his setup.

David Huff's fun 'Short Film' was created using a Kinect. I don't know how much clean up or editing he did but I'd say this is very good demonstration of how well iPi works with the Kinect. The main 'limit' with the Kinect is that you must stay within a fairly small region and you pretty much have to perform directly 'to camera' all the time. But this obviously didn't hold back Mr. Huff. :)

I'll try to post an update tonight if I can.

G.

[mrbones]

Click here for the 30 Day free demoCLICK HERE (http://www.shareit.com/affiliate.html?affiliateid=200115028&publisherid=200088208&target=http%3A//bit.ly/le6YKL)

[erikals]

sweet, way better than i would've thought... :]

[cresshead]

microsoft SDK seems to run only on win7?

http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/

[Greenlaw]

FYI for anybody who is interested:

Last week I ordered USB 3.0 Expresscard for my old (five or six years) Dell e1750 Intel dual core laptop so I could connect at USB 3.0 external hard disk at full speed for Kinect capture. The card arrived yesterday and I tested it last night.

It works great! 30fps, uncompressed, and no dropped frames.

The model I got is a two port card made by StarTech, purchased from Amazon. Here's the Amazon product page:

StarTech.com 2 Port ExpressCard SuperSpeed USB 3.0 Card Adapter

The only issue I found was that the card's drivers were not on the CD that came with it. However, the drivers were easy to find on StarTech's website. One note: If you're running with Windows 7, the drivers may already be installed on your computer; in my case, I needed to install them manually because this laptop is running with Windows XP Professional and I don't intend to upgrade it.

The hard drive I'm using is a 1TB USB 3.0 MyBook made by Western Digital. This replaces the eSATA mini-RAID I was using previously.

A couple of notes:

You should connect the Kinect to an existing USB 2.0 port and NOT to the USB 3.0 card. While this may technically work, it's certainly better to dedicate the card to the hard disk.

Also, this laptop set up is meant for recording only; for tracking you need a higher performing computer with a good modern graphics card. Technically, I can track the data with this computer but I would be waiting all year for the results.

A key advantage to using a USB 3.0 drive for capture (besides speed) is that it becomes easier to move the captured data to any computer with a USB 2.0 or 3.0 port.

Hope this info is useful to other users considering this route.

I wish I could do this with my newer tablet PC because it's actually a more powerful computer; unfortunately it doesn't have an Expresscard slot. Ah, well, this functionality gives my old computer a second life. :)

G.

[erikals]

It works great! 30fps, uncompressed, and no dropped frames.

this sounds quite cool,
one question though, wouldn't it be better to use compressed frames?
(second question,.. :] what resolution?)

[Greenlaw]

this sounds quite cool,
one question though, wouldn't it be better to use compressed frames?
(second question,.. :] what resolution?)
Thanks for bringing that up. There was a discussion about this the other day, and yeah, the devs say compression isn't terribly important to DMC; priorities are framerate and then resolution. That said, uncompressed is slightly better and certainly doesn't hurt the tracking results.

iPi Recorder records uncompressed because this puts less stress on the CPU, so even a low power CPU can use the software. The tradeoff, of course, is bandwidth, which is why you need a fast drive. I'm not sure you can turn on compression for Kinect; I'll check this tonight. If you can, this might mean you can use an external USB 2.0 drive for capture. (BTW, I do know you can record compressed when capturing from PS3 Eye cameras.)

Higher frame rate can reduce motion blur, so on this issue, the PS3 Eye system has the edge because it can record at 60fps. (Technically, PS3 Eye can record at 120fps but I'm told that at this frame rate the image quality is too grainy to be usable.)

In answer to your question about resolution, the Kinect records at 640 x 480. The PS3 Eye can record at either 640 x 480 or 320 x 240. In theory, resolution can improves tracking precision but because the Kinect has such a small capture space, higher resolution supposedly wouldn't make a lot difference for Kinect capture. For the PS3 Eye however, more pixels do help when shooting in a large capture space because the performer may be visibly smaller in the image.

G.

[erikals]

hehe, you seem to know alot about this and experiment alot.

so you have a steady system now?
i hope you do, if not i'm sure you will quite soon.

interesting to read about this, hope to give this stuff a try a day later on... :]

[Greenlaw]

Yeah, the mocap workflow is actually pretty solid now. I'm looking forward to the upcoming head tracking but that hasn't been a show-stopper. I'm also looking forward to dual Kinect but it's not necessary for me to continue either. What I really need is to just stay focused and finish this project!

These days, much of my 'free' time is spent learning Motion Builder and figuring out modifications to the native control rig. Frankly, I feel like I know enough about the Motion Builder now to be able to finish the project but every day I discover interesting features and techniques and I just can't help fiddling with it.

My wife keeps telling me I need to stop being distracted or I'll never get the project finished. She's absolutely right of course. :)

BTW, here's an interesting project by another iPi user. His name is Ian Chisolm and he spent the past year working on an 80 minute feature called Clear Skies 3 using the iPi system and a game engine as his main tools. He says he created some 1000 motion files to pull this project off, which seems astounding to me! I've only watched part one so far and enjoying it.

You can watch CS3 here:

Clear Skies 3 (http://www.clearskiesthemovie.com/)

Sigh! And here I am fussing over a three minute music video. :)

G.

[erikals]

:] distraction steals time, then again it also give you great inspiration! :]

btw, will you use zigntrack for facial expressions? or are you aiming at something else? TAFA? RHiggit? Maestro?

[erikals]

or... :]
http://www.youtube.com/watch?v=nYsqNnDA1l4
http://www.youtube.com/watch?v=KKUcECZR-yU&NR=1
 

[D.S.W.]

BTW, here's an interesting project by another iPi user. His name is Ian Chisolm and he spent the past year working on an 80 minute feature called Clear Skies 3 using the iPi system and a game engine as his main tools. He says he created some 1000 motion files to pull this project off, which seems astounding to me! I've only watched part one so far and enjoying it.

1000 motion files doesn't seem that much when you're making this sort of thing. For the short I recently completed - Scotty Chase - I had nearly 500 mocap files and that was for about 34 minutes. I wish ipi had been around several years ago before I'd bought my Optitrack system, but at the time Optitrack was the cheapest solution. Were I getting into mocap now, ipi would have another customer! I'd like an inexpensive way to mocap finger movements. :)

My short on youtube is at: http://www.youtube.com/watch?v=32OTsprfSdI

[Greenlaw]

Hi DSW,

I watched Part 1 of 'Scotty' late last night and meant to comment over on your thread after I watch Part 2 tonight. Great job and congratulations on getting this project finished! :thumbsup:

G.

[Greenlaw]

btw, will you use zigntrack for facial expressions? or are you aiming at something else? TAFA? RHiggit? Maestro?

This might sound curious since I've fully embraced 'homebrew' mocap but I really haven't been that interested in face capture. This is partly because I don't have the bandwidth to add another layer of complexity to my projects but also because I haven't been too impressed with what I've seen yet. But even if I was inclined to use face capture, it really wouldn't work with these characters anyway.

One of the guys I work with has been experimenting with Zign track and says he's been getting pretty good results, so I am curious about trying it myself someday. For now though, I really don't have the time and I just don't want to think about it. :p

So what am I using for 'Pooper'? That good old stand-by Magpie Pro. It really has been the ideal choice for this project and I think you'll understand why when you see the project. (One of these days.) :)

Well, I have a rare afternoon all to myself this weekend, so I better get back to the project. Later!

G.

[D.S.W.]

Hi DSW,

I watched Part 1 of 'Scotty' late last night and meant to comment over on your thread after I watch Part 2 tonight. Great job and congratulations on getting this project finished! :thumbsup:

G.

Thanks. I appreciate that! :thumbsup:

I think that's the most difficult part of any project - getting it finished. Starting something is usually very easy, but seeing it through to completion is (IMO) so very hard. I think I just had too much time invested in this project to NOT finish it. :) Now I guess I'll go back to trying to get more archviz customers in a market that stinks. Oh well... the fun is over.

[cresshead]

1000 motion files doesn't seem that much when you're making this sort of thing. For the short I recently completed - Scotty Chase - I had nearly 500 mocap files and that was for about 34 minutes. I wish ipi had been around several years ago before I'd bought my Optitrack system, but at the time Optitrack was the cheapest solution. Were I getting into mocap now, ipi would have another customer! I'd like an inexpensive way to mocap finger movements. :)

My short on youtube is at: http://www.youtube.com/watch?v=32OTsprfSdI

like it...
though i can't do a good yet myself there's a bit of foot slippage on the mo caps and i'd personally favor more eye animation over lipsync as the the eyes are more important i think to sell a character's performance.

[D.S.W.]

like it...
though i can't do a good yet myself there's a bit of foot slippage on the mo caps and i'd personally favor more eye animation over lipsync as the the eyes are more important i think to sell a character's performance.

I completely agree - with both the foot slippage and eye movement. The foot slippage was probably more noticeable in the scenes with LOTS of characters as opposed to the ones with few characters. Facial animation is definitely not my forte. I had a few shining moments, but overall it still needs much work. I'll chalk it up to never having done anything like this before - my first animation. :)

I really wish ipi had been available before I started this - would have saved me LOTS of cash!

And thanks! :thumbsup:

[Greenlaw]

Here's some interesting iPi motions by Jean Dellac, who uses six PS3 Eye cameras:

http://www.youtube.com/user/JeanDellac#p/a/u/0/Bku5mIJEvx4

Curious to see how this data will be used.

Today I learned out how to use Motion Builder's Relation Constraint to rig the two cats' tails. I wish I knew how to do this a few months ago because it's going to be a huge time saver. The tail mostly animates itself now and the procedural motion is very convincing, but if I need a specific motion, the rig also allows for easy manual override. For the most part, the 'set it and forget it' nature of this rig is perfectly fine for this 'lazy animator'. :p

I wish there was more advanced Motion Builder training material available, commerical or free. Believe it or not, there's far less Motion Builder information out there than there is for Lightwave. The most practical MB material so far has been the free material from 3D Buzz, and it's apparently old material. The subscription service Digital Tutors has some good videos for more recent versions but it's mainly introductory level. If anybody has other suggestions, please post. Thanks!

I'm finding certain threads on CG Society forums helpful too. This for example:

http://forums.cgsociety.org/showthread.php?t=850301

In just five simple pictures, this guy clearly explains how to set up and use the IK Constraint in MB; the same topic in the official Autodesk manual completely lacks this kind of clarity and practical example. Thank goodness for user communities! :)

G.

[cresshead]

Here's some interesting iPi motions by Jean Dellac, who uses six PS3 Eye cameras:

http://www.youtube.com/user/JeanDellac#p/a/u/0/Bku5mIJEvx4

Curious to see how this data will be used.

Today I learned out how to use Motion Builder's Relation Constraint to rig the two cats' tails. I wish I knew how to do this a few months ago because it's going to be a huge time saver. The tail mostly animates itself now and the procedural motion is very convincing, but if I need a specific motion, the rig also allows for easy manual override. For the most part, the 'set it and forget it' nature of this rig is perfectly fine for this 'lazy animator'. :p





cool, sounds very much like the tail option on 3dsmax biped character..here's some rough tests of my wip cat

rig test procedual walk, note all of this is procedual all i animated was to re pose the initial tail on frame 0.
eye's and mouth are morphs and head turn is on a new animation layer above the footsteps.
http://www.youtube.com/watch?v=sPBXMi9I8Ak

laugh mo cap
http://www.youtube.com/watch?v=XrTSodxTj4w

slinky dance mo cap
http://www.youtube.com/watch?v=2kjQYq0oOrc

miffy test using character studio's footsteps
http://www.youtube.com/watch?v=5YtosIxHaAo
note in the wireframe passes you can see red splines showing the lower base layer of animation where the procedual walk was...i added a layer of animation to repose the arms so they did not pass thru the body and add some head keys.

biped may not be as powerful as motion builder but for me it offers similar features for the most part regards mo cap.

[geo_n]

biped may not be as powerful as motion builder but for me it offers similar features for the most part regards mo cap.

I agree, if lw had something similar to characterstudio we could be doing some work in lw for games. I'm sure motionbuilder is more flexible and powerful but for studios with limited software budget it works. We haven't needed maya or mb for any projects and we actually force our one maya guy to use max :D

Here's an IP test we had
http://jnomura.blogspot.com/2011/06/ip-soft-mocap-testing.html

Hopefully we can perfect IPsoft workflow so we dont have to pay for mocap to a studio.
A game we did with paid mocap.
http://jnomura.blogspot.com/2011/06/pachinco-game.html

[Greenlaw]

Hmm. Tail rig works great but when I tried the circle walk motion there is a pop just as the characters complete the walk. I'm guessing it's a gimbal issue but because there are no actual keyframes on the bones, I can't see exactly where it's occurring. Next problem: I don't know how to plot (bake) this procedural motion data to the skeleton. When I use Plot Character, only the data from the Control Rig is applied. I guess I need to get this constraint on a Control Rig tail extension and not the base skeleton.

Not a complaint about MB, just me showing off my lack of experience. :p

G.

[Greenlaw]

Hmm. Tail rig works great but when I tried the circle walk motion there is a pop just as the characters complete the walk. I'm guessing it's a gimbal issue but because there are no actual keyframes on the bones, I can't see exactly where it's occurring. Next problem: I don't know how to plot (bake) this procedural motion data to the skeleton.
Quick followup to what I wrote yesterday: I figured out that if I use Plot All (Plot All Properties) from the Animation menu, MB will bake the Relation Constraint animation to the tail bones. After I did this I could see and edit the gimbal problem in the f-curve (graph) editor. BTW, MB has a few filters designed to deal with these specific problems; unfortunately, the results were not exactly what I wanted, and I think the keys will need to be fixed manually. More later when I can really focus on this problem.

For now, I might wind up doing this tail rig in Lightwave using Follower or I might try using the new options in the Motion Options panel. If I go this route, one thing I would miss is MB's nodal interface for this type of constraint; it really does make editing something like this very easy. Not sure how or even if I can do this joints setup using Lightwave's nodes. If anybody knows how, please let me know. Thanks!
G.

[erikals]

not sure, but i know Cageman made a tutorial on how to attach bones to vertices using nodes...

[Greenlaw]

I'm not sure that's what I need in this situation but I can think of some very practical applications for this technique.

I'll look for his tut tonight. Thanks for the suggestion.

G.

[3dWannabe]

I'm not sure that's what I need in this situation but I can think of some very practical applications for this technique.

I'll look for his tut tonight. Thanks for the suggestion.

G.
Yeah, you really have to dig for MB tutorials. Shame the Digital Tutor's are so basic. If you ever figure it all out, you should record some short videos using the free http://www.techsmith.com/jing/ or the better http://www.techsmith.com/camtasia/ (which I'm going to use to try to document all my steps, as I know I'll forget them).

Have you downloaded these tutorials for MB?
http://usa.autodesk.com/adsk/servlet/index?siteID=123112&id=13293201&linkID=9242336

http://motionbuildertraining.blogspot.com/

[erikals]

I'm not sure that's what I need in this situation but I can think of some very practical applications for this technique.

I'll look for his tut tonight. Thanks for the suggestion.

G.

maybe it was this one...
http://www.newtek.com/forums/showthread.php?t=116973

nah, i still remember another one...
edit: ok, it's this one...
http://www.newtek.com/forums/showthread.php?t=79042
ftp://cageman.dyndns.org/Cageman_Creating_wakes/ (video links are here)
ps, Dynamic Bonechain plugin can be found here... http://forums.cgsociety.org/showthread.php?t=587569

not sure if it will do what you want, still worth a watch though...

 

[Greenlaw]

cool, sounds very much like the tail option on 3dsmax biped character..here's some rough tests of my wip cat
Hey, that's cool! Thanks for sharing these videos.

That's more or less what I'm getting with the Relations Constraint setup I described. Except for the occasional gimbal pop, I'm surprised by how natural this motion looks.

G.

[Greenlaw]

Yeah, you really have to dig for MB tutorials.
Yes, it's strange considering how MB has more or less been the industry standard for dealing with mocap data. I think Autodesk purchasing it, raising the price, and apparently letting die a slow death is not helping. From what I can see, they are cannibalizing parts of MB for Maya now.

BTW, some of the really good videos on AutoDesk's website can also be found at 3D Buzz in a more complete presentation. For example, AutoDesk is missing chapters of the Reverse Foot rig tutorial, but the complete tutorial can see watched at 3D Buzz.

G.

[Greenlaw]

FYI: Solved the gimbal pop problem by replacing the Damping (3D) object with a Damp (Rotation) object in the Relations Constraint window.

[Greenlaw]

It's been months since I posted any new info about 'Pooper' and our 'homebrew' mocap system so here's a new 'behind the scenes' peek inside my computer:

Motion Capture for Sergeant Marshmallow (http://bit.ly/SgtMocap)

It took us a whole lot longer than we anticipated to get our motion capture workflow in place and stomp all the bugs out of it. We started this project last October (2010) and since that time our workflow and character rigs have gone through dozens of false starts and reboots before we established a system that was efficient and could produce results we are happy with. Finally, we can move on and finish this darn project! :)

As always, if you got questions, feel free to ask 'em here.

I'd like to thank Cageman, Rebel Hill, MaDDoX, Geo_n, and other artists here who offered their time, advice and support, and our friends at iPi Software, Autodesk, Animeeple and of course NewTek for creating the awesome software that's making this happen for us.

Okay, later; we gotta a movie to finish making.

G.

[3dWannabe]

It's been months since I posted any new info about 'Pooper' and our 'homebrew' mocap system so here's a new 'behind the scenes' peek inside my computer:

Motion Capture for Sergeant Marshmallow (http://bit.ly/SgtMocap)
G.
Thanks for sharing. Looks cool so far!

Always takes longer than you think, huh!!!

[Greenlaw]

Always takes longer than you think, huh!!!

You'd think I'd know that by now but, yeah. When brand new software and technology is involved, projects NEVER go smoothly. :cursin:

Funny thing is we actually had a somewhat working system in place in the latter part of 2010, but as we moved into production mode, flaws and problems popped up at every turn. In theory, the process and tools for getting motions from iPi DMC to Lightwave was technically all there, but in practice everything wound up being a lot harder than it should have been, and because my wife and I work professionally as animators and we both had some previous 'high end' mocap experience, we felt our 'homebrew' mocap experiment should be making our tasks just a little easier and not insanely more difficult. Alisa used to joke that if we had been manually keyframing 'Pooper' we really would have been done by last Christmas. When spring came along and we were still struggling to get some shots done that we were satisfied with, we weren't laughing anymore. But we also strongly felt that there was something here was worth pursuing.

The problem (as you know well) was that the concept of 'home brew' motion capture was very new and only a few people were actually trying to create anything with it yet. It was one thing to get a few random tests working but quite another to create and apply specific actions to unique characters in larger productions.

A lot has changed in nine months! Newtek, Animeeple, and iPi Software all listened to the issues and got their software to play nice with each other, and several artists with professional mocap experience jumped in and answered questions, posted videos, and helped us figure things out.

Newtek's response to our needs was Lightwave 10 and what a world of difference it's making! The latest beta version of Valkyrie fixes all the problems we ran into this past year and is adding a lot of new features. For that, Newtek has rightfully earned the following smiley from us: :lightwave

Okay, time to go 'dark' again as we work towards finishing this project, but I may pop up occasionally with a short progress report so stay tuned.

Later! :)

G.

[Philbert]

Looks pretty good! I really hope iPi has some kind of crazy sale IO can take advantage of for Siggraph right now my Kinect is just sitting there. I hope someone is doing something with the new SDK for mocap too.

[Greenlaw]

Question to any Lightwave riggers who are listening: Is there an easy way to switch/blend IK goal items? Here's what I want to do: Currently Lightwave allows you to blend IK/FK, which is pretty awesome, but I was wondering if it was a way to keep IK enabled and blend the reach between two or more goal items. I don't actually need to do this right now but I know this capability will come in handy at some point.

Thanks in advance for any helpful info. :)

Hmm...I was just about to click Post Reply when it occurred to me that I can probably enable IK for a goal item and have it reach for another goal item, creating a daisy chain of IK goals. Unless I'm missing something, the result might be what I described above.

I'll test it today and post what I learn.

G.

[erikals]

no idea, dynamic parenting?
http://www.newtek.com/forums/showthread.php?t=90088

 

[Greenlaw]

no idea, dynamic parenting?
http://www.newtek.com/forums/showthread.php?t=90088 
Thanks for the link! That's pretty close to what I was thinking but I want to go farther by daisy-chaining multiple parents (what I was calling 'goals'.) The issue I see with my intended setup is that I can only switch parents in a linear sequence, that is, I probably can't switch the main goal's parent to just any parent item in the chain arbitrarily. Ah, well, I don't really need this now so when I have more time... :)

Edit: I just read through the whole thread, and it sounds like this is what other artists were trying to do. I'll have to take a closer look at this.

[Philbert]

Interesting. I've always used Parenter for that.

[Greenlaw]

Doesn't Parenter require baking though? I was trying to avoid that. That said, IK Boost requires re-baking frames too.

[LW_Will]

Doesn't Parenter require baking though? I was trying to avoid that. That said, IK Boost requires re-baking frames too.

Yeah, but the "baking" in IKB isn't technically Baking. its like a change of state. You can easily bake and unbake parts of the IK chains.

Got to learn that software...

[Greenlaw]

Today my wife and daughter had to attend a birthday party and I finally got an entire day to myself to capture and track performances for our second mocap movie. (Note: Pooper is on hold until after October because now we're busy working on a new film for Gothtober 2011 (the tenth anniversary show!) to be released in October.) To keep things simple, this film was written entirely around using only the Kinect for motion capture, and the trusty but involving PS3 system would be used only if we ran into problems with the Kinect data.

I surprised myself when I realized I had recorded all the motions for three characters for a one minute film in under an hour. And this hour even included rehearsal time for each character!

Right now I'm tracking the data from this afternoon and so far it's looking good. The only tracking issue I'm running into comes when the performer sits down on the floor from a T-Pose; the tracker will consistently lose my arms as they become momentarily occluded by my legs. This behavior was more or less expected and fortunately I do not need any of this part of the motion. To work around the issue, I wait until the performer completely sits and settles down with his 'broken arms', and then I simply fixed the arms and start my actual tracking from this point.

Tip: iPi Studio allows you to copy and paste poses. It's a good idea to copy the actual start frame where you may have done some pose correction just in case you need to reset this at that frame or another frame. If you paste it on another frame, remember to click the Refit Pose button a couple of times.

At the moment, I'm already done tracking half of all the data, and tracking is taking about 0.37 second per frame. I hadn't noticed until today that Kinect data is much faster to track that PS3 data (about 0.67 second per frame,) most likely because there is a lot less data to process. At this rate, I expect to have all the motions for this movie tracked tonight, and I may possibly have all of it retargeted to the character rigs. It's too optimistic to think all the motions will be 'cleaned up' tonight but I'll pick up this up again tomorrow night and may have it finished by then.

Sorry, nothing to show off yet--I'm going to wait until after we present this new project during Gothtober--but I'm very excited about the unbelievably fast progress today and just had to mention it here. I truly feel like all the R&D I went through in the past couple of years is really paying off now. The data, even without the character specific-cleanup I intend to do in Motion Builder, looks remarkably clean, accurate and jitter free.

Unlike our first mocap film--the 'neverending' experiment called 'Pooper'--this second film is already getting done significantly faster than if we keyframed the animation, which of course was the reason we got involved with motion capture and iPi DMC in the first place. Hooray! :)

Stay tuned.

G.

[Greenlaw]

Oh, yeah, this might be obvious now but last week our proposed project (vaguely described in the previous post) got accepted for Gothtober 2011. I'm especially excited about it because this year is the tenth anniversary show!

G.

[Greenlaw]

Just a quick post to keep this thread alive and report the progress with our second iPi mocap film. In case you missed our earlier post, the first film was temporarily put on hold while we work on a second, simpler film for next month's Gothtober 2011 event.

Our new film is progressing very quickly! Despite having little time to work on this project (mostly because of my 'day job' at Rhythm & Hues Studios plus the usual 'life stuff',) we already have all the iPi DMC motion capture in place and about half the shots edited (three characters in a total of eighteen shots). A global light kit was created last weekend, and the audio (dialog and SFX) was mixed and edited last week. Last night we selected the music track.

All that remains is editing the rest of the mocap, most of which should only require minor tweaking, the lip-syncing, final lighting tweaks, rendering, and compositing. Well, okay, that's a lot of stuff but at least we're still on schedule. :)

Oh, and we now have premiere date: October 29th! More info to come so stay tuned! :)

[NinoK]

That sounds excellent, looking forward to seeing the films. Have subscribed to this thread :). I would be interested to know how you work once you capture the data with IPI. I'm working on a small but ambitious game (for me) and was using IPI and the kinect to capture some of the motions, which I then re target in Animeeple and bring into lightwave. After it gets into lightwave is where things typically slow down as editing the motion is a pain. I've set up a custom rig that *sort of* lets me do both but its certainly far from perfect. I am assuming you use MotionBuilder which is a bit (a lot really) out of my price range, but any other tips would be appreciated. Also, ill go back and reread this thread as you've covered some of these things, but any new discoveries you might feel like sharing would be great. Also, have you worked with any props yet and how did that work out. Ie, character holding object, or interacting with objects, etc. I am borderline considering hauling my treadmill from the upstairs so I can record proper walk and run cycles, lol.

[Greenlaw]

Thanks for the interest and the good questions! When we're done with this project, I plan to do a few tutorial videos explaining some of the workflow. After we're finished with the other film ('Pooper'), which is a more ambitious project, I hope to make time for a complete set of tutorials.

For now, I can answer a few things in a general way though. All the retargeting and editing is done in MotionBuilder, but there is actually quite a bit of exchange going on between MB and Lightwave. Here's a simplified breakdown:

1. Retarget the .bvh data to your character rig in MB. Apply very minimal 'clean up', fixing only the most egregious problem, and export an FBX for Lightwave. This motion is just being used for camera reference in LW; you can do proper motion clean up and editing later.

2. Apply the FBX motions to your Lightwave rig using LFS Merge Motion Envelopes Only. Import minimal props like furniture the character may interact with. Ideally, you should use proxies for efficiency.

Now you can set up your camera. When you're done, export a camera-only FBX. At this stage, I will also save out an AVI preview render to use as a 'blocking' pass in my video editing program.

The next step is to prepare optional elements for MB. For example, you may want to export an FBX for your proxy set and props to help position the character (like a chair the character may be sitting on or another character he may be interacting with,) and another FBX for props the character may be carrying. Technically, you should be able to do all this with one FBX file, but I've found it more manageable and reliable to break things down into smaller files.

3. Back in the MB scene, merge the LW camera, proxy set and props. The trick to making this work is to make sure the name of the take matches the current take in MB scene you're merging to, otherwise, any existing animation will get wiped out. When importing the FBX, you get a chance to edit the current take's name, so make sure you know what it should be before clicking the Merge button. Now set a viewport to the imported LW camera; the position and lens should be correct but you may need to correct the aspect. (I just select the HD preset and it matches my Lightwave camera settings.) Constrain any props to the character as needed, and position the character to where he needs to be relative to the set. Now you can start editing the motions. When I'm done, I export another FBX for motion transfer to Lightwave.

4. In Lightwave, I use LFS MMEO to overwrite the previous motion on the LW rig. I also use LFS MMEO to apply motion to the props in the LW scene, but in some cases I can simply parent the props to the character and let the character's motion drive the prop. Now you can finish your Lightwave scene.

Even though I'm using MB, some of these workflow ideas can be applied to other programs like Animeeple. You can even do some of this by staying in Lightwave and applying the motion to a control rig and/or using tools like IK Boost, Motion Mixer, and various motion plug-ins, in place of the MotionBuilder equivalents.

One more tip: For all the shots in my project, I only went as far as step 2 for each shot first and moved on to the next one after exporting the 'blocking' preview AVI and adding it my video editing timeline. This way I got a good idea of what motions actually needs to be worked on and not waste time fixing things that won't be seen on camera. Once the 'blocking' edit was done, I moved on to step 3 and onward for each shot.

G.

[nikfaulkner]

ninok - ive been trying to do what you're *sort of* achieving with no luck at all. care to share any pointers, screenshots etc?

it'd be a great help in ironing out my pipeline

[NinoK]

Thanks Greenlaw, that helps. I'll have to mess with the prop thing some more, as usually any props while capturing tend to throw off the actual capture for me.

ninok - ive been trying to do what you're *sort of* achieving with no luck at all. care to share any pointers, screenshots etc?

it'd be a great help in ironing out my pipeline

Well, I found that I needed a base rig that allows both FK and IK animation and toggling between the two. Also a negative Z facing character rig works the best for me in lightwave for ipi-lightwave-animpeele. That way you have the characters left hand facing in the positive X direction. Keeps everything aligned between the programs and IK later. The rig is also based on joints and not bones (although either should work in LW 10.1 as i'm told LW handles rotations better. Still I prefer joints with 0 degree rotations for cleanliness)

That control rig should be set up in a way where you have a root bone that starts all the arm and leg chains but also an 'offset' bone which you can use later to offset all the motion cap keyframes a tad (if for ex. you want the legs to swing out just a bit more at some point or whatever)

So my right leg joint chain looks like (root - R_hip_offset -> R_hip_root -> R_ThighUpper -> R_ThighLower - > R_Knee -> R_ankle -> R_foot -> R_toes -> R_toes_END )

I bring in my LW FBX into animeeple, re target the IPI captured stuff and export a collada back out which I bring into LW. I export 'animations only' when using collada.

For me this step was important in that, the root bones of my rig (shoulder root, legs root) were designated as upper arm, upper leg in animeeple. These are the bones driven by IK in lightwave.

You bring it into lightwave using the 'load items from scene - envelopes only' way.

Finally another important step is you clean up all the channels that are not overwritten by IK when IK is active. This keeps your character completely IK if that is on and FK when you want to use the mocap data. You can use the offsets to slightly alter the mocap data. I say slightly because big changes make it look bizarre and unnatural. By cleaning up i mean deleting empty or useless keyframes on those channels. This lets me do something like, keeping the legs planted using IK while using mocap data for the upper torso or something. It's wonky but it works for the most part for simple things.

Now the *sort of* part for me comes in when trying to blend the two, which I have very little luck with. There is no easy way that I know of so I essentially reduce the number of mocap keys to just the main poses, and then animate on top which is the tedious slow part if I want to drastically edit the mocap data. Still though, it gets me the right timing and makes the overall job a bit easier. Higher end programs like MotionBuilder would greatly help here.

Ok. I am done Hijacking greenlaw's thread (sorry :) )

[Greenlaw]

Thanks Greenlaw, that helps. I'll have to mess with the prop thing some more, as usually any props while capturing tend to throw off the actual capture for me.
Ah, I think I misunderstood what you were asking. iPi DMC does not actually capture props for tracking yet, though that is a feature they announced a while back for a future release.

I have used props during recording sessions but the props themselves are not for capture, they are just there to help me pantomime certain actions more accurately. For example, when I captured Toullie's performance for 'Pooper' I used a Nerf sword as a 'giant' ukulele (which of course would translate as a normal sized ukulele to Toullie.) This prop was purely there to help me lock my hands in the right space while minimizing occlusion. When I retargeted the motion to my rig, this made it much easier to place the 'paws' on the instrument and make it look like he was actually playing the instrument.

Same thing for Sergeant's performance on the harmonica. I played the instrument when capturing the motion data, which helped me with hand position and body posture but the harmonica itself was not captured. It probably wouldn't have mattered though because the scale of Sergeant's harmonica is 'slightly' bigger than my standard Hohner Marine Band harmonica. :)

G.

[NinoK]

Ah, I think I misunderstood what you were asking. iPi DMC does not actually capture props for tracking yet, though that is a feature they announced a while back for a future release.

I have used props during recording sessions but the props themselves are not for capture, they are just there to help me pantomime certain actions more accurately. For example, when I captured Toullie's performance for 'Pooper' I used a Nerf sword as a 'giant' ukulele (which of course would translate as a normal sized ukulele to Toullie.) This prop was purely there to help me lock my hands in the right space while minimizing occlusion. When I retargeted the motion to my rig, this made it much easier to place the 'paws' on the instrument and make it look like he was actually playing the instrument.

Same thing for Sergeant's performance on the harmonica. I played the instrument when capturing the motion data, which helped me with hand position and body posture but the harmonica itself was not captured. It probably wouldn't have mattered though because the scale of Sergeant's harmonica is 'slightly' bigger than my standard Hohner Marine Band harmonica. :)

G.

Yup that's what I meant. except that once I was obstructed with a prop IPI would get a bit confused and sometimes think a prop was a limb, etc. And snap the skeleton incorrectly. So I had to adjust angles so it could 'see' things better but unfortunately sometimes it means dealing with odd angles afterwards or correcting for it. I should give PS3 eye camera method a go I think. Seems like you get some great stuff with it.

However, If they built a system that could use multiple kinects, it would be one killer solution. :).

[Greenlaw]

Now the *sort of* part for me comes in when trying to blend the two, which I have very little luck with. There is no easy way that I know of so I essentially reduce the number of mocap keys to just the main poses, and then animate on top which is the tedious slow part if I want to drastically edit the mocap data. Still though, it gets me the right timing and makes the overall job a bit easier. Higher end programs like MotionBuilder would greatly help here.
Yes, Motion Builder is designed specifically for this sort of thing and it does it very well, and it does it mostly non-destructively through the use of a built-in control rig that sits on top of the skeleton, 'Photoshop style' animation Layers, Takes (variations of an animation), and mixing motion clips on a timeline (Story mode).

Lightwave's IK Boost gives you some of this capability. You can select a bone chain, apply IK Boost, and it gives you a control rig for your selection that gives you a lot of control for editing the mocap. I had some luck with it about a year and a half ago; it's not non-destructive as it requires baking the changes but it does work. Well, it did back then anyway; some users have reported that IKB may be broken in 10.x but I have no experience with IKB and 10.x to confirm it. Sadly, IKB has never been documented very well and there very few third party tutorials out there that explain it well enough for production use. (To be fair, there is very little documentation available for MB too. To learn anything beyond the basics, I've had to rely heavily on guidance from other MB users.)

And then, there's Motion Mixer, which is very similar to MB's Story Mode. I haven't used this in a while too but I know it did work well a few years ago. (We used Motion Mixer a lot for our Call of Duty commercials and the 'oil horses' in our Quaker State commercial.)

I don't know if any of this will be helpful but maybe it will give you some ideas.

G.

[hdace]

I'm very excited about all this and I've been following your reports, Greenlaw, for a long time. We've been writing a script for a new film and my partner and I are planning on investing in a small cheap studio soon. We're planning on spending at least a couple of thousand dollars on it (we've already got the computers). Can't wait to get my feet wet!

Thanks for sharing all your invaluable experiences!

Hal

[Greenlaw]

iPi Software just released the first beta of iPi DMC with dual Kinect Sensor [dKS] support. I picked up a second Kinect Sensor yesterday and started testing it today.

I wasn't planning on testing dKS because I'm busy finishing up Happy Box right now, but as it turns out I have a couple of motions I'd like to reshoot so why not? (Well, it's a convenient excuse anyway.) :)

Here's the Quickstart Guide for anybody who wants to see how this works:

Quick Start Guide with Dual Kinects (http://www.ipisoft.com/en/wiki/index.php?title=Quick_Start_Guide_with_Dual_Kinect s)

I've been using a single Kinect with my old laptop so I decided to give that a try with dKS. It turns out that even though this laptop has six USB ports, it may only have one USB controller. I would have sworn it had two but it doesn't appear that way when I open Device Manager to see why the second Kinect isn't connecting properly. If I swap the cables around, I can get either one to work but only just one Kinect. Eventually, I connected the second Kinect to the USB 3.0 Expresscard that also has the USB 3.0 drive I was using to store the Kinect data connected. I didn't expect this setup to work either...and it didn't. Both Kinects were working correctly now but when I tried to record, there wasn't enough bandwidth for the hard drive. Oh, well.

On to plan B...stay tuned. :)

G.

[Greenlaw]

Luckily my desktop computer is just outside the living room. Using two USB repeater cables, I was able to extend USB connections an extra 16 ft and now I am getting a very strong signal from the two Kinects with absolutely no dropped frames. This computer has an SSD dedicated to capturing motion data from six PS3 Eye cameras so I'm not expecting any difficulties with capturing from two Kinects.

The problem now is that I cannot monitor what's being captured because the computer is in another room. I would ask my wife to help but she's already gone to bed. Sigh!

Maybe the cats will assist?

Oh, I know. This computer has a small 'laptop' Cintiq attached which has a cable that may be just long enough to position the screen past the doorway. Yes, our house is starting to look like a crazy mad scientist lab. I better get this portion of the project wrapped up before morning. :p

G.

[Greenlaw]

Yes! The Cintiq reaches the living room and I can use the Wacom Stylus to control iPi Recorder.

[Greenlaw]

Using the Cintiq to control iPi Recorder seems to work well. I have the device propped on the floor in the doorway so I can monitor what each Kinect Sensor sees and position them precisely. Tilt is controlled by a built-in motor and I can operate the motor for each Kinect through the Cintiq. Using the Wacom stylus I can align cameras with reasonable accuracy this way. Totally unnecessary but gotta admit this is pretty cool!

One drawback. When I double-click a video panel, this should fill the screen with the live video to help with ‘across-the-room’ visibility. However, when I double-click a video panel on the Cintiq, the video expands on my main computer monitor which is in another room. Oops. Can’t use that feature I guess. No biggie.

The cats are intrigued. I have some fun ‘captures’ of Sergeant examining one of the Kinects up close.

One of the nice things about using a single Kinect was that it didn’t require calbration. Not so with dKS. However, dKS calibration is still much easier than calibration for the PS3 Eye configuration. It basically involves holding up a flat board (I’m using a sheet of foam core) to the Kinect Sensors and slowly moving it around. After tracking this data in iPi Studio, the scene data can be saved and reused for all motion tracking data shot for that session.

Note: need to get Kinect tripods. Toullie just jumped on the small table where one Kinect sits, which messed up my calibration. I put in an order for two of these:

CTA Digital Xbox360 Kinect Adjustable Tripod Stand (http://www.amazon.com/Digital-Xbox360-Kinect-Adjustable-Nintendo-DS/dp/B004TJSLEK/ref=sr_1_1?ie=UTF8&qid=1318575279&sr=8-1)

This stand looks more stable than other models I’ve found online. It’s also more adjustable and it has a built-in level. FYI, I found an interesting Makerbot template for printing a plastic tripod mounting head but I don’t have time for that right now. Cool idea though. I will have to try it out later.

Final note: The cats are definitely a problem when shooting mocap data; they’re just too curious about the process and they refuse to stay ‘off stage’. May have to lock them in the kitchen when I’m ready to shoot tonight.

Bandwidth test is good. Ready to shoot calibration again.

G.

[Greenlaw]

Decided to call it a night for dKS. I have tomorrow off so I can continue in the morning after I drop off child at school.

G.

[Greenlaw]

Okay, got a bunch of dKS stuff done this morning. Here's what happened and what I learned:

Calibration is easy. Just hold a board up to the Kinects and rotate it slowly from side to side. The trick it to make sure you don't over-rotate the board. Being able to see the Cintiq behind one of the Kinect Sensors helped me control the angle of rotation. I probably recorded way more data than I needed but I *think* my calibration is good. I'm not 100% sure what I'm looking at yet. FWIW, I only see green dots...no yellow dots anywhere. I think the ground plane might *possibly* be off but I need to do some actual mocap tracking first to be sure. :p

Calibrating dKS data is slower than calibrating PS3 Eye data. The dKS data is different though: because Studio is trying to track a featureless 3D plane instead of a single point of light, I think the software may be doing something more sophisticated. More on this later as I figure out what I'm doing.

G.

[Greenlaw]

Some info and tips:

I recorded my session this morning with the Include RGB stream option enabled. This is a pretty cool feature as it gives you a full color video to use as a reference video. This stream is optional because it isn't necessary for calibration and tracking but it can be useful if you have trouble understanding what you see in the depth video.

If you have bandwidth issues when recording, you can switch this feature off to improve the recording speed. If you record your footage with the RGB stream, you can choose to hide it or show it in Studio by clicking the appropriate button. You can even 'blend' the data for true psychedelic mess.

If you play the video using a standard player (Media Player Classic for example,) the video will display as multiple panels of video channels--not very pretty or useful. This is normal though, so don't worry about it. If you want to output a normal color video file, you'll need to load it into iPi Studio and output a new video file with the desired camera view and display options selected.

G.

Correction: The option in Recorder's Advanced Panel is actually called 'Don't include RGB Stream', which meant I had to disable the option to not have an RGB stream, which means I got the stream. How's that for confusing? :)

[Greenlaw]

Okay, I understand what I'm looking at now. After calibration is completed, in each frame you should see three 'dots' on the board, which define the angle of the plane to iPi Studio. What you want to do is make sure a majority of the frames contain only green dots. If you have any yellow dots, this means the tracker could not resolve the angle of the plane in that frame, and too many bad frames means your calibration video is poor and motion tracking will fail.

My calibration video is 1400 frames long, which is probably way more data than is required for good results, but luckily I'm only seeing green dots throughout the entire video. (The cats really behaved themselves this morning. Good boys!.)

Another tip: Make sure the 'plane' object you use in calibration is perfectly flat. I used a large sheet of foam core board which was slightly warped from standing in the garage all year. I think my results are okay but if the board is warped or bent too much this will obviously affect the quality of the calibration results. (Time to get a new sheet of foam core.)

I wonder if the color of the board affects calibration quality? I'm certain a black board will not work because it may absorb the IR rays from the Kinect, but what about a pure white board? I'm asking because the foam core board I'm using it pure white. I think the results I'm getting are okay but maybe it would be better to use a board with a neutral color value.

Yay! Now, on to tracking. :)

G.

[Greenlaw]

Right now I'm running my first tracking passes. Setting up the scene was easy; first I loaded my mocap video data, then the calibration scene settings I created earlier today. Next I loaded my actor from past sessions and clicked the Refit Pose button a few times. Placing the rig inside the T-Posed volume was was not necessary; the actor just snapped into place on its own. The left leg was slightly out of place on the first 'Refit Pose' click but by the third click the rig was perfectly in place.

Next, I clicked the Recenter Origin button to orient the scene and ground plane to the actor's origin. The ground plane appears to be a little high compared to the video reference but maybe that's okay. I proceeded anyway.

I'm seeing that dKS tracking is just slightly slower than with single Kinect tracking, but the time difference is negligible. With a single Kinect, I was getting about 0.67 sec per frame; with dKS I'm seeing about 0.7 sec per frame. I think that's very good for my graphics card. (My graphics card is a Nvidia GTX 460, which these days is probably considered a mid-range card.)

At a glance, the dKS tracking results appears to be more stable than with a single Kinect. I'll know for sure once the tracking process is complete. My video is more than 3700 frames long, so 'first pass' will take about 40 minutes unsupervised. The individual takes will be much, much shorter of course, and most of the 'first pass' data won't be used at all. I just find it easier to let the software do its own thing first before I start paying attention to the details.

More soon....

G.

[Greenlaw]

Another tip: I think my Kinects are too low; currently about three feet plus a few inches off the ground. I think this might be why the ground plane looks a little low. When my Kinect stands arrive next week, I think I will position the Kinects closer to four feet off the ground.

Edit: I don't know, maybe it's okay: the recommended height is actually between 0.5 and 1.0 m. Four feet is probably too high.

[Greenlaw]

The results of my first attempts at tracking dKS data were a little mixed today. It looks like the system did its best to track my data but there were a few problems with this session.

Part of this is my fault: I was standing a little too close to the sofa behind me and I think the tracker had trouble distinguishing my left leg from the sofa, causing the leg to flip out backwards. I was aware of this potential problem from past single Kinect sessions but it was made a little worse today with dKS because this configuration requires a bit more room, which forced me to step back a little further than I did with the single Kinect. I think I can push the two Kinects back much farther if I change their orientation in the room and add another repeater cable.

Another issue is that the calibration result positions the floor a couple of inches higher than normal, which caused issues with the legs. Switching off feet tracking helps a little but this isn't really desirable of course. (In this case it doesn't matter because in this shot you never see my character's feet on the ground, so the data is technically good for me.) I'm not sure why this happened. Maybe the depth sensor is slightly confused between the shiny wooden floor and 'matte' carpet. May try calibration with and without the carpet and see if there's a difference in calibration accuracy.

I have a few other adjustments in mind; I'll post again when I have something new to say.

G.

[Greenlaw]

I'm running my second dKS session data now with improved camera placement. Originally, I used the length of the room to space the cameras as wide as possible to increase coverage. The problem with this orientation is that I didn't have a lot of depth data, which affected calibration and tracking accuracy. This time I moved the cameras to the end of the room to maximize the range; the cameras are not as far apart now but I now see a broader range of colors in the depth data.

Calibration result is significantly better; ground plane looks correct now. This time I tracked about 500 frames and that seems plenty. A new feature is the ability to 'hide' the background. Basically, the room gets differenced out using 2 seconds of 'background only' data, which is especially cool to visualize in 3D by rotating the perspective view.

I've attached five screen caps below to illustrate the dKS calibration method:

Image 1. This is what the raw depth data from Kinect 'camera' 1 looks like.

http://img7.imageshack.us/img7/4729/image1jgd.jpg (http://imageshack.us/photo/my-images/7/image1jgd.jpg/)

Image 2. This is the same data with the background data 'hidden'.

http://img254.imageshack.us/img254/4650/image2kw.jpg (http://imageshack.us/photo/my-images/254/image2kw.jpg/)

Image 3. The same data seen from Kinect 'camera' 2.

http://img707.imageshack.us/img707/1808/image3bx.jpg (http://imageshack.us/photo/my-images/707/image3bx.jpg/)

Image 4. Looking from behind the 3D volume at the two Kinect Sensors.

http://img254.imageshack.us/img254/3064/image4jz.jpg (http://imageshack.us/photo/my-images/254/image4jz.jpg/)

Image 5. Even better, depth data from Kinect 1 and 2 merged together! This gives you a fairly good idea of the size of the capture space.

http://img401.imageshack.us/img401/9646/image5wb.jpg (http://imageshack.us/photo/my-images/401/image5wb.jpg/)

BTW, if you look to the left of the board you can see Sergeant Marshmallow sleeping on the couch. :)

I enabled 'Don't Record RGB Stream' for this session because I'm shooting at night and you really can't see much with the low-intensity living room lighting; the extra data would just be wasteful. If the RGB data was recorded, you could see a normal video image in the background or as zoomable PIP images in the upper right corner. I'll do a quick session in the morning with embedded RGB data when the living room lighting is much brighter.

Okay, getting prepared to track the new mocap data now.

G.

[Greenlaw]

BTW, I rolled up the carpet that was on the floor and moved it aside. There is a lot of black in the pattern which I think created depth errors for the Kinect. Just one more thing to be aware of.

[Greenlaw]

Note: this will mess up your mocap...

http://img854.imageshack.us/img854/9118/20111015030740.jpg (http://imageshack.us/photo/my-images/854/20111015030740.jpg/)

This time it was Toullie. "Meow!"

G.

[Greenlaw]

Okay, I have several tracked dKS motions now and I just finished applying iPi Studio's Configurable Jitter Removal to a few clips. I have to say, this mocap looks incredibly solid! The results are far superior to the data I was getting using a single Kinect.

More tomorrow. Right now I need to get some sleep. :sleeping:

[erikals]

Note: this will mess up your mocap...

This time it was Toullie. "Meow!"

G.

LoL..!! ; D

[Philbert]

Using the Cintiq to control iPi Recorder seems to work well.

When I was playing with my Kinect and mocap my computer was too far away to see what I was doing and even hit the record button, so I got my tablet and ran TeamViewer on it so that I could control my computer from the tablet. Like using Remote Desktop. I also love team Viewer because I can leave a long render going while I go sit at Starbucks. I can check on the render from my tablet and make changes if something goes wrong while still enjoying my coffee.

[Greenlaw]

LoL..!! ; D
The motion is pretty funny. As soon as Toullie occludes the leg, the leg totally spazzes out and starts kicking him. (The virtual Toullie, not the real one.) Fortunately, I already got the motion I needed.

G.

[Greenlaw]

When I was playing with my Kinect and mocap my computer was too far away to see what I was doing and even hit the record button, so I got my tablet and ran TeamViewer on it so that I could control my computer from the tablet.
That's a great idea. Thanks, I'll check it out.

G.

[Greenlaw]

Hi Phil,

Team Viewer works great for this! I'm able to control iPi Recorder wirelessly from my tablet PC, and having the extra mobility while setting things up is very convenient. Thanks again for this tip.

G.

[Philbert]

Glad I could help.

[Greenlaw]

Today I tracked a segment that required a lot of spinning in place while waving arms in the air. This was a shot that I felt would be a real challenge for the single Kinect setup, and probably for the dKS setup as well.

I had previously recorded this motion using a single Kinect, standing in place and faking the motions of turning around. The idea was that if I framed the shot so you didn't see the character's feet, I could cheat the body rotation in 3D. In practice, this trick worked well for the purpose of the film but I still wanted to see if the new dKS feature would allow me to capture this motion properly.

Even though the shot in the film is only a couple of seconds long, I 'rough tracked' a sequence that ran about 2600 frames, counting on the expectation that I could find in there at least two seconds of good motion to cut around. When the track was done, I was surprised to see that iPi Studio was able to track most of the footage with fairly good accuracy. There were brief moments were occlusion caused a tracking error (the 'hidden' leg or arm spazzing out,) but I felt some of errors could be easily corrected. A few times, iPi Studio was able to track a full turn perfectly.

There were a few moments where the motion 'broke' the rig's rotation limits. In these situations correcting the pose within iPi Studio is impossible. You may be able to correct the motion in a program like Motion Builder but most of the time it's a lot easier to simply recapture the motion differently to avoid stressing the rig. For my purpose, I have plenty of good motion in this sequence to select two seconds from without bothering with any of the errors.

There were a few moments where a pose 'broke' the rig's rotation limits. In these situations correcting the pose within iPi Studio is impossible. You may be able to correct the motion in a program like Motion Builder but most of the time it's easier to simply recapture the motion differently to avoid stressing the rig. For my purpose, I have plenty of good motion in this sequence to select two seconds from without bothering with any of the errors.

I can't show any of the footage until after my project is finished but I thought this info would encourage artists who were asking if this kind of motion was possible using iPi DMC and dKS. Short answer, sort of; the key is in camera placement. My cameras are not anywhere near 90 degrees, but if you have the space to be able to increase coverage, perfect tracking of full turns of many motions should be quite possible.

G.

[Greenlaw]

Just for kicks I took a shot at correcting all the errors. Surprisingly, I was able to clean nearly all the errors in the entire 2600 frames in just a few minutes by making rough adjustments, hitting Pose Refit, and then tracking backwards through the trouble spots. In fact, there was really just one spot where occlusion made it impossible to correct the motion but this error lasted less than a second. Not bad at for almost a minute and a half of constant motion . Rig stress in the hips and legs was still a problem in a couple of spots where the body was twisting, but I was more surprised that this error didn't occur more frequently. Overall, the results are looking pretty good.

[Greenlaw]

There was one tracking error that puzzled me a little: for a couple of seconds one arm seemed to lag behind unnaturally. Then it occurred to me that this was might be a motion blur error, which can be expected because the Kinect's frame rate is only 30fps.

Anyway, that's another consideration when deciding between Kinect and PS3 Eye setups: the PS3 Eye's framerate for iPi DMC is 60fps, which can minimize and even eliminate motion blur for many fast actions.

[NinoK]

Anyway, that's another consideration when deciding between Kinect and PS3 Eye setups: the PS3 Eye's framerate for iPi DMC is 60fps, which can minimize and even eliminate motion blur for many fast actions.

Oh, that's interesting, i thought both were at 30. Good to know, thanks!

[Greenlaw]

Oh, that's interesting, i thought both were at 30. Good to know, thanks!
Technically, the PS3 Eye cameras are capable of 120fps. iPi limits the framerate to 60fps because the image does get a bit grainy at 120fps.

G.

[Greenlaw]

My Kinect tripods arrived today. The tripods are a bit flimsy for what they cost but they do the job. The Kinect mount was what I was most interested in and it will attach to any camera tripod. The design is good but unfortunately it's are made from a very cheap plastic that may break easily if you're not careful. That said, these tripods do work and this is better than resting the Kinects precariously on a chair or table edge.

After I'm finished with this project, I think I'm going to print out one of these on a Makerbot and see if it works any better:

http://www.thingiverse.com/thing:8569
http://www.thingiverse.com/thing:5110
http://www.thingiverse.com/thing:5601
http://www.thingiverse.com/thing:11573

G.

[Greenlaw]

Yesterday I began retargeting the new dual Kinect Sensors data to the characters. Here's a sneak peek at the first dKS motion applied to a character:

iPi DMC Dual Kinect Sensor Test for 'Happy Box' (http://www.youtube.com/user/LGDTestTube#p/a/u/0/XLWygMW12HI)

Enjoy

G.

[Greenlaw]

For comparison, here's a motion test for the film using a single Kinect Sensor:

iPi DMC Single Kinect Sensor Test for 'Happy Box' (http://www.youtube.com/user/LGDTestTube?feature=mhsn#p/a/u/0/68W7Ye86gH8)

G.

[Greenlaw]

http://img695.imageshack.us/img695/5739/s002framegrab.jpg (http://imageshack.us/photo/my-images/695/s002framegrab.jpg/)

It was a ton for work and it took a month of sleep deprived evenings but the sneak preview of Happy Box is finally live at Gothtober.com (http://www.gothtober.com/)! To see it, go to the Gothtober website and click on the 29th.

For those who need more info, Happy Box is our first Brudders 'mini film' and our second short film for Gothtober (the first was Hello Frankenstein (http://www.youtube.com/user/littlegreendogmovies#p/u/7/qN2Z8kTND00).) Gothtober is an annual Halloween inspired internet event featuring short films, animation, slide shows and recipes from a variety of crazy people (a.k.a., artists) for the entire month of October.

The reason I'm calling this a 'preview' version is because I had to cut some corners to make our October 29th deadline.

Most noticeably missing is the lipsync animation—I simply ran out of time to finish it so I decided to leave out the lipsync completely because it looked weird to have only half the film with lipsync animation. Arguably, the show still plays well without it.

There are also a few 'bugs' that need to be fixed, props that need to be properly finished, a few FX added, and I really wish to re-render the entire project in HD. (My little render farm lost half it's capacity last week so I wound up rendering this project at 320 x 180; surprisingly this 'web res' actually holds up when scaled to double-size (640 x 360) for Gothtober.)

The final release version of Happy Box, with lip sync animation, bug fixes and other improvements should appear at our website later in November. In the meantime, please enjoy this exclusive Gothtober preview!

Now, I beg your indulgence as I take a moment send out heartfelt thanks to my friends:

To JP and Cristin for presenting Gothober every year. Gotober has been a great motivator in getting me to do 'big things' just for fun.

To the guys at work, for their curiosity and encouragement, and for just being a lot of fun to hang out with.

To the Lightwave and iPi Soft user and developer communities for their interest and support. Especially to Cageman and Rebel Hill for their technical advice and assistance in setting up my motion capture pipeline.

And finally, extra special thanks and love to my wife Alisa and my daughter Sienna Kai for putting up with all my nonsense. I could not have done this without you guys.

Come Monday morning, I'm taking a break and going back to work for a while. :)

G.

[Greenlaw]

Oh, and in case anybody was wondering...

All of Happy Box was originally 'puppeted' using a single Kinect, and much of the action was reshot using dual Kinects when that beta feature became available in iPi Desktop Motion Capture about three weeks ago. The retargeting was done using Motion Builder (mostly required because of the tricky 'feline' rigs.)

Modeling was done using Modo and Lightwave 3D, textures painted using 3D Coat, and rigging, shading, lighting and rendering was done using Lightwave 10.1. Compositing was done in Fusion 6.2, and video editing and sound mixing using Vegas Pro 10.

G.

[Greenlaw]

My friend JP posted a nice blog entry for Happy Box here:

Happy Madness with Brudders • Day 29 (http://www.gothtober.com/gothblog/2011/10/30/happy-madness-with-brudders-%e2%80%a2%c2%a0day-29/)

G.

[quantizer]

Nice short film hehehe. About the stands, would you please post the video on how to mount it to the camera?

To mount the cameras you'll want to add a $18 Helicoil kit that lets you tap the bottom of the cameras so you can mount them to the c-stands. This is easy to do because the PS3 Eye cameras already have a hole in the metal base (it's under the sticker.) I shot a video of the process; I need to edit it and then I'll post it on our website. A single kit is good for 12 cameras, so I guess you could split the cost with another iPi user.

[Greenlaw]

Thanks!

I'll try to do the video as soon as I finish this film. I got most of the 'bugs' squashed last week and I'm getting close to finishing the lipsync animation, and most of the shots will have been re-rendered at double-res by tonight (no HD yet though...I think that will have to wait until I have a little more render capacity.)

After that I'm eager to move onto the next film. But before that I'll spend a little time catching up on other projects that have been neglected (like my webcomics and a handful of how-to videos like the 'helicoil kit' video.)

G.

[geo_n]

Funny mini chain saw! :D
What are you using for lipsync?

[Greenlaw]

I'm using Magpie Pro. The specific technique I'm using isn't very standard and it wouldn't work for most character animation, but I think it's working for these characters.

It might be self-explanatory when you see it but I'll explain it after it's done.

G.

[Philbert]

Wow I haven't touched Magpie in years. We learned it at Dave School.

[Greenlaw]

Wow I haven't touched Magpie in years. We learned it at Dave School.
Same here, and this is the first time I've used it to generate 2D elements (as opposed to 3D Morph Mixer data.)

Well, I started out generating Morph Mixer data. Basically, I created morph targets for every mouthshape image and was using step frames to pop the desired mouth image in place. It's kinda like a 3D version of the head/face replacement technique used in some stop motion films. This method worked very well in my early tests but after doing a few shots this way, I decided it was on it's way to becoming impractical because the number of mouth shapes I wanted started to grow at a crazy rate. (I'm doing at least three sets for each mouth shape for each of the three characters, that is left, right, and center, to mimic how it's drawn in the comic strip. I didn't need all that for Happy Box but this setup was meant to be re-used in future Brudders shorts.)

Earlier this week I decided to revisit my original strategy, which was to simply render out images sequences of mouth shapes from Magpie to use as textures on a single 'mouth' object. I turned away from this approach during the last week before my Gothtober deadline because I just didn't have the head at that time to do more R&D. But after a few tests, I was able to put together a fast and efficient workflow and the results are not bad. (The idea very simple and kinda like what they do for Robot Chicken--but hopefully the mouths in Happy Box will look a little better.) :)

I'll post more info after it's done, maybe even demo each of the lipsync setups in a video. Even though the final results between teh two methods are identical, each technique has it's own strengths and weaknesses. Right now, it looks like I'm still on track for finishing Happy Box this weekend.

G.

[Mr Rid]

Dennis, I saw you get killed in Deep Core just now on Encore.

[Greenlaw]

I took Chan down with me.

G.

[Greenlaw]

Quick update: We're very close to finishing Happy Box tonight. This final version is much better than the sneak preview we showed a couple of weeks ago. I still need to finish modeling and texturing Sparklepus, and that might have to wait till tomorrow evening.

Then I'm done and it's on to my next movie project: RVJ. Stay tuned for what will be the last update for this project. :)

[Greenlaw]

Hey guys,

I posted the final version of Happy Box this morning. You can watch it here:

Brudders in 'Happy Box' - Final Version (http://www.youtube.com/user/littlegreendogmovies#p/u/0/gy3hSz7_zb8)

So what's next? I've got a few comic strips to draw and then it's on to the next short film. :)

G.

[3dWannabe]

The link seems messed up?

Hey guys,

I posted the final version of Happy Box this morning. You can watch it here:

Brudders in 'Happy Box' - Final Version (http://bit.ly/uXmKJq)

So what's next? I've got a few comic strips to draw and then it's on to the next short film. :)

G.

[Greenlaw]

Oops! Sorry, try again. :p


Brudders in 'Happy Box' - Final Version (http://www.youtube.com/user/littlegreendogmovies#p/u/0/gy3hSz7_zb8)

[3dWannabe]

I like the 'Jason' reference, very funny and well done!!!

[Philbert]

That's a lot of fun, I'll be sure to share it.

[allabulle]

I love it!

[Pavlov]

hi,
i'm really interested in a mocap tool like this, if it allows walks (or broad-range anim) too.
Are there issues with camera distance ? Any example ?


PS - sorry if it has been covered, no time to read the whole thread ;(

thanks,
Paolo

[Greenlaw]

hi,
i'm really interested in a mocap tool like this, if it allows walks (or broad-range anim) too.
Are there issues with camera distance ? Any example ?
Yes, you can do walks but you're limited by the size of your capture space.

With Kinect, the capture space is very small, and with two Kinects it's even smaller. If this is the system you choose, you may need to loop your animation as a walk cycle. You can do this using tools like Motion Builder or Lightwave's Motion Mixer.

The other option is to use PS3 Eye cameras. The Eye cameras are more complicated to set up but with four to six cameras you get significantly wider coverage, plus twice the framerate. Framerate is the most important feature in motion capture because higher frame rates decrease motion blur and provide more details to the tracking system. For comparison, Kinect is only 30fps but PS3 Eye is 60fps. Technically PS3 Eye is capable of 120fps, but the developers say the image is too grainy to be useful at that speed.

Anyway, you asked for an example. I have a short one here:

iPi Studio Markerless Motion Capture: Deform Test Mocap (http://www.youtube.com/user/LGDTestTube#p/u/4/G2KLtGsl-L0)

At the end of the video I walk around in a complete circle. This is a pretty old test using only four PS3 Eye cameras.

A more impressive example was Danse Kabyle created by 1k0. He used only 3 cameras (two highend cameras and a PS3 Eye,) and composited the footage in After Effects to make it iPi DMC compatible. The original video is here:

Danse Kabyle (http://vimeo.com/22028464)

Be sure to watch his 'Making of' video. It shows him not only walking but running and leaping a good distance. That video is here:

The Making of (Part 1) (http://vimeo.com/22036677)

G.

[3dWannabe]

Greenlaw - are the Kinects reading head motions and is iPi using that data? In this 'making of' of your mocap, it looked like the might be?
http://www.youtube.com/watch?v=XLWygMW12HI&feature=player_embedded

I've heard MS is going to update the Kinects. Any idea of frame rate or capture space will improve?

[Pavlov]

very interesting, i'll dig this thing more in next weeks.

thanks a lot,
Paolo

[Greenlaw]

Hi 3DWannabe,

Greenlaw - are the Kinects reading head motions and is iPi using that data? In this 'making of' of your mocap, it looked like the might be?

Sorry, no...that must be an illusion in the poor video quality. The head motions in the final were added in Motion Builder, and the setup was pretty easy. (You can do this easily in LW too.) All I did was create a null (or a sphere object in my case) and told the head bone to look at it using an Aim Constraint. I have a few other simple controls on that control null, like rotating its bank will tilt the head from side-to-side, but that's about it. This very easy to do in MB or LW, all that was required for this project.

In some situations it can be useful to constrain the null to the character. This minimizes the amount of keyframing you may need to do because the null will move with the character. In other situations, you want the null to move independently of the rig and not inherit any motion from the body.

Believe it or not, I found it useful to keyframe my head motions because the characters needed to look at very specific things, like the boxes, food, toys, other characters, and sometimes change their focus within a shot. This was tricky to do in live performance because 1.) none of the props exist in the real world, 2.) I performed all three characters myself, and 3.) in some shots the character performances had to be in-sync with each other. If I had actual head motions, chances are likely that they would be wrong and need tweaking in post anyway.

One important thing I discovered is that you can totally tell when an item like the head is being keyframed because the motion will look cleaner and smoother compared to the mocapped motion on the body. I was able to 'blend in' a slight wiggle by adding a parent null with the 'wiggle' motion applied to it. The 'wiggle' motion can be extracted from the mocap, generated procedurally, or by simply keyframing it.

I've heard MS is going to update the Kinects. Any idea of frame rate or capture space will improve?

I have very little knowledge about this as I was too busy working on the film to keep up with outside developments. My guess is yes, eventually, but a guess is all that is.

Hope this helps. :)

G.

[Greenlaw]

Pavlov,

My explanation may have appeared to favor PS3 Eye so I want to clarify a few things. Each system has strengths, so which system you go with depends on the equipment and space you have available and what you intend to do with it.

Here's a brief explanation:

Kinect is very easy to setup and use, and most users will probably go with this setup. A single Kinect will work with most laptops; two Kinects will require a beefier laptop or desktop computer. You can use this system in an average living room space.

PS3 Eye gives you more accurate captures and allows for a huge capture space. Four to six cameras are recommended and may require a desktop computer with a very fast drive or SSD (recommended.) A powerful laptop may work with four to six Eye camearas if you use compression and lower your resolution. Compression doesn't really hurt quality, and resolution is only important for very large spaces. The cameras are cheap ($35 each) buy you may need accessories, and I found it useful to physically modify the cameras for mounting on conventional light stands.

Both systems may require USB extension cables. For that you want what's called a Repeater Cable. This type of cable actually boosts your signal and you can daisy chain up to five of them for incredibly long lengths. You can buy them from NewEgg for about $12 each. Typically, I will use one to two cables for each camera. (Except maybe the camera nearest to the computer if it's close enough.)

At the moment I'm really digging the dual Kinect configuration because it's so easy to set up and use, and I can get immediate feedback. For the PS3 configuration, I need to move my computer to another location for capture and then bring the computer back to my home for tracking. This is worth the effort if I have something very complicated to record or an action that requires a lot of space. Most people will not need this set up but it's good to have the option when you do need it.

G.

[Philbert]

My Kinect and laptop wouldn't work for me, the iPiSoft recording stops with an error after about 10 seconds. Of course my laptop is 4 years old, but it was top of the line at the time.

[Greenlaw]

My Kinect and laptop wouldn't work for me...Of course my laptop is 4 years old, but it was top of the line at the time.
I can get a single Kinect to work with my old laptop (Dell E1705) but I had to add a USB 3.0 Expresscard and USB 3.0 drive to get there. I could not get dual Kinect to work with this laptop.

I think some users at the iPi forums have said they got dual Kinect to work with newer laptops but I have no personal experience with this.

G.

[Philbert]

Oh, I'm sure it's the USB2 I'm using then.

[Greenlaw]

I should clarify: USB 2.0 is all you need for the Kinect...USB 3.0 will not improve anything here because it's a USB 2.0 device.

The USB 3.0 Expresscard was for the USB 3.0 drive only, which is a lot faster than my USB 2.0 external drive, opening up the bandwidth for data capture. This setup turned out to be faster than my little eSATA RAID and allowed the laptop to work with Kinect.

I stopped using the laptop when the dual feature came out--simply not enough horsepower. My quad-core desktop has absolutely no trouble though. I'm capturing the data to an SSD dedicated to motion capture but it might actually be overkill for Kinect data.

G.

[Philbert]

Looks like iPi Soft is 30% off this month if anyone's interested.

http://www.ipisoft.com/sales.php

[Philbert]

Kinecthacks.com posted Happy Box:
http://www.kinecthacks.com/kinect-short-film-happy-box/

[Greenlaw]

Cool! Thanks for the heads-up Philbert! :)

G.

[Greenlaw]

Thought you guys would enjoy this music video:

Freak Flag by Here Come the Mummies (http://www.youtube.com/watch?v=HhG2lBLCtdA)

The artist who made this, Bobby Brane, says the mocap was created using iPi DMC with Kinect. (It was rendered in Max but that's besides the point.) Looks great and the song by HCTM is pretty awesome. :)

G.

[Greenlaw]

For those who have been following, here's a quick update about the next Brudders mocap movie:

Alisa and I are finally back to working on the second Brudders film. I've been on 'holiday' for a few weeks and trying to get as much done with this project before I have to go back to work. Not sure when this film will get done; it's a lot more complicated than Happy Box was. Oh, well, at least now the iPi DMC-to-Lightwave workflow is charted territory for us. Right now I'm mostly solving creative issues, which are a lot more fun to deal with than the technical ones I kept running into last year.

One thing we're changing this time around is how we generate the mouth animations. Magpie Pro worked amazingly well for Happy Box, but only when it wasn't crashing. I had frequent random crashes with Magpie Pro on three different computers but unfortunately the developer wasn't able to reproduce the crashes on his system. I really liked using Magpie Pro but its instability here is forcing us to move on to something else. :(

If anybody has any ideas why Magpie Pro is crashing so much for us, please post.

Right now I'm looking into using Toon Boom Animate Pro for creating hand-drawn lipsync animation. TBAP is actually a full-featured 2D animation program. I got it a while back to create hand-drawn Brudders cartoons for a later time, and it apparently has a good built-in 2D lip sync system similar to Magpie Pro. So far I've only read about it and it sounds like it might work for us.

Because this topic was recently discussed here (or in a related thread anyway,) I'll post info about how well TBAP works for us and if it's any easier/harder to use than Magpie Pro.

Also, this weekend I'll be posting more 'behind-the-scenes' stuff for Happy Box at the website and YouTube channel--some production art and maybe a couple of short tech videos.

[allabulle]

Thanks for the update! Please continue to do so. I'll be paying attention to your website too. :)

[goakes]

+1 This has been a very valuable thread, keep at it.

[Greenlaw]

Some interesting news from iPiSoft: Version 2.0 was announced recently and it should come out in a few weeks. So what's new with 2?

In April:

- Improved accuracy
- New 'actor' models, which means the software will be compatible with more body types, including female performers.
- Support for Kinect for Windows and Asus 'Kinect' clone. (Just added to 1.0 actually)
- Higher compression for improved efficiency.

Features announced for later this year (this is what really excites me about 2.0):

- Head tracking
- Props tracking (swords, rifles, sports stuff.)
- Multiple performers (Standard edition only.)

Price goes up a little for Standard to $1195 (two Kinects; six PS3 Eye cameras,) and goes down $100 to $295 for Express (one Kinect only.) (BTW, Express 1.0 is on sale now for $195 and upgrade to 2.0 should be free.) Basic remains $595 (two Kinects; four PS3 Eye cameras.)

Here's the press release:

http://ipisoft.com/V2_0_ann.html

The Basic edition is a great value now; throw in the cost of two Kinect Sensors and you have a pretty decent motion capture system for about $900!

Just thought people who have been following this thread would like to know.

G.

[erikals]

Head tracking,... was waiting for that one... :]

[Greenlaw]

Quick update on our projects: We made a lot of progress with the new Brudders short in January and February but then I got sidetracked by a mocap test for a friend's feature film, and then I had to go back to my job at R+H, so personal projects have slowed down again. Sigh!

For Brudders I was mostly working in Vue 10 so not much mocap stuff was happening anyway. At this stage I'm really tempted to switch back to RVJ right now because it would be a much easier film to finish quickly. Alisa insists we stay focused on Brudders though. She's probably right.

Right now I'm being sidetracked by a little mocap test involving my five year old, and it should be interesting to see how compatible iPi DMC is with 'little people'. Some of you may recall that over a year ago I shot a mocap test with my daughter just for fun. I wasn't surprised that tracking failed because at the time she was less than a meter tall. I told iPi Soft about it as a joke but I guess they took me seriously because a week later those guys adapted the software for capturing smaller performers.

So after a year of feeling guilty, I'm finally checking out this 'new' feature. Let you know how it goes. :p

G.

[erikals]

...Some of you may recall that over a year ago I shot a mocap test with my daughter just for fun. I wasn't surprised that tracking failed because at the time she was less than a meter tall. I told iPi Soft about it as a joke but I guess they took me seriously because a week later those guys adapted the software for capturing smaller performers.

So after a year of feeling guilty, I'm finally checking out this 'new' feature. Let you know how it goes. :p

G.

hehe, cute http://erikalstad.com/backup/anims.php_files/smile.gif
this will be fun http://erikalstad.com/backup/anims.php_files/smile.gif
 

[LW_Will]

I have a bit of a question.

I want to make a rig out of the Basic ipi software and two Kinects... (btw, two Kinects <referb> from Microsoft, $80 online... not too shaby) so I want to ask... what is the difference between the Basic and Standard? Near as I can tell, they both use the dual kinect setup. What is the difference?

Mocap for All!

[LW_Will]

Oops... just found my answer on the ipi sight... now, in my favor, it has been redesign, so... but the ONLY difference between the Basic and Standard is that you can add more cameras to the Standard... but not Kinects. So, with Basic and two Kinects, I appear to be golden!

Or will be golden...

[Greenlaw]

Yup, the only difference with version 1.0 is the number of allowed cameras. Both can use two Kinects, but Basic is limited to 4 PS3 Eye camera and Standard can capture from six PS3 Eye cameras. (Just to be complete, the Express edition only works with a single Kinect.)

The differences will become more significant when version 2.0 is released: iPi DMC Standard 2.0 will be able to capture two performers at one time. However, this feature is slated for later this year (3 qrt), not next month.

G.

[LW_Will]

Right... golden... ;-)

[Greenlaw]

Quick update on the 'kid-mocap' test: My daughter and I recorded the test motions this morning. The session was pretty funny, especially with the audio. This session was recorded using the 180 degree setup (two Kinects facing each other) and calibration went perfectly. I'll post a video of how this works soon.

We ran into a problem during tracking however--we found a serious bug in the current release that causes iPi to scale the actor from its center instead of its base, which causes the feet to rise above the floor if you scale the actor down. This may sound minor except I can't seem to translate the rig down on the Y-axis to align the feet with the ground plane. Curiously, if I scale the rig up, I can translate the rig upwards to align it with the ground plane. Anyway, in past releases, the feet were always aligned to the ground when you scaled the Actor, so this must be a new bug.

Now, it's possible that this issue has been going on for a while but I didn't notice before because I normally load a pre-existing Actor based on my own body proportions. The last time I tried to define a 'tiny' Actor in iPi Studio was the last time I tried this test, and that was over a year ago.

Oh, well. All I can do now for this test is wait for the developers to release a fix. More as things develop so stay tuned.

G.

[Greenlaw]

Figured out a few things.

First, the bug has existed for some time. I regressed from 127 to 119, and it's present there too. I think nobody noticed it before because most users are not scaling the iPi rig down to 1 m. (Geez, my daughter is tiny. Of course she was much shorter when we tried this last year so thank goodness she's growing.) :p

This bug probably explains an error some users reported earlier with floating feet. If you have to scale the rig much smaller than the default size, I can see how this will affect you.

Second, here's a workaround fix: Click Pose Refit. This throws your iPi Rig way off because it's nowhere near the performer to begin with. However, it also 'releases' the rig so you can translate it to align with the ground plane. It's not idea because you may need to adjust the limbs too.

Third...I can't explain this but sometimes it will allow you to scale down the rig and translate down on Y. Not always but sometimes it will.

Fourth, about that 180 degree thing...yeah, it's not nearly as stable as the 90 degree configuration. Part of the instability is because five year olds move like total spazzes, but I can also tell the constructed 3D volume is not nearly as 'dense' as when I do a 90 degree capture with two Kinects. I suspect the extra density comes from overlapping surface data.

We might have to shoot this again using the 90 degree setup.

More later.

[mrbones]

On the contrary Greenlaw/

Ive found 180 is more accurate.

http://www.youtube.com/watch?v=msZCsMh43JU

Now we just vibrate one of the kinects to remove inteference .

Tracks like a charm.

[erikals]

hm, quite good...!

[Greenlaw]

Pretty neat stuff.

I've been thinking about my results yesterday and I've come to the conclusion that what I'm actually running into here is the 30 fps limit with Kinect. I'm seeing a lot of the 'temporal' errors which must be equivalent of motion blur artifacting. This makes sense since their is an 'offset' and I think this increases to one side or the other as the performer moves closer to one device and away from the other. This type of error does not happen when using PS3 Eye cameras because these are 60fps devices and because PS3 Eye depends on RGB video rather than infrared.

TBH, the motion blur artifact is not huge problem for many motions but five-year-olds are insanely fast when they're excited, and I think this is breaking the limits of Kinect. That and the fact that iPi DMC's actor rig really isn't optimized for children yet. :)

If I can convince my daughter to do a few more tests (she's getting impatient with me already,) I want to do a 90 degree test. FWIW, I still disagree regarding 180 being more accurate; 180 resolves occlusion issues but it clear lacks the increased resolution due to merged data. Not that one configuration is better than the other, I'm just observing see strengths and weaknesses with each technique.

That said, what I really need to do is come up with a controlled comparison (meaning not my child's choatic kitty cat dance,) just to be sure--what I've been going with so far has been based on general obsveration and not controlled testing. I hope I can find time this week to do a repeatable 'stress test' between twin Kinect 90, 180, and six camera PS3 Eye configuration (which has full 360 coverage at 60 fps.) This should be a very interesting and telling comparison. More later.

G.

[mrbones]

HI Greenlaw,

I know what your talking about.

You can also manually adjust the calibration so that the 2 halfs of the depth are more aligned inside iPi.

Once the 2 depth movies are more aligned for the calibration then iPi works more efficiently and is faster. In addition we now shake the sensor to eliminate intereference in the initial scan.

make sure no loose fitting clothes or long hair is capped because those can affect the markerless tracking process.

For example braids or long hair affects the shoulder tracking.

Heres another succesful example..

http://www.youtube.com/watch?v=LfDeQr_vfvw&feature=share

[Greenlaw]

Yesterday, right before dinner time, my five year old daughter Sienna Kai and I decided to do a quick motion capture test in our living room.

Mocap Test with Small Child (1 meter tall) (http://bit.ly/HFd5oJ)

We used iPi DMC 1.0 with two Kinect Sensors positioned at 90 degrees. This is similar to the test we did last week using the 180 degrees setup but the result this time was more steady with far fewer errors. Also, at the full 90 degrees positioning, occlusion is clearly not an issue.

Both Sienna Kai and I are very pleased with this result. :)

G.

[LW_Will]

cute... and pretty good if it is raw. Is it?

Getting my second Kinect this week, btw... ;-)

[Greenlaw]

cute... and pretty good if it is raw. Is it?

Getting my second Kinect this week, btw... ;-)

Well, not completely 'raw' so to speak. Just to be clear, this how the mocap data looks in iPi Studio before exporting to other 3D programs. However, in this case I found I had to supervise the tracking process a bit more than usual. The main issues I ran into were motion blur artifacts (Sienna moves very fast and 'poppy', and at 30 fps, the trajectory of her limbs wasn't always clear to the tracker,) compounded by the fact that the built-in Actor rig isn't proportioned for child performers. That said, I'm amazed that iPi Studio could track her motions this well.

By contrast, most of the motions for 'Happy Box' were captured from an adult performer (me) and iPi Studio's tracker made very few errors, if any. 'Supervised' tracking in this case was basically: move the T-Pose in place, hit Refit Pose a couple of times, hit Track Forward, then save the result when it's done. (This does not include adjustments made after we retargeted the data to the cartoon proportioned characters--that was done in Motion Builder. This isn't the same as fixing tracking errors because it's character specific.)

BTW, if you haven't used iPi Studio before, 'supervised' tracking (i.e., error correction) is pretty straightforward. When you see an error occur you can do any of the following to fix it:

1. Go to a good frame and just track backward through the error. This is the easiest trick and sometimes it works the best. If it doesn't work...

2. ...Reposition the mis-tracked limb (iPi Studio supports IK and FK) and track through forward and backward through the 'offending frames. If that fails...

3. ...Reposition the mis-tracked limb and hit the Refit Pose button and then track trough the error forward and backward. Or...

4. ...Reposition the mis-tracked limb and hit Refit Pose, step through manually (< and > keys), and repeat as necessary through the error.

Once you get the rhythm for it, it goes very quickly. When you're done it's a good idea to 'bump' the Trajectory Filtering slider to smooth the transition between 'fixes'. Worst case scenario is to manually matchmove through the error but I've never had to do that. Honestly, if the footage is THAT bad, it's much easier to just record the motion again.

My typical workflow is to allow iPi Studio to process an entire Take from T-Pose to end frame by itself (unsupervised), and when it's done I'll play through the take and fix any errors I might see. Typically, if there are errors in a take, they can be corrected in a few a few minutes. However, in the case of this video it probably took an hour. These errors typically occurred when Sienna is on the floor with her legs tucked under and for brief moments it became difficult for the tracker to see where her legs supposed to be. Once I understood the error, fixing it was pretty easy.

When you're all done, you will need to run Configurable Jitter Removal. The real 'raw' data can be very jittery if you don't apply at least a little CJR. CJR is run in a layer on top of the mocap data so it's 'non-destructive.' This means, if you feel you used too aggressive a setting, you can simply dial it down and run it again. Also, using CJR is no cumulative, so you can run it as often as you like.

Typically, I dial in the strongest CJR setting for the legs, a little less for the torso, and the least amount for the arms. If you wish, you can vary the settings at different points in the Take but I usually run one setting for the entire take unless it's a special situation. If you use varying CJR settings in a Take, you need to 'bump' trajectory filtering to smooth out the transitions when you're done, otherwise you may see notice a 'twitch' at the frames where you changed CJR settings.

That's probably more info that you wanted to hear but there you go. :)

G.

[Greenlaw]

Here's an example of a tracking error I missed: If you scrub all the way to the end (about 1:32), you will notice that the Actor lowers its left hand a moment sooner than Sienna does. This doesn't really bug me but if I wanted to fix this error, I might...

1. ...go the frame where Sienna's hand is all the way down and track backward from that frame through the error. Or...

2. ...position the arm somewhere in the middle of the frame range around the error, hit Refit Pose and then track backward and forward from that frame to correct the surrounding frames. (Be careful not to track through the corrected frame.) Or...

3. ...if Refit Pose fails, just position the hand manually and track backward and forward to correct the surrounding frames.

One of these techniques should work, and it shouldn't take more than a few seconds.

Anyway, let me know when you get your second Kinect. :)

G.

[LW_Will]

That's probably more info that you wanted to hear but there you go. :)

G.

No. I want to hear EVERYTHING from the guy who invested the money on the setup that I am investing in... ;-)

my intent is you get the two Kinects, get the free software, and do a ton of mocap. (hopefully, I want to get a video trailer together to get some funding from funding types) I've done some work with iPi before, which is why I was so down on the original concept. But, your 2 kinect tests convinced my that is the way to go.

Will do about the Kinect!

[Greenlaw]

This is something I learned a only few minutes ago, and it corrects a mistake in what I wrote earlier about the Kinect: even though the Kinect is a 30fps device, motion blur is actually a very small issue for it because the device has short exposure times and its own light source. That said, the resulting data is so sharp, it can lead to another problem if you use two devices: spaciotemporal mismatch. I've seen this issue creep up occasionally and it can briefly confuse the tracker--the random mismatch in the two volumes was a little disturbing the first time I saw it. (As described in earlier posts.) That said, this happens rarely and I can't say it's been a big problem for me.

Just an FYI for anybody else who gets confused by this.

G.