Page 1 of 3 123 LastLast
Results 1 to 15 of 41

Thread: Devices to enhance Kinect capability for mocap

  1. #1

    Devices to enhance Kinect capability for mocap

    Something I'm going to be working on over Febuary in regards to Nevron is basically to answer this question:

    "What devices could one use to enhance motion capture even further? Are there devices or parts of devices (preferably low cost) that could be feasible and practical for the animator's workflow?"

    Currently, I'm looking at the feasibility of using a Wiimote motion plus (because you can get it insanely cheap) to allow motion capture that tracks the rotation of the user's wrists and head, as I feel it would greatly cut down on hand-animated "post work" required after recording full body mocap. If it works in a usable fashion, it'll mean for about $40 users of Nevron can add more rotation tracking capability to their kinect setup. I will report my findings here after I receive my hardware in the mail. I know a bit of working around will be required to get this functional and the possibility it may not work at all so we'll see.

    I do know the PS move devices work well from what others have said... but from what I understand that route is incredibly expensive and only really benefits people who already own a PS3 and the corresponding camera. EDIT: Ok, so from the post below by Greenlaw, it appears only the PS move controller is needed... still more pricey than the wii motion controller, but knowing this makes it much, much more feasible.


    OTHER DEVICES:

    -GAMEPAD:

    This can be used to control the character's hands if pre-animated constraints like cyclist are used. Thus far, my tests with some junker playstation 2 controllers (with a USB converter) have yielded some pretty nice results as they are decent quality devices... the most useful of course being the thumb axis rotational sticks. My current idea is that, just like with facial capture, hand animations driven by controllers could be made much easier to create if the mocap clip is playing at 1/3rd of its original speed... this gives the user more time to react and manipulate the character appropriately.

    That's the only device i've really tried out thus far... the only other devices I have only return boolean values so i'm not sure how I could make use of that. If I could find a way to refresh layout and make the calculations apply repeatedly over time, theoretically booleans could be used, for instance, to make it so that when a button is held down, a value will increase until the button is let go, then it will decrease until it reaches the "floor". I don't know how to make this happen yet or if its even possible though...
    Last edited by Ryan Roye; 01-24-2014 at 09:43 AM.
    Professional-level 3d training: Ryan's Lightwave Learning
    Plugin Developer: RR Tools for Lightwave

  2. #2
    Eat your peas. Greenlaw's Avatar
    Join Date
    Jul 2003
    Location
    Los Angeles
    Posts
    7,136
    I think this could be a very informative thread. Thanks for starting it.

    Quote Originally Posted by chazriker View Post
    Currently, I'm looking at the feasibility of using a Wiimote motion plus...to allow motion capture that tracks the rotation of the user's wrists and head...I do know the PS move devices work well from what others have said... but from what I understand that route is incredibly expensive and only really benefits people who already own a PS3 and the corresponding camera.
    Technically speaking, you don't need the game console or camera to use PS Move if you're only recording rotation--all you need is a bluetooth adapter. That's how iPi Soft did it for their mocap system to record head and wrist rotations, using only the PC and a cheap USB bluetooth adapter. For mocap you don't need the position data because you already have it in the bones position--all you need is to overwrite the rotation data. I don't know how you would do this for LightWave but I imagine it's possible. BTW, iPi Mocap studio can use the WiiMote in the same way. The PS Move's magnetometer is more accurate though. To record wrist motions, some users have made gloves that hold the device to the back of the hand--I prefer to hold mine because I like to use the buttons to remote control the software. For the head rotations, I modified a hard hat--you can see the rig here:

    My DIY Mocap Helmet

    The PS Move data is recorded simultaneously with the body capture. After the body motion has been tracked and edited, you simply apply the rotation data from each device to the appropriate joints (I use three, one for each hand and one for the head.) It works surprisingly well--I used it recently for background character mocap for a feature film, as well as in our Brudders music video.

    Here's another option: there's a video on YouTube that shows a user recording rotation in LightWave using an iPod Touch:

    Using the iPod Touch in LightWave

    The Myo looks really cool. It's an arm band that uses EMG sensors to read what your muscles are doing and it's supposed to accurately interpret forearm, wrist and finger movements. iPi Soft has been looking into that but the device is not officially out yet. Soon though and it's surprisingly inexpensive. Here's more info:

    Myo

    The existing Kinect system for LightWave Nevron Motion shows interesting possibilities for alternative data input. I've been curious to use it puppet other channels in LightWave besides the usual ones but haven't had time to explore this yet.

    G.

  3. #3
    Thanks for the wealth of info! And the DIY helmet setup is really slick and I'll see if I can dig up my old hardhat from the days when I worked as a road traffic operator (basically, stand there and hold a stop sign for 6+ hours). If not, I know someone who has a bunch in their closet

    I'm mainly seeing the feasibility of wiimotes first due to their low cost. The biggest question there would be whether or not it is good enough to simply track the bank rotation (or calculated equivalent via formula) of the user's hand and nothing else... I can settle for less accuracy if the results are still usable. Of course, I'll also test it for head rotation as well. I got my test Wii motion plus as low as $14. If I get usable results, I'll post a video showing my findings.
    Professional-level 3d training: Ryan's Lightwave Learning
    Plugin Developer: RR Tools for Lightwave

  4. #4
    Eat your peas. Greenlaw's Avatar
    Join Date
    Jul 2003
    Location
    Los Angeles
    Posts
    7,136
    Sure. I'm hoping others will add to this thread--it could become a very interesting and useful resource.

    Re: the hard hat--it's really cheap. I bought this one at a hardware store for about $6 I think. I had to track down the chin strap on the web though--with shipping it probably cost more than the hard hat itself.

    Another iPi Mocap Studio user found a removable suspension that you can wear without the outer shell, and he just tie-wraps the motion device to that. It's probably less bulky but the hard hat version I have works well for me.

    Tip: if you use this hat with Kinect, paint it a flat gray so it doesn't bounce the IR rays. I unfortunately used a paint that doesn't bond to the plastic very well, and the paint is chipping and flaking now. I think Krylon Fusion spray paints will bond to plastic--I'm going to try that soon for the hard hat and for painting 3D printed parts.

    My bicycle helmet actually worked better, and it looked cooler too, but I needed to use it as, well, a bicycle helmet.

    Here's what that version looked like:

    DIY Mocap Helmet - Bicycle Helmet Version

    G.

  5. #5
    Eat your peas. Greenlaw's Avatar
    Join Date
    Jul 2003
    Location
    Los Angeles
    Posts
    7,136
    I think any wrist motion should look better than no wrist motion (i.e. 'paddle hands',) so long as the wrist is moving in a natural direction and with reasonable limits. With iPi Mocap Studio, it's remarkable how much realism you gain by including the wrist and head rotations.

    Here's a little more info about how it works in iPi Mocap Studio. When recording with a Move or Wii device, you need to point it directly at one of your Kinect devices for a few frames. This gives the software a reference for device heading, relative to the scene and performer's arm/head. Then, after the body has been tracked, you parent a 'virtual' version of the device that has the recorded motion to the appropriate skeleton bone (hands or head) and orient the device to how it was positioned in the real world relative to that body part. You can do this on any frame where the correct orientation is very clear (having the RGB video data from the Kinect is helpful here.) When you're ready, simply click the Apply button and the data is transferred to the bone the device is parented to. Play your animation to check the results. Usually, it should work fine the first time you apply it but if the rotation axis looks off, you can make an adjustment and apply it again--the software will overwrite the previous motion for you.

    I'm not sure how this method can be translated for Lightwave/Nevron Motion but it works very well in iPi Mocap Studio.

    G.

  6. #6
    Super Member spherical's Avatar
    Join Date
    Dec 2004
    Location
    San Juan Island
    Posts
    4,686
    We have yet to get into NM. Right now, my head is swimming with all of the terms and concepts being thrown about, but I'm sure things will fall into place once some experience has been gained. A couple of initial questions, though. We have a PS3 and a Move Controller. Up above you mention a Bluetooth adapter. Adapting from what to what? Do you have any shots of the wrist set up and can more than one Move Controller be used at the same time?
    Blown Glass · Carbon Fiber + Imagination

    Spherical Magic | We Build Cool Stuff!

    "When a man loves cats, I am his friend and comrade, without further introduction." - Mark Twain

  7. #7
    Eat your peas. Greenlaw's Avatar
    Join Date
    Jul 2003
    Location
    Los Angeles
    Posts
    7,136
    Oh, sorry...I was speaking theoretical as far as LightWave/Nevron Motion goes. The whole bluetooth/PS Move/WiiMote thing works with iPi Mocap Studio, not Nevron Motion.

    I haven't spent much time with Nevron Motion yet but I believe the only direct input currently available is what's available from the Microsoft API. I don't know enough about LightWave's Virtual Studio to tell you what's possible there either.

    G.

  8. #8
    Eat your peas. Greenlaw's Avatar
    Join Date
    Jul 2003
    Location
    Los Angeles
    Posts
    7,136
    But as far as how it works in iPi Mocap Studio, it's as described above. I don't have a video of a workflow posted online but the B2 Music Video Excerpt we released last fall features wrist and head motions captured with 3 PS Moves for each character. I have to admit the wrist movements aren't very interesting because the characters are performing mostly repetitive motions, but you can see what it did for head motions fairly well.

    A much better example would be the stuff I did recently for background characters in a film but I can't show that. When I have time, I'll make a quick workflow video with a more generic example.

    G.

    Edit: wait, I take that back. I have a partial workflow video online. The Sister Mocap Test from last spring shows a little bit of the PS Move in action. If you look at the live action in the first part of the video, you can see that I'm holding two PS Move controllers. The bluetooth adapter is on the end of a repeater cable on the floor near the capture space. The wrist motion was applied to the character in the second and third parts. It's not perfect because this was literally the first time I used the feature and I didn't know what I was doing. If I was doing it again, I would correct the orientation and reapply the motion, but at the time I was just excited that it worked at all.

    I don't have the head rig in the video because that feature was added a couple of weeks after I shot this.

  9. #9
    Super Member LW_Will's Avatar
    Join Date
    Feb 2003
    Location
    Somewhere, Out there...
    Posts
    1,188
    Hmm... seems like some Duck Tape would do okay to me. Just a bit sticky after the chin... ;-)

    My one question is, how can I adapt the output of LW to work on my iDevice or Nexus7? If I could make a floating camera rig, THAT would be amazing!
    Will Silver
    Animator
    digital TRiP
    "Frak'em all! I USE LIGHTWAVE!"

  10. #10
    Eat your peas. Greenlaw's Avatar
    Join Date
    Jul 2003
    Location
    Los Angeles
    Posts
    7,136
    If you manage to make your iDevide or Android device work with LightWave, I think you will only get rotation data, no translation. Apart from the current LightWave Virtual Studio system using a PS Move and game console, I'm not sure. Without the console, you can get device rotation but the tricky part is getting translation, which is what the game console and cameras are doing. Without that, you will need an additional controller to navigate position.

    A few years ago in the Box, our programmer experimented with using a Wii Motion Plus in Maya to behave like a camera. The Wii uses its game console and an infrared sensor to capture translation. This system has potential as an alternative to PS Move console and cameras and it's a lot cheaper. Anyway, he somehow got it working in Maya and then the motion was ported to LightWave. I don't know if you can use Wii Motion Plus with LightWave but I imagine some smart person could probably make it work with Virtual Studio. The downside is that the Wii is old tech and I don't think anybody will want to put much effort into this. Sorry, I have no further information about this.

    The Space Navigator gives you movement in all six directions (XYZHPB) and it does work with LightWave right out of the box. It's not quite the same thing though because it doesn't have the 'handheld' interaction you get with a PS Move. Maybe somebody could combine the two devices? That is, use the PS Move with bluetooth to capture rotation and use the Space Navigator (or even just the keyboard) for XYZ motions? Just a thought.

    In the foreseeable future, I think the best bet for a 'low cost' handheld virtual camera, beside the PS Move with console, might be Leap Motion, or maybe even the Kinect 2--but it's too early to say.

    I have to confess, I actually enjoy setting up my own camera choreography, so virtual camera isn't for me. But I know a lot of other users have been asking for this for some time.

    G.

  11. #11
    Eat your peas. Greenlaw's Avatar
    Join Date
    Jul 2003
    Location
    Los Angeles
    Posts
    7,136
    I just remembered this:

    https://www.lightwave3d.com/assets/p.../w_gamedevice/

    Don't know if it still works but maybe somebody will want to try it out.

    G.

  12. #12
    Super Member LW_Will's Avatar
    Join Date
    Feb 2003
    Location
    Somewhere, Out there...
    Posts
    1,188
    Looks very cool... except that it doesn't use the Virtual Studio. ;-(

    Also, the thing I want is the monitor on the iOS/Android (N7) to show a (presumably VPR) view!

    I've seen a video from 3rd Powers, awhile ago, where he was manipulating LW on a tablet AND a screen. I want to do that! ;-)
    Will Silver
    Animator
    digital TRiP
    "Frak'em all! I USE LIGHTWAVE!"

  13. #13
    Super Member dwburman's Avatar
    Join Date
    Feb 2003
    Location
    Washington State, USA
    Posts
    2,880
    Quote Originally Posted by spherical View Post
    We have a PS3 and a Move Controller. Up above you mention a Bluetooth adapter. Adapting from what to what? Do you have any shots of the wrist set up and can more than one Move Controller be used at the same time?
    The bluetooth adapter would be to use the Move Controller without the PS3 by connecting it directly (wirelessly) to the computer like you would a mouse or keyboard or any other bluetooth device.

    - - - Updated - - -

    Quote Originally Posted by Greenlaw View Post
    I just remembered this:

    https://www.lightwave3d.com/assets/p.../w_gamedevice/

    Don't know if it still works but maybe somebody will want to try it out.

    G.
    I used that a bit a few years ago, but I don't think you need it anymore with the virtual studio tools. It might be slightly easier to set up, though.
    [mr] Dana W. Burman | http://dwburman.com
    Motion Graphics for sale at Pond5 Stock Footage
    Check out my tutorials at Liberty3d.com & my YouTube channel!

  14. #14
    Super Member spherical's Avatar
    Join Date
    Dec 2004
    Location
    San Juan Island
    Posts
    4,686
    Quote Originally Posted by dwburman View Post
    The bluetooth adapter would be to use the Move Controller without the PS3 by connecting it directly (wirelessly) to the computer like you would a mouse or keyboard or any other bluetooth device.
    Ok, so the PS3 isn't needed at all or, if you have one, does it help any?
    Blown Glass · Carbon Fiber + Imagination

    Spherical Magic | We Build Cool Stuff!

    "When a man loves cats, I am his friend and comrade, without further introduction." - Mark Twain

  15. #15
    Eat your peas. Greenlaw's Avatar
    Join Date
    Jul 2003
    Location
    Los Angeles
    Posts
    7,136
    As explained above, the PS3 is needed for capturing positional data of the PS Move controllers. This is done using the PS3 Eye cameras to track the illuminated balls I guess. (Sorry, I don't have a PS3 so I have no experience with this.) In theory, because all of these devices can connect to a PC, I think somebody could write software to track the balls on a PC while simultaneously recording the rotation data by bluetooth but I'm not aware of anything like that. (In the case of the Wii, it does this using the IR bar that sits in front of you.)

    What iPi Mocap Studio does is it records only the rotation data from one to three PS3 Move controllers via bluetooth directly to the PC, while at the same time it records a point cloud of the body with one to three Kinects or up to eight PS Eye cameras. It doesn't need the PS3 or Wii consoles for this, just the PC and a bluetooth adapter because the PS Move and Wii Motion Plus are bluetooth devices. Then, after a skeleton has been tracked, you apply the rotation data to the wrist and head bones. There is no need to record positional data in this case because you already have the position from the bones.

    The position data is needed for 'free floating' objects, like a virtual camera.

    I guess theoretically speaking, you could mocap a person to for the positional data and then apply the 'camera' rotation but IMO, the data isn't nearly accurate enough to make this work for accurate camera motions. (Not that there's a setup for that anyway.)

    To summarize, it appears the best low cost solution for LightWave for a virtual camera that behaves like a real hand held camera (rotation and position) is still the PS Move with a PS3 console. You can use cheaper controllers like a gamepad or Space Navigator but obviously that's not the same as using a 'hand held' camera--it's more like running a video game.

    Then there's the rotation only method (a device with accelerometer or magnetometer, like iPod Touch, Wii Motion Plus, or PS Move) which can get some natural motions but with obvious limitations (but you might combine this with another controller or maybe a 'nunchuk' attachment for position.) I don't know anything about how to do any of this though.

    G.

Page 1 of 3 123 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •