View Poll Results: Keyframer - What needs to improve?

Voters
12. You may not vote on this poll
  • What is Keyframer?

    5 41.67%
  • I love it as it is!

    0 0%
  • I use it a lot.

    0 0%
  • I use it now and then.

    1 8.33%
  • It could be improved.

    2 16.67%
  • I never use it.

    4 33.33%
Multiple Choice Poll.
Page 2 of 2 FirstFirst 12
Results 16 to 26 of 26

Thread: Keyframer - What needs to improve?

  1. #16
    Super Member vncnt's Avatar
    Join Date
    Sep 2003
    Location
    Amsterdam
    Posts
    1,779
    Step 2: Pitch Analyser function in Legato.

    This is a voice track: white = frequency, cyan = amplitude;
    @ current time: 46.84 % average peak amplitude, and an average base frequency of 556 Hz.

    Click image for larger version. 

Name:	PitchAnalyser.png 
Views:	39 
Size:	220.8 KB 
ID:	147099

    Even without sound it's measuring the base frequency of the background noise!
    Measurement samples are stored in memory per 1/5th of a single frame (current frame rate = 25 fps).

    The amplitude data from Scene Editor is not available in LScript.
    That's why it's measured again.
    It's needed to distinguish vowels and consonants.

    The frequency range is corrected by pow(sample,0.5) to enhance visibility of the lower frequencies.

    The highest currently measured base frequency is about 10 kHz.
    That should be enough to detect 't' and 's' but that range is too large for visualizing vocal notes, generated by the human voice.

    To create a simple pattern of estimated vowels and consonants, I need to interpret frequency + amplitude to :
    - detect 'b' and 'd' in relative low frequencies
    - detect 's', 't' and 'f' in relative high frequencies
    - detect 'a', 'e', 'i', 'o', 'u', 'oe', 'eu', etc in relative mid frequencies
    - detect absense of voice to distinguish between words, 'd' and 't', and to suppress measurement of background noise

    That simple pattern of estimated vowels and consonants can be used as a guide during automatic positioning/stretching of text in the 'Speech' track, in the middle of the Legato viewport.

    The current implementation has limitations (like the lack of wav filename selection, or support of stereo wav files) that need to be addressed before posting a new build of Legato 2020.

    Anyway, I think this Pitch Analyzer function has a bright future to be used as a tool in lip-sync animation.

  2. #17

    Very cool.  

    Please post a test animation if possible, would be neat to see it in action!





    Last edited by erikals; 03-21-2020 at 01:09 PM.
    LW vidz   DPont donate   LW Facebook   IKBooster   My vidz

  3. #18
    Registered User
    Join Date
    Mar 2016
    Location
    Oxford, UK
    Posts
    1,069
    Yes, I only recently became aware of Legato, with no idea you have been quietly labouring away for years on this project. Congratulations on all the effort and also thanks for making Legato available to us.

  4. #19
    Super Member vncnt's Avatar
    Join Date
    Sep 2003
    Location
    Amsterdam
    Posts
    1,779
    Step 3: Hiss Detection function in Legato.

    Added a pre-sample LP filter to smooth measurements, Auto-FrequencyRange and Manual-FrequencyRange setting, red color for frequencies > 1.500 Hz, muted envelope luminosity for low-level amplitude, support 8bit wav, and basic support for stereo (currently left channel only).

    Click image for larger version. 

Name:	HissDetection.jpg 
Views:	24 
Size:	64.2 KB 
ID:	147111

    The voice (no music) is singing "The Night". The envelope color is now red during relative high base frequencies in this audio: inhale, "Th", "gh", and "t".
    I'd like to add a sub-LF filter to detect explosives like "B", and "P", and a flip-flop to slightly group the detected hiss moments.
    Also need to add more controls to Legato Remote Control.


    Quote Originally Posted by erikals View Post
    Very cool.  

    Please post a test animation if possible, would be neat to see it in action!





    Quote Originally Posted by TheLexx View Post
    Yes, I only recently became aware of Legato, with no idea you have been quietly labouring away for years on this project. Congratulations on all the effort and also thanks for making Legato available to us.
    Thanks guys.

    Hotspot selection in Legato Remote Control.

    This feature works (save/load also) standalone, without connecting to Legato.
    New: add, modify hotspot parent, add new parent (not linked to other hotspots), modify the Item that is linked to that hotspot, move selected hotspot, move all hotspots, and the drawing of lines between parents and child hotspots.

    Click image for larger version. 

Name:	HotspotSelector.jpg 
Views:	34 
Size:	70.8 KB 
ID:	147112

    I need to look at an issue with following selections in Layout itself. Probably looping.
    Maybe want to add a magnetic raster to place hotspots more predictable.

  5. #20
    Super Member vncnt's Avatar
    Join Date
    Sep 2003
    Location
    Amsterdam
    Posts
    1,779
    Before continuing with the High-Pass and Band-pass filter, I'm experimenting with
    the Low-Pass filter to suppress incorrect samples, caused by strong harmonics.

    The 16bit stereo wav file contains a vocal glissando, the left and right channels are identical,
    the left LP filter is turned on, and the right LP filter is turned off.
    In here, the Left- and Right channels are presented in the same graph.

    Click image for larger version. 

Name:	3band.png 
Views:	22 
Size:	36.9 KB 
ID:	147164

    According to the blue envelope, I forced my voice to start at 96 Hz
    and stopped at 750 Hz (I won't share that file).
    At the extreme low and high frequencies, I wasn't able to produce a sound without strong harmonics (as in screaming, see amplitude envelope!).
    Without LP filter, it's impossible to reliable measure that kind of signals with the current measuring algorithm.

    Current frequency display range is 10 Hz .. 6 kHz, values are drawn Log10.
    Parameter overSample is set to 1. That's 1 sample per frame.
    If set to 0.25 (1 sample per 4 frames), the remaining sample errors in the blue envelope (caused by harmonics that passed the LP-filter) are smoothed out even more!
    If set to 5 (5 samples per frame), the current implementation is unable to detect frequencies below 125 Hz (5 x 25 fps).

    The frequency graph has been split into 3 envelopes:
    1. red: highest (selected) frequency, per frame (it needs a HP filter)
    2. green: average frequency, per frame (it needs a BP filter)
    3. blue: lowest (selected) frequency, per frame


    The 2 cyan envelopes at the bottom are the left/right amplitude values, values are drawn linear.
    Their values are sampled after the LP filter. Since the LP filter of the right channel is turned off, the right channel contains more HF components and is more loud.

    Parameter overSample can be set between 0 and 1 to be able to measure frequencies below the (25 fps) frame-rate.
    For explosives like 'p', and partly 'k', 'j', it can go as low as 10 Hz.

    In this implementation a high overSample value would severely limit low frequency measurements.
    When set to 5 or larger, it's already influencing the measurement of regular male speech, at 25 fps frame-rate.

    Next on my list is to implement a general Low-Pass filter to remove errors from all frequency bands.
    Then, insert LF/MF/HF filters for all three envelopes to distinguish between: explosive consonants (10..100 Hz), voiced consonants + vowels (100..1500Hz), and unvoiced consonants (> 6000 Hz).
    Last edited by vncnt; 03-29-2020 at 09:25 AM.

  6. #21

    nice, looks like Eathorne Ghettolink also is working on something...
    https://www.facebook.com/groups/ligh...95435433895774

    LW vidz   DPont donate   LW Facebook   IKBooster   My vidz

  7. #22
    Super Member vncnt's Avatar
    Join Date
    Sep 2003
    Location
    Amsterdam
    Posts
    1,779
    Many other solutions seem to be limited to morph envelopes, with or without automatic timing.

    Legato's pose/animation Library is a universal solution for body/face rigs.
    It's an additional layer, on top of existing envelope manipulation systems of LightWave, that makes it easy to interchange poses/animations between lws scene files.
    Remove the Legato plugin from your lws scene file, and nothing happens. All motion data (even in the Library section) remains safely in the envelope data of each 3d item. Only Legato's references to frames, items, hierarchies, groups, text, etc are gone (unless you've exported the plugin data to a safe place on your hard disk).

    The Audio Analysis function is a companion function to the Legato Library.
    In Legato, I'm trying to create a universal method (language independent) that suggests timing, for an important part automatically, by identifying the most important cues in the speech audio track.
    These cues can be utilized to align text on the text track with the audio track, manually or (in the future) as much as possible automatic.

    Audio cues + text cues together should result in a pattern that has enough detail to guide me to manually set extremes, manually anticipate on facial expressions, and manually determine inbetweens. With or without poses from the Library.
    The pose type track can be used to keep track of the animation structure, for future modifications and reuse.

    Notice I've used the word manually a lot.
    That's because automatic systems (like Auto-Focus, Auto-WB, Auto-Iris) tend to have a life of their own, at the wrong moments.

    The end result is a tool box with tools that complement each other.
    Last edited by vncnt; 03-29-2020 at 04:33 PM.

  8. #23
    Super Member vncnt's Avatar
    Join Date
    Sep 2003
    Location
    Amsterdam
    Posts
    1,779
    Audio Analysis function almost ready for download.
    Seems compatible with LW2019 and LW2015.

    Click image for larger version. 

Name:	AudioAnalysis.png 
Views:	24 
Size:	213.7 KB 
ID:	147199

    Open issues for next release:
    - Audio Analysis settings confirmation from Legato to Legato Remote Control
    - save settings in 'Audio Analysis' tab of Remote
    - separate amplitude graphs for LF/MF/HF bands
    - handling of extreme values in graph

    I haven't developed code yet for interpretation of sounds into predictive patterns, to identify vowels and voiced/unvoiced consonants, or to match text with the audio track.

    That's for another release, somewhere in the future.

  9. #24
    Registered User caralvive's Avatar
    Join Date
    May 2006
    Location
    Los Angeles
    Posts
    23
    [QUOTE=erikals;1589756]
    nice, looks like Eathorne Ghettolink also is working on something...
    https://www.facebook.com/groups/ligh...95435433895774

    I'm working on an upgrade from 2D to 3D of my animated series "Carlin Cartoon" in lightwave. I made a set up for the face of my characters (I still want to keep the original 2D faces) using D-storm's YS Character Setup plugin. I'm also using the plugin as a character picker and for a few Lightwave's shortcuts to speed up my forkflow. It is working perfectly. I'm planning to ad the MG-Papagayo Lip syng to this workflow. I'm using RH RHiggit and IKB for simple BG character animations.
    Hoping to have the first 3D episode comming soon!

    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	Carlin 3D Test.png 
Views:	17 
Size:	879.5 KB 
ID:	147281  
    Last edited by caralvive; 04-15-2020 at 05:31 AM.

  10. #25
    Super Member vncnt's Avatar
    Join Date
    Sep 2003
    Location
    Amsterdam
    Posts
    1,779
    Working towards a final version for latest additions: Selector and Audio Analysis.

    Resizable hotspots, multi-branch, multi-character, add/delete, re-assign hotspot to selected item, free move + raster move, single move + all move, lock hotspot definition, Tool assignment (8 different tools, activated upon item selection), stretching interface (+ full screen), Legato ecosystem:
    Click image for larger version. 

Name:	Selector.png 
Views:	21 
Size:	231.3 KB 
ID:	147376

    Audio Analysis: mono/stereo + 16bit/8bit file support, mono/mix/stereo graph, filters optimized for 48kHz/44.1kHz, noise gate, oversample (+ undersample), Low/Mid/High frequency band + amplitude, frequency range presets, Legato ecosystem:
    Click image for larger version. 

Name:	Audio.png 
Views:	23 
Size:	188.1 KB 
ID:	147380

    The simplicity of the static filters makes it unsuitable for speech recognition but it does produce some interesting patterns that may help navigate a sound track. Results will vary. For example, a recording of speech seems more predictable than a vocal track that is produced with heavy compression filters.

    Thanks to additional downsample algorithms I was able to improve my filters. On solo tracks, the yellow MF dots did follow the fundamental frequency more accurate than I expected - but not as good as Neptune or Auto-Tune.

    It has been fun to play with. At least it can visualize stereo tracks with separate graphs for ch1 and ch2. Native Lightwave doesn't handle stereo very well. Maybe interesting for dialogs.
    Last edited by vncnt; 04-22-2020 at 04:07 PM.

  11. #26

Page 2 of 2 FirstFirst 12

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •