PDA

View Full Version : LightNet: Cloud v1.5 Released!



JoeJustice
04-17-2011, 01:56 PM
I just released Lightnet: Cloud version 1.5. You can grab it here:

http://www.joejustice.org/books/lightnet-cloud

This is a significant update with a lot of new features and a totally new file transfer system under the hood. Most of the documentation still applies as it is, but I will be updating things is the coming weeks. Here is the the change log entry:

v1.5

IMPORTANT UPGRADE INFORMATION:

To use v1.5 with an existing v1.0 configured server, you will need to manually change the name of the configuration (.ini) files. Simply add “0_” to the beginning of the files. So that lnc_onsite.ini will become 0_lnc_onsite.ini and lnc_offsite.ini will become 0_lnc_offsite.ini

* Totally new dynamic FTP System

The biggest change is the totally re-written FTP system. This system is dynamic and offers fantastic user feedback on what is going on. Now files move back and forth as LnC is working on them, rather than it completing a process and then sending/receiving files.

This does come at the price of making the entire system much more complex, so I can't confidently guarantee it will work with every FTP server out there. There is a fair amount of diversity among FTP servers and automatic, dynamic data transferring is much more complex than procedural data transfers.

With that said, I have attempted to limit any kind of compatibility issues by doing a lot of standardization techniques. For example some FTP servers have difficulty with filenames that contain spaces so I programmed LnC to systematically replace all spaces with underscores on scenes and content zip files (it also updates the scene file with the new content archive file's name if it has to rename it). LnC also makes all filenames lowercase.

I have done extensive testing with vsftp (Linux), Pure-FTPd (Unix) and the Filezilla FTP server (Windows). LnC performs properly with each.

If, while rendering, LnC drops it's connection from the FTP server, for example times out due to inactivity or there is an interruption in your network connection, don't sweat it; on the next check interval LnC will attempt to reconnect and will pick up right where it left off. It will continue to reconnect on the check interval indefinitely.

I have added “0_” to the queue configuration (.ini) files so they are easier to find in file listings – they should be at the very top now.

* Improved interface

Watcher has a grid view for information rather than a basic text display along with improved FTP progress bar.

Cloud has FTP message box along with improved FTP progress bar and information.

* Keyboard shortcuts

I have added several keyboard shortcuts. (Version 1.0 had no shortcuts)

Ctrl-C: Standard copy keys, will copy selected text in a grid or text box

Ctrl-V: Standard paste keys, can be used in Watcher's command boxes

Ctrl-S: Launches the Screamer

Ctrl-F: Forces an immediate server check in both Cloud and Watcher (Watcher also has a force check button, but Cloud does not)

Ctrl-Q: Sets LnC to finish what it is doing and then shutdown (Quit).

Alt-C: Change settings (In the settings panel You can use the tab key to move from one field to another, Enter to accept changes and Esc to cancel changes)

Alt-L: Load Setting

Alt-S: Save Settings

Alt-D: Run Diagnostics (Both Enter and Esc close this panel)

* New shutdown procedure

LnC now has a new shutdown procedure. If you attempt to close it while it is doing anything (uploading, rendering, downloading) you will be prompted how on you would like to shutdown.

You can choose to cancel, which will do nothing.

You can choose to shutdown immediately, which will simply close it and it's screamer, meaning rendering will stop and any files currently uploading or downloading will be stopped.

You can have LnC finish what it's doing and then shut down. This will close the screamer when it is finished, upload any files that are pending and then, when everything is finished, close the program.

* Self maintained local working directory

There is no longer a setting for the Local Working Directory. Once started, LnC will create it's own LWD in the directory it is currently in. It will then manage this directory itself. Whenever a new scene is loaded, it will automatically deleted all of the files used by the previous scene, keeping only a few necessary files. You should never have to do anything in the LWD, or even know it exists, for that matter.

This does not effect in any way the Watch Folder. That is still a settings, so that you can have it as a local folder or a shared, network folder.

* Option to preserve output filename

Note that v1.5 does come with a new LScript file that has an option to preserve the output filename. In v1.0 LnC automatically changed the output filename to the name of the scene, with this option enabled, it will not change the filename.

The final frames will still be uploaded to the FTP server, so the path you set for the output files in the Lightwave scene is totally irrelevant, but with this option, the name will be preserved.

* Date and time stamp added to scene files upon upload via Watcher

To avoid any name collision issues, all scenes are date and time stamped. This way you can upload the same scene multiple times without it overwriting the previous one. A corresponding output directory, with the date and time stamp, is created on the FTP server. The output frames do not receive the date and time stamps.

This way you can upload a low quality scene, then change it to a high quality scene and upload the same scene file and not worry about it overwriting the low quality one. Use the low quality output for editing and timing while you wait for the high quality to finish. When it's complete, simply replace the low quality frames with the high quality frames.

In conjunction with the preserve output filename feature you can easily render many different versions of a scene without having multiple scene files. For example, if your scene has multiple cameras you can select a Camera1, make the output filename “Camera1” save and upload, then select Camera2 make the output filename “Camera2”, save and upload.

Content archive files are NOT date and time stamped, so if you upload a zip file that has the same name as a zip file already on the server it will overwrite the zip file already on the server. Note that multiple scenes can use the same content zip file.

* Dynamic content archive work-flow for offsite rendering

Content archives (zip files) are re-downloaded and localized each time a new scene is downloaded. This work-flow is designed to allow content archives to be dynamic without having to constantly change the filename. Multiple scene files can use the same content archive; by re-downloading the zip file with each new scene, it allows you to update the zip file without having to change the scene files every time you change the content archive or worry about whether the individual computers have downloaded the latest version of the content archive.

archijam
04-17-2011, 03:03 PM
Hi Joe. If you still need help organising the docs, let me know.

jmelsom
-
at
+
gmail
dot
com

:)

edit: and congratulations! :thumbsup:

zapper1998
04-17-2011, 05:02 PM
This is awesome... Thanks will try it out ...


Be cool would it not all members pool together, and use it... All the Kibbles for da puppies to munch on.....

I have 4 nodes in the garage .... hmmmm
interesting idea...


Mike

omichon
04-18-2011, 01:18 AM
Awesome news, Joe ! I was expecting some of these features for a better workflow. Now it's there, ready for testing :)
Thanks for your great job !

sandman300
04-18-2011, 11:58 AM
I just started to put together a render farm, this update is perfect timing since half of the nodes are Lenix.

JoeJustice
05-02-2011, 09:45 AM
Hey guys, sorry I haven't posted any follow-ups. Been real busy the past few weeks.

I was correcting a few bugs with v1.5 and ended up adding a new feature. I just posted LnC 1.5.3!

For one thing I found out my build was not working with Windows XP. No one has complained, but I gotta figure there are some folks still using WinXP than could use LnC. So this update should take care of that, I've ran it on several old XP machines I have and all appears to be working.

I added a Local Network Cache setting that should save you a lot of bandwidth if you have multiple offsite computers on local networks like I do. Check out the change log for details:

http://www.joejustice.org/books/lightnet-cloud/change-log

omichon
05-02-2011, 10:05 AM
Thanks for the update, Joe !
I had some strange behavior with 1.5 on win7-64 (lnC disappearing with no reason), but had no time to investigate and report about it, unfortunately.
So I am back to 1.0, but I'll try this new version as soon as I have some time for it.

stiff paper
05-02-2011, 03:10 PM
Joe?

First off, thanks for making it work with XP. I'll stop using XP when they pry it out of... etc.

Is it 32 bit Windows only? I'm looking for the info but I haven't found it yet...

calilifestyle
05-02-2011, 03:18 PM
So is lightnet cloud the new version of lightnet classic

JoeJustice
05-02-2011, 04:04 PM
Cardboard:

The previous version only worked with Windows 7, 64-bit. This one will work with XP-32, XP-64, Vista and 7.

calilifestyle:

Yes and no. I've ceased development on Classic because it would require a total re-write in order to update it to a modern state. Since I was going to have to re-write it anyhow, I thought I'd just write something new with a more modern approach. While I intended to cannibalize LightNet Classic, I ended up writing LightNet: Cloud totally from scratch.

So it's kind of the descendant of LightNet Classic in spirit, but really is something totally new.

m.d.
05-02-2011, 05:03 PM
will light-cloud render out image filter plug ins like exrTrader ? or is it restricted to the standard render globals options

thanks

Mike

p.s. I did PM you the same question...but knowing how much most people check there inbox I thought I should post it here as well

05-02-2011, 05:06 PM
Very nice, I look forward to testing this out when I have some free time.

JoeJustice
05-02-2011, 06:30 PM
m.d. I think you'll have trouble with exrTrader. I have not tested it, but LnC is designed to work with LightWave's output settings.

Lightwolf
05-02-2011, 07:02 PM
m.d. I think you'll have trouble with exrTrader. I have not tested it, but LnC is designed to work with LightWave's output settings.
I haven't tested it either.
But that implies that it'd work with the dummy saver in conjunction with exrTrader using the render path and precisely the same frame numbering settings as LW is set to in the render globals (which exrTrader can't, unfortunately, read out automatically).
That and not saving any other images from exrTrader.

@Joe: If you're interested: I've got docs on the exrTrader settings format for developers and I could also send you a free license.
Mail me: michael(dot)wolf(at)db-w(dot)com.

Cheers,
Mike

Cageman
05-02-2011, 09:06 PM
Hmmm....

230 Logged on
200 Type set to I
LnC: Uploading: lnc_test.txt
LnC: Waiting on file data on lnc_test.txt
250 CWD successful. "/" is current directory.
227 Entering Passive Mode (192,168,0,20,215,12)
150 Connection accepted
226 Transfer OK
LnC: Waiting on file data on lnc_test.txt
250 CWD successful. "/" is current directory.
227 Entering Passive Mode (192,168,0,20,215,14)
150 Connection accepted
-rw-r--r-- 1 ftp ftp 6 May 03 05:00 lnc_test.txt
226 Transfer OK
LnC: Downloading: lnc_test.txt
250 CWD successful. "/" is current directory.
250 CWD successful. "/" is current directory.
227 Entering Passive Mode (192,168,0,20,215,16)
150 Connection accepted
226 Transfer OK
LnC: Waiting on file data on 0_lnc_scenes_onsite.ini
250 CWD successful. "/" is current directory.
227 Entering Passive Mode (192,168,0,20,215,19)
550 Directory not found.

Not sure why this happens. I mean, connecting to the ftp works, writing and reading files works, but I wonder what LnC: Waiting on file data on 0_lnc_scenes_onsite.ini tries to do?

Local Working Directory, LightWave 3D configs and Watch Folder are all OK after Diagnostics, but for some reason, FTP Server doesn't seem to to work, even though full access is provided.

Something I'm overlooking perhaps?

:)

m.d.
05-02-2011, 10:00 PM
I haven't tested it either.
But that implies that it'd work with the dummy saver in conjunction with exrTrader using the render path and precisely the same frame numbering settings as LW is set to in the render globals (which exrTrader can't, unfortunately, read out automatically).
That and not saving any other images from exrTrader.

@Joe: If you're interested: I've got docs on the exrTrader settings format for developers and I could also send you a free license.
Mail me: michael(dot)wolf(at)db-w(dot)com.

Cheers,
Mike

awesome....got the 2 best minds working on it...
cloud rendering is a dream come true...unfortunately now that i've found exrTrader....I can't do without

omichon
05-03-2011, 07:31 AM
Joe, I finally got some time to upgrade LnC to 1.5.3. All is rendering fine so far.
I just have noticed some random FTP error messages every now and then, on one of my nodes, but with no real consequences. Could it be a ftp setting issue on the FTP server side ? Any idea or recommended settings ?
Thanks.

JoeJustice
05-03-2011, 08:04 AM
@Lightwolf

I'll get in touch. It'll probably be an easy enough fix.

@Cageman

That's the configuration file on the server. It appears you have not initialized it your server for use with LnC.

Please review How-To Setup LightNet: Cloud for the first time:
http://www.joejustice.org/books/lightnet-cloud/how-setup-lightnet-cloud-first-time

@olivier

Errors are probably just the FTP server timing it due to inactivity. It's nothing to worry about. If the files aren't getting uploaded or downloaded properly, then it could be an real problem. If it's just losing connection after not sending or receiving for awhile, then it's nothing.