PDA

View Full Version : New LightNet: Cloud - onsite and offsite ScreamerNet render controler



JoeJustice
01-28-2011, 08:36 AM
Hey guys, I've been working on an entirely new version of LightNet for a little while now and I think it's ready for public beta testing. It's currently at version 0.9.9 and whatever bugs I get, I'll fix and pop out version 1.0!

This is called LightNet: Cloud and is really in no way related to previous LightNet (which I'm now calling Classic). I have totally re-written it from scratch.

LightNet: Cloud takes a totally different approach to rendering. It uses a FTP server to store scene data and output frames. Render nodes can be on the onsite network or offsite connecting via the internet. The settings are simple and I've created a diagnostics tool that will help tremendously in setup. Scenes are sent to the sever via a Watch Folder, so all you have to do is drop a scene in the folder and everything automagically happens from there.

Right now it's only available for Windows but I hope to have a Linux and Mac version at some point in the not-too-distant-future.

Here's all the info you need to get start:
http://www.joejustice.org/lightnet-cloud

Let me know what you think and if you run into any bugs!

Exception
01-28-2011, 10:45 AM
Oh wow, that's amazing! I've used LightNet for a long time and was just looking for a new render solution. I'll try it out at once.
Thanks, Joe!

Exception
01-28-2011, 11:41 AM
Hi Joe,

It looks great! I'm about 95% ready to run a test. It was easy and painless to set it up, much easier than before.

Edit: anwering some of my own questions.

A few small things:
- What's Onsite and Offsite rendering? How does this translate to the traditional configuration of a bunch of networked office PC's on a LAN?
---- Figured this one out on the website.

- The Content Manager doesn't exist anymore. It's called Package scene now and looks/works different.

- The Watcher's progress bar 'crashed' after moving the watcher's main window when it was showing some upload progress. This was apparently only a cosmetic one, since it uploaded fine and the entire process went fine afterwards.

- I'm wondering why the LightNet-Cloud doesn't seem to automatically spawn a screamer when there is a scene to be rendered.
--- Figured this out, Autostart was off.

- When I launched a screamer before putting a scene in a watchfolder, nothing happened. The scene would not render, even after waiting an eon. Only after I relaunched the screamer did it pickup on the scene.

- The progress bar for downloading of the content stuck on 10% and seemed frozen, just like the earlier report. But it still continued doing the download.

- Will it support frame splitting at some point?
- It would be nice to be able to set screamer process priority.

So far... :) Hope it helps, Cheers!

JoeJustice
01-28-2011, 12:07 PM
Okay...

Onsite vs. offsite. Basically the difference is that Onsite scenes use a network path as the content directory, offsite scenes use a ZIP file as content directory. If the scene is onsite, LightNet: Cloud (LnC) simply passes that path to ScreamerNet. If it is offsite, LnC downloads the ZIP file to the local working directory, unzips it and then passes that new, unzipped folder to ScreamerNet as the content directory. This way you can use PCs that aren't on the local network (i.e. off site) to help render scenes.

Yeah, I'll need to make a separate little tutorial on LW10's package system. It's very similar to the 9.6 Content Manager. Also, you can just zip up an entire content directory if it's not too big. Multiple scenes can use the same content ZIP.

LightNet-Cloud doesn't do anything until the screamer is launched, then it checks on the server in regular intervals. So if you have it set to check every 5 minutes, then it'll check and if there's nothing there, it'll wait 5 minutes and then check again. 5 minutes is a good time because it'll keep pressure on the sever down.

The routine I'm using for the FTP protocol isn't great, that's something I've got to work on. It works fine, but it's visual feedback isn't stellar.

Frame splitting would be cool to do, but complicated. Might work on that for the next version.

By default the Watcher checks the Watch Folder every 10 minutes, that might feel like eons :p

Thanks for the feedback! Let me know anything else you come up with.

rezman
01-28-2011, 09:43 PM
I am getting a config files fail error message. Is there something specific I should be looking for that might be entered wrong. They are all under the config directory and have the extension cfg.

JoeJustice
01-29-2011, 03:24 AM
The problem is probably the config suffix under the Lightwave tab. In Lightwave 8 the suffix is "8", in 9.6 64 bit it is "9-64". It's the number at the end of the config files.

Exception
01-30-2011, 04:40 AM
Yeah, I'll need to make a separate little tutorial on LW10's package system. It's very similar to the 9.6 Content Manager. Also, you can just zip up an entire content directory if it's not too big. Multiple scenes can use the same content ZIP.

Yeah, I spent a lot of time pressing for the new system, since the old content manager was broken since it was launched. The Package command is very stable. I just mentioned it as a point of attention. Not very important :)



LightNet-Cloud doesn't do anything until the screamer is launched, then it checks on the server in regular intervals. So if you have it set to check every 5 minutes, then it'll check and if there's nothing there, it'll wait 5 minutes and then check again. 5 minutes is a good time because it'll keep pressure on the sever down.

By default the Watcher checks the Watch Folder every 10 minutes, that might feel like eons :p

Hm, I set both on 30 seconds for testing... and it still didn't do anything, only after I closed and relaunched the screamer...

Will test more! Thanks so much for this already. It's yummy! :thumbsup:

Exception
01-30-2011, 01:54 PM
What do I do if my network path has spaces?
I tried putting it as content "path/to/content" in the item info, but that failed.

Also, the item comments box truncates a long path. So, network rendering becomes almost impossible here, with a corporate network structure.

JoeJustice
01-30-2011, 03:12 PM
Network paths will probably need double slashes as an escape. So instead of:

\\Main Server\Hard Drive One\Lightwave Content

You would use:

\\Main Server\\Hard Drive One\\Lightwave Content

I'll have to verify this at the office tomorrow, but I'm pretty sure that's it.

The comment issue is something I'm looking into...

rezman
01-31-2011, 09:49 AM
The problem is probably the config suffix under the Lightwave tab. In Lightwave 8 the suffix is "8", in 9.6 64 bit it is "9-64". It's the number at the end of the config files.

Ah, that fixed it. I entered in "9".

I have a odd rendering issue now though. Here's my setup briefly:
- I have an FTP server set up on the main computer, so any computer can connect via the net.
- Running Lightnet Cloud on two computers
- Lightwave is installed on both, licensed on just one. Yeah, overkill for the second computer, but I wanted to make sure that everything worked. Probably just lwsn.exe and plugin folder would be all I needed right?
- Watch folder on the main computer.
- Everything renders perfectly on both. Which is surprising because this setup would never work on a standard Screamernet setup. :)
- The frames render just fine on the second machine (png 32) but when they are transfered to the first machine by Lightnet the image gets corrupted. Is it possible that you are using ASCII instead of Binary for the transfer type?

JoeJustice
01-31-2011, 10:11 AM
I've rendered PNG 32s with no problems on my network and via internet. So I'm wondering what exactly is happening. What do you mean when you say they're getting corrupt? How are you checking them before they are transferred back to the FTP server?

rezman
01-31-2011, 11:08 AM
I've rendered PNG 32s with no problems on my network and via internet. So I'm wondering what exactly is happening. What do you mean when you say they're getting corrupt? How are you checking them before they are transferred back to the FTP server?

Hi Joe,

I check them from the LocalWorkingDir output folder on the second machine before it's transfered to the main machine. Here is the image before and after it's transfered.

http://goo.gl/IKzfN

http://goo.gl/lz3HJ

Exception
01-31-2011, 12:01 PM
- Everything renders perfectly on both. Which is surprising because this setup would never work on a standard Screamernet setup. :)

Why not? LWSN renders without a license. You get near infinite LWSN licenses for one LW license.

rezman
01-31-2011, 12:44 PM
Why not? LWSN renders without a license. You get near infinite LWSN licenses for one LW license.

No I didn't mean that :) I meant that Screamernet would normally run screaming (heh, pun) with the setup Lightnet has. It doesn't get anything from the first computer, it has it's own config file, the content folder is completely wrong, it doesn't have to have all the insanely pesky network setup that you normally need...and yet it renders happily away.

Actually, since the config is saved in a simple text file...I don't see any reason why you couldn't set it up, zip up the folder, send to anyone you know with a computer and an internet connection and painlessly add them to your cloud. They would simply have to run lightnet cloud. Of course they would have to really like you. But, as far as setup, there would be none.

Exception
01-31-2011, 12:59 PM
No I didn't mean that :) I meant that Screamernet would normally run screaming (heh, pun) with the setup Lightnet has.


Oh yeah, totally! I won't touch Screamernet with a 10 foot pole. That's why I've alsways been very happy with LightNet. And this is completely awesome.

JoeJustice
01-31-2011, 01:10 PM
Hmmmm.... rezman, I don't see any images attached. As far as I know my FTP protocol is set to binary, not ASCII. I have tried with several formats and they all seem to be working fine, including PNG32. Also, of course, the ZIP file downloads properly, so I don't see room there for there to be corruption of the files.

Could it be your FTP server? Have you tried to just upload a file from a remote computer to your FTP server via something like FireFTP? Make sure it is receiving properly?

Yeah, the idea being LightNet: Cloud, just to just be able to send friends and colleague your configs and program files and they could join in on rendering with you. I've tried to make the setup as straight forward and easy as possible.

rezman
01-31-2011, 02:22 PM
Hi Joe,

If I upload via ftp to the server it saves the file perfectly, but if Lightnet sends it, the image is corrupted. To double check if it was the server, I switched the software I was using for the server. Same issue.

Exception
01-31-2011, 04:49 PM
I think for the success of the program it would be important that the program that runs on a client sits silently in the background, has its priority set to low and doesn't pop up anything, ever.

Currently I see some status messages flickering onto my screen every now and then, but this could also be the watcher (equally annoying tho). The screamer is not set to low priority, and is launched in full view. Would be best launched minimized and even better: in the tray.

JoeJustice
01-31-2011, 05:32 PM
I see launching the screamer as something you do when you're not using your computer. I'm curious, do you see it as something you would run while still working?

The Watcher is something you would leave running, which is why it goes into the system tray. that is actually why I made them two separate apps rather than one, so that you would run Watcher while you're working and then when you're done with your PC you would run Cloud (have autostart enabled) and just walk away. A few minutes later the screamer would go.

The Watcher shouldn't show status updates unless it's set to manage the server.

rezman
01-31-2011, 11:11 PM
Quick update on the corrupted image issue. I tried an offsite server (Rackspace - Linux based) and it has the same exact issue. All firewalls are turned off. The machine has XP installed on it. The main machine is running Windows 7.

omichon
02-01-2011, 01:39 AM
I see launching the screamer as something you do when you're not using your computer. I'm curious, do you see it as something you would run while still working?

The Watcher is something you would leave running, which is why it goes into the system tray. that is actually why I made them two separate apps rather than one, so that you would run Watcher while you're working and then when you're done with your PC you would run Cloud (have autostart enabled) and just walk away. A few minutes later the screamer would go.

The Watcher shouldn't show status updates unless it's set to manage the server.

It's the way I use it too. Some inherited habits from my Smedge usage, probably ;)

JoeJustice
02-01-2011, 04:37 AM
rezman, I'm totally clueless. I don't know what could be causing that.

I can't replicate that problem. I've put your PNG file in my local dir and had LnC FTP it and it FTPs just fine to multiple FTP servers. I altered the code to force a ASCII connection and it just corrupts the file, it becomes totally unreadable. There has to be something wrong with that specific PC, but I can't guess what it might be.

Can you use other FTP clients on that computer? I don't see how Windows XP could possibly be the problem either...

geo_n
02-02-2011, 12:35 PM
How to set this up with internet that has a dynamic ip address?
Is is possible to be like dropbox that doesn't require an ftp server?

Exception
02-02-2011, 01:33 PM
I see launching the screamer as something you do when you're not using your computer. I'm curious, do you see it as something you would run while still working?

Sure, when I'm reading a long document. I wouldn't want myself or 'my friend' to be bothered with manually launching any screamers either... just wait till idle for a while, then just launch it silently in low priority mode.


The Watcher shouldn't show status updates unless it's set to manage the server.

Right, but it is managing the server, and it's popping up silly things that interrupt my work. I would rather not have that :)

Hieron
02-02-2011, 05:53 PM
Oh yeah, totally! I won't touch Screamernet with a 10 foot pole. That's why I've alsways been very happy with LightNet. And this is completely awesome.

Hey Tom, you didn't use BNR or similar before?

/agree on the 10 foot pole for screamernet.


ps: you pushed them into redoing package scene? Thanks for that... such an important tool.

JoeJustice
02-02-2011, 08:40 PM
Alright... working on a few bug fixes and adding a few features now...

Will add a LScript for settings content information and settings segment number.

Will add segmentation, can split into 4, 16 or 25 segments. It will create the segments but you'll have to use Photoshop or something to put them back together. I might make some kind of joining tool in a future version, but that'll require some thought, so it won't be in v1.0

Will add low priority option for ScreamerNet

Will add "silent mode" to disable all pop-up interfaces

May preserve existing output file name, I'm on the fence on that one for a couple of technical reasons and I don't know how many people that will actually effect

JoeJustice
02-02-2011, 08:41 PM
geo_n,

You can use http://www.dyndns.com/ for dealing with dynamic IP address. It's what I use and it's worked great!

geo_n
02-03-2011, 05:34 AM
geo_n,

You can use http://www.dyndns.com/ for dealing with dynamic IP address. It's what I use and it's worked great!

Thanks I'll try it out. I was going to try www.no-ip.com
Will lightnet work with third party renderers like kray?

JoeJustice
02-03-2011, 08:43 AM
I'm not familiar with Kray, but LightNet: Cloud is designed to work with ScreamerNet, so if it works with ScreamerNet, then it'll work with LnC.

geo_n
02-03-2011, 10:51 AM
Not sure if kray works with screamernet. I haven't used screamernet yet only other render controllers.
Is it possible to use lightnet cloud for local network? I removed ftp offsite rendering and check onsite only.
Nothing happens when I put scene in watchfolder.

omichon
02-03-2011, 03:54 PM
Is it possible to use lightnet cloud for local network? I removed ftp offsite rendering and check onsite only.
Nothing happens when I put scene in watchfolder.

You have to install a FTP server on your local network to run LnC locally.
I use Filezilla Server 0.9.37 here to reduce the huge stress on my Internet connexion, and all is running fine so far. You just may have to open your firewall to the service. The IP address for the connexion should be the local IP of the machine running the FTP server.
There are also other some other solutions, depending on your OS (like IIS Webserver in Windows 7).

geo_n
02-03-2011, 07:34 PM
You have to install a FTP server on your local network to run LnC locally.
I use Filezilla Server 0.9.37 here to reduce the huge stress on my Internet connexion, and all is running fine so far. You just may have to open your firewall to the service. The IP address for the connexion should be the local IP of the machine running the FTP server.
There are also other some other solutions, depending on your OS (like IIS Webserver in Windows 7).

I installed fire server last night but found out that the ports are blocked by the isp. So I was hoping I could run this locally without an ftp server.
Anyway I set up filezilla server as 127.0.0.1 and set up username accounts and password and put that in lightnet but in diagnostics it says ftp error.
I followed this tutorial to setup an ftp server
http://www.raymond.cc/blog/archives/2007/10/19/how-to-setup-ftp-server-on-windows/comment-page-1/#comments

Exception
02-03-2011, 07:49 PM
If you have a NAS, most of them have an FTP server built in. We use that at the office for the LightNet setup. Fast, efficient, and keeps running even if your control computer is shut down.

geo_n
02-03-2011, 08:18 PM
If you have a NAS, most of them have an FTP server built in. We use that at the office for the LightNet setup. Fast, efficient, and keeps running even if your control computer is shut down.

Will look into it but its a bit expensive for personal use. If I could just use this lightnet version locally would be great.
Have you tested kray with lightnet by any chance?

omichon
02-03-2011, 11:46 PM
I installed fire server last night but found out that the ports are blocked by the isp. So I was hoping I could run this locally without an ftp server.
Anyway I set up filezilla server as 127.0.0.1 and set up username accounts and password and put that in lightnet but in diagnostics it says ftp error.
I followed this tutorial to setup an ftp server
http://www.raymond.cc/blog/archives/2007/10/19/how-to-setup-ftp-server-on-windows/comment-page-1/#comments

If you haven't missed any part of the tutorial, you should be fine.
Just a thought, but this error could be about some missing files in your Watch Folder. Try to run the Watcher first, then LightNet-Cloud.exe.

JoeJustice
02-04-2011, 04:38 AM
I think I understand geo_n's original question better now...

geo_n, you want to use a local FTP server, but your local IP address is dynamic, right? I thought what you were wanting to do was use a FTP server on the internet, which would require a dynamic dns service.

Okay if you have a DHCP system, like a router, that controls your local IP, you have to modify it to only use a certain range, like .100 - .255, and then manually make the IP on your local FTP server static, like .2 or something (.1 is usually the router's number).

Let me know if you need more details in setting that up, we're getting into network administration now.

geo_n
02-04-2011, 07:03 AM
Not sure I missed anything in the tutorial except for the setting up the ftp which I purposely now skipped because of my isp blocking ports. The router ports have portforwarding but testing the ports it says connection refused. So it must be the isp.
What I intend to use lightnet for is similar to lightnet classic, bnr,etc. Seeing the setup is easier and its a new rewrite, I would like to use this instead of lightnet classic. Just one pc controller and nodes.
I will figure out the isp issue when I call them up or if I can afford to buy a nas, that will probably be the easy option.
I thought I could test this out first locally in one pc with the master and node but the screenshot here shows nothing happening when I put files in the watch folder.
http://www.newtek.com/forums/showpost.php?p=1108001&postcount=30

omichon
02-04-2011, 07:17 AM
geo_n, I had a similar error message with 0.9.5. It was due to the missing files _off_site.txt and _on_site.txt. Since I can't see them in your Watch Folder, on your screenshot, it may be the culprit. But your problem could be elsewhere. I am afraid I can't help you more on this, sorry, you will have to wait for Joe ;)

JoeJustice
02-04-2011, 07:43 AM
Do you have the Watcher setup properly and running?

geo_n
02-04-2011, 09:51 AM
I had the _off_site.txt and _on_site.txt file but I restarted the instructions and deleted all files. This is the new setup with no ftp config but still nothing happens.
Is there anything else to setup with the watcher.exe besides the watch folder and manage server?
Its running but with that error.

JoeJustice
02-04-2011, 12:34 PM
You have to have an FTP server. It's not an option.

I've gotten several e-mails from others thinking the the FTP server is an option and leaving it blank will somehow make the system work. I don't know if I have to spell that this out better in the final documentation or what... (BTW, I'm open to any help in developing documentation. And ideas to contribute or typo corrections, I'm dyslexic so it takes for FOREVER to write anything and even when I do I always have tons of typos.)

Back to the point. Philosophically the reason the system uses an FTP server is because it is designed with the idea of being self controlled with only data being centralized, not control being centralized. It has pros and cons, but for my money the pros outweigh the cons. For example, a con is that you never really know exactly how many computers are working or what they are working on at a given time because they're each independent. It's not a client/server relationship. That in and of itself is a pro as well, the independent systems just download the information they need and then get to work.

So let's do an analogy (everyone loves those). Let's say a scene is a big bag of kibble and a bunch of little puppies (again, everyone loves those) are the render drones.

The traditional concept of a render farm control would be like all of the puppies coming into the kitchen, lining up and you handing them each a piece of kibble. They take the kibble, and go to the corner to eat it. When they're done, they get back in line and wait for another piece. This goes on until the kibble is all gone. Works great, but it requires you to train all the puppies to get in single file, wait, eat in the corner and then get back in line. And, oh yeah, they all have to be in the house too.

LightNet:Cloud is totally different. With it, your puppies walk into the kitchen every five minutes or so and check for new kibble. All you have to do, at any given point, is to walk into the kitchen and dump a big bag of kibble on the ground. When a puppy wonders in, it sees the kibble, grabs it and takes it to another room to eat it. When it finishes it comes back to get another piece. Pretty soon you have all of the puppies all coming in an grabbing pieces of kibble. Well it might only take them a minute to eat it, so they're grabbing five pieces at a time and happily eating it. It's not being parceled out or managed, they're just coming in and getting it.

Now you got a whole bunch of kibble you need eaten? Well you call your friends and tell them you need their dog to help you out. Some have great danes some have chiwawa, not big deal because every little bit helps. They say "okay", let them outside and point them to your back yard. You dump another big bag of kibble outside and all of your friends dogs come and grab a few pieces and head back home to eat them. Now when your kitchen is all cleaned out, all of your puppies wonder outside too and start grabbing kibble. Dump another bag in the kitchen and your puppies come back in and get to work on that while your neighbors puppies are still working on the pile outside.

Of course the only problem with this analogy is if the scene files are kibble, what does that make the final rendered frames??

archijam
02-04-2011, 02:51 PM
Of course the only problem with this analogy is if the scene files are kibble, what does that make the final rendered frames??

I'll answer your question with a question: Can I help tidy up your CRAP documentation?

..


And where do the puppies do their stuff? In the kitchen? (This is a technical question ...)

JoeJustice
02-04-2011, 03:15 PM
You may and in the kitchen. Yes this analogy really falls apart when it comes to output. :(

archijam
02-05-2011, 10:41 AM
Will do :) .. I'll email you this week ..

JoeJustice
02-05-2011, 10:54 AM
Incidentally, don't work too much on doing any doc work at the moment. I've made some major changes to work-flow in the soon-to-be version 1.0. Hope to have it ready by the end of next week.

When it comes to the documentation I think I'd like to do a series of how-tos with links to more details. Like this:

How-To Setup LightNet: Cloud for the first time (Or getting started)
How-To prepare a scene for rendering
How-To Setup a Watch Folder on a network
How-To Setup a Watch Folder on an offsite computer
How-to add an onsite screamer to your render cloud
How-To add an offsite screamer to your render cloud
How-To setup a FTP server

And the maybe a list of breakdowns and details:

FAQs
All about the Settings
All about LightNet: Cloud's operation
All about LightNet: Watcher's operation
Tips and Tricks
Best Practices
Bugs

With documentation in mind, does anyone know the status of LightWiki? I'd love to use that for documentation rather than my own site because I don't have any kind of wiki system installed and don't really want to make a new site for such a thing.

Lightwolf
02-05-2011, 11:04 AM
With documentation in mind, does anyone know the status of LightWiki?
It's still there: http://www.lightwiki.com/wiki/mediawiki-1.4.4/index.php?title=Main_Page it does look like Ben needs to set up his mod_rewrite again though.

Or so I think...

Cheers,
Mike

rezman
02-05-2011, 12:30 PM
I finally got the second computer to work, but it is on a completely different system running Windows 7 as well. So I guess we'll never know if it was somehow an XP issue.

I had a quick question on the configs. The only way I got the second system to work was to add the Lightwave program, config, and plugins folders (and content of these obviously) on the second computer. Which makes sense if you are going to have someone off your network render as well. So, my question is...how are the config files used on the render on the client machines? Or are they at all? It seems that the content directory and the version of the config files are irrelevant with LNC.

Also, another question about processor cores. As in, if you have a 4 core machine, are all cores being used with just one node rendering per machine?

JoeJustice
02-05-2011, 01:43 PM
Lightwave is multi-threaded, so launching one screamer will take advantage of multi-core processor.

You do need to have all of the Lightwave programs and plugin files on your computer or on a centralized location. I didn't really go over any of this with the tutorial so far, but it's something I'll need to explain.

The Lightwave config files tells Lightwave (and ScreamerNet) where the plugins are located. So if you have Lightwave installed on the C:\ drive under C:\Lightwave, the easiest thing to do is copy the entire folder over to each PC and then copy the configs to a sub directory like C:\Lightwave\Configs this mean anytime you add plugins you'll need to update these PCs with the new plugin and new config.

Another option is to map a drive to all of the computer of a centralized location for the Lightwave programs. Say you map a network drive Z:\ and copy the Lightwave folder there so that it will be Z:\Lightwave Do the same thing with the configs as before EXCEPT, you need to edit them from your PC and change all of the references to C:\Lightwave to Z:\Lightwave

The config file to edit the EXT file. Also note, use double slashes. So it would be Z:\\Lightwave rather than Z:\Lightwave

rezman
02-05-2011, 02:38 PM
Thanks Joe! Very enlightening. Things are a bit different than Screamernet and other render controllers (as you know) so it really helps to understand these little details.


All the best!
Stephen

Chrusion
02-06-2011, 08:09 AM
Requesting editorial tweaks to the node/screamer log: (red added, green edited, black deleted)

2/6/2011 9:37:15 Frames X - Z gathered for scene_name.lws
ScreamerNet rendering more frames from same scene - 2/4/2011 22:33:42
2/6/2011 9:39:00 Node/ScreamerName rendering frame(s) 1191 - 1191
2/6/2011 9:40:51 Rendered frames X - Z in 00:03:34
2/6/2011 9:40:53 Uploading files...
2/6/2011 9:41:09 Uploaded frames X - Z to FTP: /folder/scene_output/rgb/

Chrusion
02-06-2011, 11:25 AM
If I'm the owner of an anim (my PC is the manager of my FTP) and I invite you to help me render it by having you enter my FTP account info in your LNC, but you forget to uncheck your LNC from being a manager, what happens? Will total confusion result? Will ini files get corrupted? How does one fix whatever happens after it has? Is there another way to prevent any ill effects other than reminding all your "peeps" (via phone, chat, email, et. al) to set their LNCs to non-manage mode?
.

JoeJustice
02-06-2011, 11:59 AM
There shouldn't be any problems if they have Manage Server checked because they all process the queue files in the exact same ways.

The only problem I could perceive is if they forget they are on your cloud and add their own scene into their Watch Folder, thinking it is still their system. If they do intend to only help you render, and don't need to upload their own scene or manage the queue, then they should probably keep Manage Server unchecked.

Chrusion
02-07-2011, 10:00 AM
Yay! Version 1.0 posted at joejustice.org (http://www.joejustice.org/lightnet-cloud)
.

Chrusion
02-07-2011, 11:37 AM
More Observations...

Like the new Silent options in v 1.0.

Testing the off-site "mode" now using two local boxes on a my home LAN and one box at my university office.

Discovered after 20 minutes of waiting for the home box set as manager (win 7 x64 running LW10) to download the zipped content directory and get going, that it wasn't going to. Then it dawned on me, how does LNC know what the name of the zip file is that was uploaded by the Watcher? It doesn't! Why? Because the name of the zip is "inside" the LWS which was zipped up with the rest of the content directory! LNC can't see inside. Soooo, what if I put the scene file in the watched folder along side the zip? OK, there it goes up to my FTP. Wait... wait... still waiting... after several minutes, LNC FINALLY "woke up" and down loaded the zip, then the scene, then unzipped the contents into the work folder, and then, after pausing a few seconds, issued commands to its screamer to load up the scene and render the specified first frame! Yay! It works.

On to the next machine in the house... after launching LNC and Watcher, I again had to wait several minutes before LNC decides to download the zip and scene and do the same thing. Yay! Box two on winXP x32 running LW 9.6 is rendering!

On to the office box. I remote in via Windows Live Mesh and get its LNC and Watcher running and wait those several minutes before it downloads the files and begins rendering using LW 9.6 on win7 x64 there!

So, with this experience understood, the docs need to say explicitly that the LWS MUST NOT be zipped up with the rest of the content directory. In fact, the 'scenes' folder is NOT required either. Just zip up the 'images' and 'objects' folders and any root files like caches, etc. Then put the zip AND the LWS into the watched folder. The docs say put a LWS in there and/or content zip. I took this to mean LWS only for on-site rendering and just the Zip only for off-site rendering.

Also, I think the wait time after starting LNC and when it decides to check FTP is determined by that setting in the FTP config tab... default is 300 seconds (5 min). Sounds good to have that, but we're impatient folk, we animator artists ya know. We want things to start NOW! So, Joe, if you could put a button in LNC to "check server" the same as there is for Watcher, this would be very helpful as the Watcher is only for uploading and its check button doesn't inform LNC that it just put files up. So as soon as LNC is running and we manually get its screamer up, we don't want to wait the X amount of seconds for LNC to check FTP for scene and content. A button to check Now solves this.

Question... Is it Watcher or LNC that uploads the rendered images? You've said LNC only downloads and Watcher only uploads, but if watcher isn't running on a box, then images won't be uploaded, so since it appears they are, then LNC does upload, but only images, not scenes or content. Wondering again why all this can't be consolidated in the one LNC exe.
.

JoeJustice
02-07-2011, 12:54 PM
Question... Is it Watcher or LNC that uploads the rendered images? You've said LNC only downloads and Watcher only uploads, but if watcher isn't running on a box, then images won't be uploaded, so since it appears they are, then LNC does upload, but only images, not scenes or content. Wondering again why all this can't be consolidated in the one LNC exe.
.

Cloud uploads images.

Both Cloud and Watcher can upload and download. I have no recollection of saying they do one or the other.

Cloud downloads and renders scenes. Watcher uploads scenes and content in the Watch Folder and manages the queue. These are two difference jobs that not every computers in the clouds needs to do. So it is split into to separate applications.

Chrusion
02-07-2011, 01:58 PM
Cloud downloads and renders scenes. Watcher uploads scenes and content in the Watch Folder and manages the queue. These are two difference jobs that not every computer in the clouds needs to do. So it is split into to separate applications.
What if in the watcher config tab there's an "Enable Watcher on this computer" option check box? This would serve the same purpose as running or not running a separate exe. Just thinking out loud.

Did that "check server now" button for LNC tickle your brain at all? :D

400 frames rendered via off-site, 1,750 more frames to go... smooth, no hiccups so far...
.

Chrusion
02-07-2011, 02:10 PM
Another scenario...

What if two or more computers check the FTP at the exact same time (within one second). Or what if two or more computers check the FTP one second apart, but they all experience a 3 second delay (for whatever reason) in uploading their edited copy of the queue (lnc_offsite_scenes.ini) file? Won't all three get the same next frame to render? Rare, yes, but not impossible, correct?
.

JoeJustice
02-07-2011, 04:40 PM
I might end up giving the Watcher a configuration forum. The Watcher's interface in general needs a lot of improvement.

I could put a "Check Now" button on Cloud, but it's not something I would use. It's designed to sit, wait and check, not really be hovered over. But I guess people are going to hover.

My focus right now is to get the FTP system working a bit more cleanly, then (and the FTP system is part of this) making the current version fully cross platform. After that I'll focus on the Watcher interface. Other than that, I only plan on bug fixes and cleaning up the code.

There is the possibility for cross-communication if two computers access the server at the exact same time or seconds apart, but I've minimized it as much as possible. LnC checks the server in cycles and the very last thing it does is update the server's ini files, and only when it's actually updating them. They're small, text files, so it's unlikely there would be any cross-communication to a great degree.

Chrusion
02-07-2011, 05:41 PM
I could put a "Check Now" button on Cloud, but it's not something I would use. It's designed to sit, wait and check, not really be hovered over. But I guess people are going to hover.
hehe... well, umm... when setting it up and testing to be sure each system is working, yeah, the check now button certainly aids there. But, you're right, once you're confident it's working, no need to hover, you're able to relax, sit back and basically ignore it.


The Watcher's interface in general needs a lot of improvement.
Agreed. Don't know what you're thinking, but I can see an interactive method of managing the queue as a visual listing of scenes with frame status (current start/end), buttons for add and remove that pop up a dialog to select desired scene file and fields for entering start/end frames, and selecting a scene in the list to drag/drop to set priority.


There is the possibility for... two computers access the server at the exact same time...
Yeah, didn't think it would be a big deal should it happen, but the more systems accessing the FTP, the more chance of "collisions."
.

omichon
02-08-2011, 01:04 AM
I second Chrusion about the various Watcher interface improvements.
Here is a couple more suggestions : Would it be possible to have alternate FTP server settings ? I mean, when rendering onsite only, there is no reason to use a distant FTP server (right now, I avoid it since it use all my Internet bandwidth). It would be nice to choose between different profiles depending on the cloud strategy.
Another little thing : currently, if you drop any scene in the Watch folder, it is uploaded to the server, even if the scene is not ready for LnC (no master plugin). So could we imagine a warn in the Watcher like "missing config settings" or something like this before uploading it ?

Edit : Another point, about cleaning the local Workdir. It would be convenient to a have a way, from the Cloud interface, to cleanup the folder, or to open it to manually clean it up (if the first option sound a little 'dangerous').

Chrusion
02-08-2011, 07:26 AM
One large niggle...

When LNC is running, screen(s) will not go to sleep. I must manually turn them off. Annoying.
.

JoeJustice
02-08-2011, 09:52 AM
I plan on making a cleaning procedure for the local working directory at some point.

I like the idea of checking to verify information is present in the scene before uploaded.

You can make on of the computers on your local network a FTP server very easily. You don't have to use a internet based FTP server for onsite rendering. http://filezilla-project.org/ is a good option. I plan to make some documentation on the site for how to setup various FTP solutions.

I just noticed the screen won't blank when LnC is running as well. I'm not sure what's causing that... interesting...

omichon
02-08-2011, 10:03 AM
You can make on of the computers on your local network a FTP server very easily. You don't have to use a internet based FTP server for onsite rendering. http://filezilla-project.org/ is a good option. I plan to make some documentation on the site for how to setup various FTP solutions.


Agree. I use Fillezilla server for my onsite rendering. But when I want to add some computers out of my local network, the local Fillezilla server is not a solution anymore, right ? In this case, I have to switch to another FTP server. Then, having different server profiles in LnC wouldn't be handy ?

JoeJustice
02-08-2011, 10:10 AM
Well, I did create load and save settings just for this situation.

omichon
02-08-2011, 10:19 AM
That's a solution, right.
Regarding configuration files, when saving, if you select an existing one to overwrite it, it will just add an extra .conf extension.

Chrusion
02-09-2011, 05:33 AM
I use Fillezilla server for my onsite rendering. But when I want to add some computers out of my local network, the local Fillezilla server is not a solution anymore, right ?
Not necessarily. I have a Filezilla server running on my local box that's seen by the outside world (haven't used it for LNC as I use my web host account for that). Just use your internet IP address as the server name instead of your local, internal network IP and set your router to forward port 21 to the internal IP.
.

omichon
02-09-2011, 05:41 AM
Thanks for the info, Chrusion. I was expecting it could work that way, but never tested it. Maybe using the FTP server from a NAS could be an option, too.

omichon
02-10-2011, 12:26 AM
I have noticed that we could run more than one instance of the Watcher (on the same machine). I can see the point with Cloud (optimizing CPU cores use), but what is the point with the Watcher ? Wouldn't it be better to wake up an already running Watcher when trying to run a new one ?

JoeJustice
02-10-2011, 07:26 AM
If you had multiple watch folder and multiple FTP servers you could manage them with one computer running Watcher from multiple locations. With issues like this, the second I change it to only be able to run one instance, I'll get an e-mail from someone saying, "I run all kinds of multiple instances, why did you change it?"

I don't mean to toot my own horn here, but had a great outcome on a render last night. My file server shut down at some point last night, taking my onsite filer server down. When I turned it on this morning my first thought was that my renders were all screwed up since the content file was no longer there. I went and checked my drones and they were all rendering happily, LnC had a big list of error messages saying the content directory was not accessible.

They had rechecked after I had restarted the file server, found the content and went chugging right along.

This was great because this is exactly how I designed the system to work. It checks in on it's intervals and determines if everything is in place, the FTP server, content directory and so on. If everything isn't in place it gives an error and resets everything for the next check. On the next check it goes through the process again. As soon as everything is ready, it gets going. no muss, not fuss. No blank frames or missing objects or images.

It's nice when things work :)

omichon
02-10-2011, 07:40 AM
If you had multiple watch folder and multiple FTP servers you could manage them with one computer running Watcher from multiple locations. With issues like this, the second I change it to only be able to run one instance, I'll get an e-mail from someone saying, "I run all kinds of multiple instances, why did you change it?"
Don't get me wrong, I wasn't complaining at all but just wondering about multiple Watchers. I haven't had figured out this kind of situation.


Nice to read LnC is quite robust ! Great News and good work :thumbsup:

omichon
02-10-2011, 12:39 PM
Here is why I would like to be able to define the base name for the rendered frames : All my scenes use multi-cameras (Camera Selector plugin). Since this plugin crashes Screamernet, I have to render a scene for each camera. The problem is that I can't play back the full animation without renaming each sequence or use some compositing application.

JoeJustice
02-11-2011, 04:55 PM
I'll work on making the output an option on the next version.

omichon
02-11-2011, 10:46 PM
I'll work on making the output an option on the next version.
Great ! Thanks Joe :thumbsup:

omichon
02-12-2011, 11:25 PM
I get this error message from the watcher every now and then :
"Unable to create file "LnC-*scene*-onsite.ini"
Press OK to ignore and risk data corruption
Press Cancel to kill the program."

How to avoid this situation ? Is it a config issue ? and what is the best choice when being asked ?

JoeJustice
02-14-2011, 12:48 PM
I've gotten that a time or two as well and I have not yet been able to track down the cause.

omichon
02-24-2011, 05:13 AM
Running LnC intensively here for a couple of weeks now (with a local FTP server), and all is fine except the error message issue with the Watcher.
Hope to see some improvement regarding the tedious cleanup task (that has to be done manually for now) on the local Workdirs and FTP.
That said, I really appreciate the flexibility and reliability of LnC :)

JoeJustice
02-28-2011, 10:21 AM
Awesome! Still working on a better FTP system at the moment. Will have some update out eventually.

omichon
03-01-2011, 12:51 AM
Awesome! Still working on a better FTP system at the moment. Will have some update out eventually.

Cool ! Can't wait to test it for some offsite render situation (since the current one really eats up all my Internet bandwidth).

Riza
03-05-2011, 06:36 AM
thanks so much! have been lightnet user till now, and it's nice to see a new version..will take a spin!