Shouldn't Network Rendering Be Easier?

[drclare]

Although i have never tried network rendering with Lightwave, from what i have read and heard, it is a real pain in the *ss. I did try to help someone set up network rendering with Electric Image Universe, which ive heard is the best for network rendering, but we couldn't get it to work. It seems like it is always a hassle. What i am wondering is how hard would it be to integrate an automatic network renderer in Lightwave. What i would like to see is when you hit "render frame" or "render scene" Lightwave automatically finds any and all available cpus connected to your network and uses them. It should analyze how much of a processor's power is being used and decide if it should be used for rendering or not. That way the user wouldn't have to do anything but sick back and wait.

[Beamtracer]

Over the last number of years there have been many posts on this subject. A lot of people have put forward ideas on how ScreamerNet could be improved or replaced.

My recent suggestion was to create a network renderer that used Rendezvous technology:
http://vbulletin.newtek.com/showthread.php?s=&threadid=7932

Other people have offered various other suggestions which would improve things. So many good technologies are out there, but remain untapped.

Screamernet discourages many people from using a network for rendering.

[prospector]

What i would like to see is when you hit "render frame" or "render scene" Lightwave automatically finds any and all available cpus connected to your network and uses them. It should analyze how much of a processor's power is being used and decide if it should be used for rendering or not. That way the user wouldn't have to do anything but sick back and wait.

Already out there

Just called the company Friday thinking that this was what I was looking for but it wasn't.

What we call network rendering they call clustering.
I thought clustering was something else and they said what I wanted was Parallel computing.

They can use any computers in your network (for network rendering) and it will automatically send out frames to be rendered by checking all CPUs usage, and sending next frame to it,(cpu not being utilized)
And not computer A does frames 1 to 100, computer B does 101 to 200 and so on.

So your faster computers will render more than your slower but all will be utilized 100% automatically.

[Beamtracer]

Originally posted by prospector
So your faster computers will render more than your slower but all will be utilized 100% automatically.
ScreamerNet already does that, but it aint easy.

The host computer should be able to find all the services (ie the available render nodes) itself, then pass any relevant information to them automatically (ie where the content files are). The user should not have to think about this.

Imagine you work in a big office and want to use the network after hours to do rendering. Your host machine uses Rendezvous to broadcast a message to the network asking "are there any Lightwave render nodes here?"

Using Rendezvous the computers with a Lightwave render node respond and handshake with the host. Other computers without rendering software don't respond.

After this initial Rendezvous handshake has taken place, other technologies come into play for further communication (ie regarding content files).

Rendezvous gets the ball rolling.

[drclare]

Yeah, that sounds good. The main thing is that the user shouldn't have to think about any of that (like you said) and should just be able to hit "render" and be good to go. This is probably just wishful thinking though to hope that newtek might institute a system like this.

[paintboy]

Drclare,
just curious why you would post a Question/complaint on something you havent tried(perhaps you have and have not mentioned it?)
despite what you may have heard, it is not that difficult to implement
i have 5render nodes operating underX.2.6. Yes it probably could be easier(what does not fall into that catagory)
once set up (which requires a touch of patience, and carefull typing of command lines,something which seems to make most mac users shudder? changing of a few network settings, to insure proper comm.)bada boom bada bing, yer done...
less that 30 min.
then it requires very little maint. and works about as reliably as anything else associated with LW?
this statement is comming from a super"rightsided" artist with little or no experience in such matters(read: not a propeller head/geek).theres a double buttload of setup tutes available.
if Paintboy can get it to work...it cannot be that hard.
maybe you might try it yourself? or you can let the "crowd" convice you its impossible?
"argue for your limitations, and they are yours."

[Beamtracer]

Hi Paintboy. If you set it up once and never touch it again that's fine.

But.. if you are in a situation where you have to change the set-up every time, then you waste 30 minutes every time getting it going.

For example, the office scenario in my previous post, or any situation where your available machines are always changing.

Why should the Screamernet users be fretting about configurations and decisions that the machine could be handling?

The command line is archaic!

[paintboy]

Beam,
i have a Cheep third party controller which handles matters of this sort for me.
SNC
anytime a new node is launched the conroller picks it up and goes to work.
my comments do not amount to a blanket endosement of the "stock
network controller" it stinks.... but to say that it does not work is not true.
for a little guy like me, it can work... and is quite powerful if you have some extra hardware around(who doesn't these days).
like some other stuff with LW on the mac i have had to learn to rely on other stuff to "prop up" or supplement LW, so maybe i am just used to it?
Even on the intel side you will find that most everyone who is serious about network rendering has , third party rendering solutions, NT cannot provide
all solutions to all things. you can't be good at everything, and really be good?
the nominal cost of this controller were quickly offset on the first usage.
not to mention that with a little creative thinking in respect to how LW uses directorys a lot of the problems you mention can be avoided by a little planning
ahead? oops!.. but that would require the user wouldnt it, dont want that;)

[mlinde]

Originally posted by Beamtracer
The host computer should be able to find all the services (ie the available render nodes) itself, then pass any relevant information to them automatically (ie where the content files are). The user should not have to think about this.

This is a good idea, a bit of an upgrade to the technology behind the LWSN application.


Imagine you work in a big office and want to use the network after hours to do rendering. Your host machine uses Rendezvous to broadcast a message to the network asking "are there any Lightwave render nodes here?"
Using Rendezvous the computers with a Lightwave render node respond and handshake with the host.

This doesn't require Rendezvous. This requires a "backgrounded" active version of LWSN that is always running, with the upgrade you mentioned above.

I was tracking your redezvous thread earlier Beam, in so far as to spend some time at O'Reilly.com reviewing rendezvous based applications. The thing is, what rendezvous does best is already done in an office network -- the accessiblity of computers on the network. What you are really looking for is an upgrade to the technology behind LWSN, and a better render controller (along the lines of Jon's SN Controller). I know this came up before, but all rendezvous would do to improve network rendering is to automatically detect new computers on the network, and allow communication between them. If those computers already exist (via IP and a DHCP server, for example) then rendezvous is unnecessary.

[Beamtracer]

Originally posted by mlinde
all rendezvous would do to improve network rendering is to automatically detect new computers on the network, and allow communication between them.
Hi Michael. Rendezvous would not only detect other computers on the network, but it would also detect what "services" they are offering (ie which ones have render nodes and which don't).

Sure, it's only an initial handshake between the nodes. It does nothing else. But it's easier to do this with Rendezvous than for Newtek to devise its own method of node I.D. You also wouldn't have to worry about where you placed your nodes or what directories they are in.

Rendezvous is also dynamic (it updates to changes) and would itself be invisible to the user (a list of nodes would simply appear in a Lightwave window). I think it would be better than having render nodes always running in the background, which would themselves have to continually ping the network to check for changes.

It's one of many improvements that are needed for network rendering in Lightwave. It's just a small start, and the trigger for other communication and technologies that would follow.

Personally I don't think it's good enough that other 3rd party Screamernet controllers are necessary to achieve this. Network render control is such a basic feature of 3D animation that it should be offered within Lightwave and it should be easy to use.

[Lightwolf]

Hi there,
yup, me again, the PC troll ;)
Seriously though, Beamtracer, the issues you bring up about detecting a renderer across a network aren't really the hard part.
The tough bits come into play once you have to install LW with all plugins on every machine, make sure they have the right plugins actually installed, and using a content directory properly so that any cpu on the net can find the files (and not just images, scenes and objects, but thirs party files, like partivles etc, as well). This does require so though on the set-up, but I've actually installed a LightNet / PC based netwrok rendering a couple of times at other offices, and it took me around 15-20 Minutes incl. testing for all the machines.
The backround task that checks for render requests would be nice, and Stealthnet did go into that direction, but I guess NT scrapped it (well, it was buggy as hell).
None of the functionality you request here would make Rendevouz neccessary btw, actually, the question is whether it would be in the way concerning cross-platform compatibility (which includes mixed PC / MAc render farms, offices). I also think that the hurdles I mention above would probably by much harder to get over, than the "let's detect all machines in a network" bit, that is relatively easy to do.
Other apps have that functionality already, combustion /3dsmax, and DF by eyeon come to mind here.
I would like to see the network renderer do distributed single frame rendering as well, also for previews (Imagine a GI preview rendered across a farm... whooosh.).
Cheers,
Mike

[Beamtracer]

I'm not a programmer but I can't understand why distributing a single frame across a network is so hard to do.

The frame could be split into 4 parts , distributed to the nodes, then the pieces need to be sent back to the host, cached, and reassembled.

Is reassembly of 4 parts of an image a really hard task for a machine to achieve?

[mlinde]

Originally posted by Beamtracer
Hi Michael. Rendezvous would not only detect other computers on the network, but it would also detect what "services" they are offering (ie which ones have render nodes and which don't).

Sure, it's only an initial handshake between the nodes. It does nothing else. But it's easier to do this with Rendezvous than for Newtek to devise its own method of node I.D. You also wouldn't have to worry about where you placed your nodes or what directories they are in.

But this already happens. When a node is running the render application, it periodically looks for information about rendering. The technology is a bit backwards, since it uses a text file, but it already exists in the current system.

What Newtek needs to arrange is an install option for the LWSN node, which would install just the software required to render.

Rendezvous is also dynamic (it updates to changes) and would itself be invisible to the user (a list of nodes would simply appear in a Lightwave window). I think it would be better than having render nodes always running in the background, which would themselves have to continually ping the network to check for changes.

Are you suggesting that the controller have the ability to add and remove nodes while rendering? This exists in many 3rd party rendering controllers. As far as service detection and background applications, the rendering application must be running on the node to be detected. Rendezvous does not have the capability to launch applications across the network. There are improvements that are necessary (or at least should be seriously considered) for the network rendering system in Lightwave, but nothing you are suggesting requires the use of rendezvous. Just because you can do a thing, doesn't mean you should.

Personally I don't think it's good enough that other 3rd party Screamernet controllers are necessary to achieve this. Network render control is such a basic feature of 3D animation that it should be offered within Lightwave and it should be easy to use.

In fact, the built in render controller in Lightwave does work. The issue that most users seem to have is the complexity and vague documentation involved in getting the system set up for the first time. This definitely needs to be addressed, and hopefully will.

As far as network rendering being such a basic feature, I'd tend to disagree. I think network rendering is an advanced feature. It may be simple to you, but I know at least a dozen very talented Lightwave artists who can't even get two computers networked without extended technical support.

I'm not a programmer but I can't understand why distributing a single frame across a network is so hard to do.

This functionality exists in 3rd party render controllers. And if it were easy to write image editing software, everyone would be doing it, and Photoshop wouldn't be a juggernaut in the field.

[Lightwolf]

Originally posted by Beamtracer
I'm not a programmer but I can't understand why distributing a single frame across a network is so hard to do.
You mean for preview purposes?
First you have to find all active nodes that are idel, then tell them which files to load (the scene, objects, etc), then tell them which part of the final image to render and wait for that to happen, also having a fail safe option if one of them crashes. All of this with LW that isn't as extendable as it should be SDK wise (no way to abort a render from a script for example)...
The renderer would have to be heavily re-coded for a option like that to make sense. mental ray and final render do it nicely for example.
As for not being a programmer, try this: Just step through the process step by step, and then think of all the things that could go wrong at any time that you'd have to cater for. That should give you an indication of how hard it is. And having a way working in theory is still a lot different from actually having code that works (allthough it is a good first step).
Cheers,
Mike

[mlinde]

So I am sitting here, watching two of my three machines render.

I spent about an hour this morning trying to get Screamernet to work again. Last time I ran a network render (must have been around December) I was able to get all the machines happy and working. Today, my LWSN nodes wouldn't read the correct config files (LWSN0 was reading LWSN3!), then when they finally were reading the right files (god knows why) my fourth node (third computer) wouldn't write an ACK file. I tested my r/w access to the command directory, and it was fine, but after three startups, it just wouldn't write. Now I wonder if this is an OS X.1 - X.2 issue, where my one node that isn't behaving is 10.1.5, while the others are 10.2.6.

Anybody else have this occur with mixing 10.x version of Mac OS?

Oh, and on the "render time" topic, this scene (rendering at 320x240 w Enhanced Low AA, raytraced shadows and transparency, HV background for "thickness", with all 2600 objects moving each frame--it's a blood flow simulation) is taking about 30 minutes per frame on my iBook/600, about 15 min per frame on each node of my G4/800. Fortunately it's only about 7 seconds long.