PDA

View Full Version : Chilton Xgrid !



RonGC
11-23-2006, 11:45 AM
Hi Chilton, tell me that you guys are going to add Xgrid support to the UB.

No really, tell me! LOL. I have a few spare Nodes laying around here that could use an Xgrid savvy LW app to put them to work.

Xgrid has to be one of the least generaly supported yet most useful apps on the Mac, heck if you can use it to "Map the human Genome" surely we can use it to render animations.

Science knows the value of the app when doing massive 1000 man year calculations. Hint, Hint...

Ron

Chilton
11-23-2006, 12:28 PM
Hi Ron,

No support for XGrid has been announced.

-Chilton

toby
11-24-2006, 12:20 AM
I don't think X-grid would be a big advantage over network rendering for animations would it?

RonGC
11-24-2006, 01:10 AM
Xgrid will speed up any computations by using all your available computers as a cluster.Think super computer, these are what they use to model weather simulations,etc. When you have a lot of calculations and need more processing power just add more computers to the cluster, skies the limit. Each processor(node ) is available to work on the calculations. Think for example 10 , 20, 100 or more processors dedicated to rendering your scene.

I have 9 nodes available here,soon to be 13, this is major computing power unharnessed. Xgrid is the controller that makes sure that all the processors are fully utilized for the task, no nodes sitting idle at any time during the task. Handles all the movement of tasks to the various processors and collection of data and assembles this into the finished output (render).

Much more powerful than a simple render farm.

Ron

kip
11-24-2006, 07:02 AM
Is this related to the xgrid question?

http://www.newtek.com/forums/showthread.php?t=59592

RonGC
11-24-2006, 11:20 AM
Yes similar, Xgrid is really set up to use all its cluster service nodes to process a given task. Qmaster is designed to control the availability of service nodes for many simultaneous projects. You can prioritize tasks so as one task is finished the nodes are then added to the cluster working on the next priority task. Great if your working with several projects at the same time.

All in all the 3d app that can utilize these programs will be the king of Rendering speed. Processor for processor no other non cluster network rendering system can compete.

Ron

swpspce
11-24-2006, 01:43 PM
Yes similar, Xgrid is really set up to use all its cluster service nodes to process a given task. Qmaster is designed to control the availability of service nodes for many simultaneous projects. You can prioritize tasks so as one task is finished the nodes are then added to the cluster working on the next priority task. Great if your working with several projects at the same time.

All in all the 3d app that can utilize these programs will be the king of Rendering speed. Processor for processor no other non cluster network rendering system can compete.

Ron

and what is the speed of your network - 100Mbps... as we discovered here, 100Mbps was too slow and had to upgrade to 1000Mbps to *really* take advantage of a disrtibuted render system...

kip
11-24-2006, 02:06 PM
and what is the speed of your network - 100Mbps... as we discovered here, 100Mbps was too slow and had to upgrade to 1000Mbps to *really* take advantage of a distributed render system...

Well it might depend on a variety of variables but 100t should be fine (not that gigabit isn't better, especially on an all OS X network). As long as machines are spending more time rendering than they are transferring files any network render scheme is better than none.

Qmaster or xGrid combined with something like RenderFarm Commander would be a welcome development. Qm or xG providing a solid way for RFC to discover machines and get them in and out of the cue as they are available or needed elsewhere. That lets mixed use studios gain the most cpu time from machines that may not be dedicated to 3D without requiring users to have any network/render savy. It has the potential to become as easy as turning on iChat.

John the Geek
11-24-2006, 02:15 PM
:agree:

http://dvt.science.nd.edu/Qmaster_Maya.jpg

I definitely doesn't say Lightwave, does it?

=(

swpspce
11-24-2006, 04:10 PM
:agree:

http://dvt.science.nd.edu/Qmaster_Maya.jpg

I definitely doesn't say Lightwave, does it?

=(
although I have never tried it - Qmaster does include a generic setup template for Lightwave 8 - though....

eblu
11-28-2006, 09:03 AM
gentlemen.
asking for xgrid and Qmaster for Lightwave is not as easy nor as beneficial as it first seems.

3d rendering is Not really a great candidate for clustered computation. the problem lies in the resources that go into a render. Clustering is very powerful when the i/o is lightweight, that way the data (think mathematical formulas, NOT Image Data) can be moved around without crippling the network. But in 3-d we have scene files, models, and image files. Clustered rendering solutions must be able to read those resources, and that means that when your clustered processors are trying to render (in a cluster ALL the processors try to render one frame at a time), each machine is trying to get the same resources... which is a great way to bring your network to its knees, especially with image sequences.

So don't assume ANY clustering software is necessarily good for 3d rendering.

and QMaster?
its no different than Lightwave's default Network rendering controller. If Newtek released a Version Of Screamernet that ran in the Mac os X Terminal, I could get Qmaster to issue the appropriate commands in a few hours. But I wouldn't. There are already better tools out there, and even better ones could then be built from scratch Much easier.

What we need is a screamernet that runs as a Unix tool in Mac os X. Let me rephrase that so that it illustrates itself MUCH better. What we need is for the rendering engine for ALL of Lightwave to be turned into a Unix Tool that runs in the mac os X terminal, such that we can issue -2 or -3 commands from said terminal the exact same way Lightwave Itself does.

In that way Screamernet will become drastically more flexible, open itself up for more 3rd party support, applescript support, unix support (which by itself is EXTREMELY powerful), and several layers of bugs, legacy technology, and support issues will disappear.

kip
11-28-2006, 11:24 AM
Indeed, xgrid may not be an appropriate answer. I'm not a programmer so I was wondering if the existance of a "Generic Render" entry might point to some hooks in Rendezvous/Bonjour that programmers could use to simplify the "find available nodes for render" part of network rendering setup. Also anything that would make it easier for people to add their machines to a network render when they go home at night would be welcome. Rendezvous/Bonjour seems to employ a lot of network-watching/updating already so maybe it could save someone writing a lot of code.

eblu
11-28-2006, 03:02 PM
kip,
I looked into screamernet/Qmaster integration a while back. I'm going from memory but I believe that you'd have to build an intermediate application that :
1. performed all of the appropriate network traffic for screamernet
2. translated all of the Qmaster stuff into something that screamernet likes (job file)
3. translated back to Qmaster from Screamernet (ack file)
4. and made sure everything was going ok.

the problem is... its a LOT of work with very little ROI. And there are already some excellent 3rd party controllers out there. take a good look at render farm commander, its just about perfect.

sure it'd be nice if Newtek would embrace some apple technologies, or in screamernet's case, any modern technologies. But polishing a rotten apple can only make it look pretty. Screamernet has languished now for over a decade, and on the mac it has suffered more than on the pc.

a more modern approach which I am hoping for, means a great deal of change under the hood for Lightwave, but in the long term will be a massive net benefit for the whole app.

1. take the rendering engine out of Lightwave. make it a unix tool in mac os x (I'm not sure of the terminology for Windows, so I'm ignoring it)

2. set lightwave up to make unix calls to the rendering engine for its rendering. (separation of UI and implementation, it greatly increases your ability to fight bugs, support the app, and increase Development.)

3. 64 bit the renderer. and make it respond to user commands from the terminal. You don't need a 64 bit ui. But you "could" benefit from a 64bit renderer.

4. release clear and concise documentation about how to work with the unix based renderer. develop a simple external controller... release the source code as shareware. encourage 3rd parties to release low cost render controllers of their own design.

writing a slick app that makes unix calls is a Very simple and straight forward process on mac os x. I've learned how, and done it in a few hours myself, and I am Not a seasoned pro. there are tutorials, and help files all over the internet.


these changes imply a Lightwave that runs under XCode. You realistically can't make a unix tool for mac os x with any other dev environment, so I am hoping that this becomes the dev path newtek follows with lightwave for mac os x. its true to LW's industrial strength roots, and puts a great deal of flexibility and power into the end user's hands.

kip
11-28-2006, 03:15 PM
take a good look at render farm commander, its just about perfect.

That's what we're using now. Its great but for us the "just about" part is getting non-lightwave computers logged in/out of the render farm. Its so un-apple'ish and inelegant to just "crash" things . . .

Thanks for the detailed breakdown.

I hope someone at Newtek looks through your post and gets an idea or two. I suspect more than a few people will soon be looking for an alternative to Vista.