PDA

View Full Version : Paralellel computing Newtek?



prospector
08-21-2003, 01:26 PM
Is LW and /or VT2 coded to run on a parellel computer system and not just dual processors?

Am looking into upgrading to cluster computing and the cluster company wanted to know about this, which I didn't know offhand.

prospector
08-22-2003, 01:19 PM
Any Newtek code writers know the answer to this?

Extent
08-22-2003, 07:01 PM
I don't know much about clustered computing, but if it dosen't divide tasks by threading then LW won't use it. The only multi processing options that LW has is the # of threads option in the render pannel. LW was designed around the renderfarm concept for massively MP systems. If you need more than 2 processors for a workstation computer then I would suggest finding a backplane somewhere and running one of those, but for animation rendering the best bet is a renderfarm. For single image print work it's a little trickyer, but there are plugins that will split single images up for farm rendering.

As far as I understand it, splitting the rendertasks up in a manner that would take advantage of a large cluster is not worth the overhead compared with each processor working serially with a single frame.

prospector
08-23-2003, 10:13 AM
Yep..what I called them about was clustering.
But what the rest of the computer world calls clustering, we call network rendering.

But I wanted all computers in my network to be used as 1 supercomputer (to keep the waiting for geometry updating in LW to an absolute minimum)(to do texturing in realtime)(and testrenders in realtime).

And they can do it if LW was written this way to take advantage of parralellel computing.

IE taking 10-2 GHz computers and making 1 20GHz computer workstation.

But I didn't know if it was.
Or if it even coded to run on Quad CPU boards.

wacom
08-23-2003, 10:28 AM
Are you guys talking about ScreamerNet (SN)? This is what you can use with LW for distributed rendering. It works with most plug-ins...but make sure to check before you get started on a scene. ScreamerNet requires only one seat of LW- you can then use it on any other systems in the network free of charge. You can use it with Macs, PCs, and Linux boxes. There are also several other free apps on the internet that let you do this- try www.flay.com (spider comes to mind). I don't know about VT, but DFX also allows network rendering, but you have to buy a plugin from eyeon to do it...

Hope this helps and I'm sorry if I'm way off...:)

wacom
08-23-2003, 10:29 AM
sorry...I think my last post is stuff you already know...whoops...sorry

Extent
08-24-2003, 07:13 PM
any software will automatically take advantage of as many processors as as many threads it can run in. the LW renderer will run in 1, 2 , 4, or 8 threads. So it will take advantage of anything up to 8 processors in a single box.

Network rendering is distinctly different from clustering as I understand it. Network rendering is made up of independent and self contained nodes, while in a cluster all computers work as a single unit. I don't know anything about how a clustered computer appears to applications running or what kind of software you use to put together a cluster but I would assume that LW wont work with a cluster, but that is totally out of my area of experties.

The fastest single machine you could use would be an 8 processor backplane, but that's overly expensive IMO. The most cost effective machine would probably be a basic dual system unless you've got a bottomless budget. I would venture to say that you will not be able to do realtime renders with any system. you'll still have to work with the same time saving techniques most other people use.

prospector
08-25-2003, 12:53 PM
Extent,
this is what I was informed of when digging into faster workstations.

your statement,

Network rendering is made up of independent and self contained nodes,
this is correct


while in a cluster all computers work as a single unit.
incorrect, this is what we call network rendering (clustering to rest of PC world).


while in a cluster all computers work as a single unit

incorrect, this is called Parallel computing.

That's what I want to do but can't get info if LW will work in that situation.

Company I called said the Application must be able to do it.

toby
08-25-2003, 11:09 PM
what kind of system can run on all those processors?

good luck, we'd all love to hear if you can do it - and how much it costs

prospector
08-26-2003, 12:34 AM
Linux as base OS
but can also use Wine to run LW over Linux and they say it will work just fine if LW can handle it.

Extent
08-26-2003, 03:57 AM
Originally posted by prospector

incorrect, this is what we call network rendering (clustering to rest of PC world).



I might not have been quite clear, at least in the way I see things (I have never dealt with any clustering software, nor have I ever researched any, so I probably am misusing terminology) With screamernet running as a renderfarm, each node is a complete selfcontained renderer. No node interacts with any other node, each node completes one unit of work on it's own, turns it in, and is finished. If you're render is only a single frame, only one node is taken advantage of. It's alot of small pipes putting out data each at their own rate, just all into the same pile. I see clustering as more of an extension of multi processing within the same box, where one work unit is devided into many paralell tasks and the whole "cluster" reports to one location, kind of like the polar opposite of a Citrix server. So one big pipe pumping out data. A screamernet farm does not appear as one big "virtual" computer. That's what I meant by my comments.

AFAIK LW has no documented multiprocessing functions other than the threading you set in the renderer, if the clustering software takes advantage of threading to devide tasks to the different nodes on your network then you're golden for up to 8 processors. It will only work for the Layout application tho, as the Screamernet renderer is not multi-threaded, that would have to be run in the traditional manner.

prospector
08-26-2003, 10:40 AM
A screamernet farm does not appear as one big "virtual" computer. That's what I meant by my comments.

Correct, a cluster=screamernet
Paralelle= virtual computer

Like the new one just built with 300 PS2s networked together to get 1 400GHz computer, (with all program codes and instructions being computed by all computers at same time.)

But another point I just noticed..

The threads in LW means that each thread will go to a different CPU? But presently 'only' on the same Mobo?

And when LW starts to render and it says 'Optumizing polygons", then that specific code to do that is going to all CPUs on that mobo? or is it being done by only 1 CPU?

Extent
08-26-2003, 06:02 PM
Your OS handles how the threads are excecuted. You can run 8 threads on a dual machine, and the OS will ballance which threads runs on which processor. If your paralelling software will assign threads to processors on the rest of the network then you'll be ok. I dont know how the software might work, or even if it can run normal applications that aren't specifically written to take advantage of that kind of processor setup. Thats why I'ld immagine that that kind of setup would not be quite as fast as a 1 box solution, because you've got additional latencies between each "processor" on your system.

Certain parts of the renderer are multi-threaded, other parts arent. Optimising the geometery is not, it will only run on a single processor, precalculating interpolated radiosity is also only single threadded. I think most of the "setup" part of the render will only run on a single thread. The part of the renderer where you can see the polys being drawn if you have the display window open is multithreaded, all of that processing is distributed in as many threads as you have set. It segments the single frame into as many horizontal strips as you have threads set, then it assigns a thread to each strip. What is done with each thread after that is up to the OS. Some plugins are mulitthreaded and some arent as well, it depends.

bradl
08-26-2003, 06:47 PM
Why re-invent the wheel?

Massively Parallel Raytrace Rendering Project: Kilauea

http://www.hi.is/~arnij/pdf/kilauea.pdf

Cageman
08-27-2003, 12:44 PM
Originally posted by Extent
Screamernet renderer is not multi-threaded, that would have to be run in the traditional manner.

Extent: We have two dual-machines at school and alot of single machines. It would be nice to take advantage of those dual machines in screamernet, so my question is: Can I add two screamernet nodes on a dualmachine, give them uniqe numbers and they would act like two single cpu machines?

bradl
08-27-2003, 12:59 PM
Cageman,

Yes you can run two nodes on a dual, just give them different numbers. Actually, on a hyperthreaded machine, like my Dual Xeon 2.2, you can run four nodes.

I now use Spider (and recommend it- it is now free) and it allows multithreaded rendering by creating additional user accounts in the background.

ResPower
08-27-2003, 03:47 PM
Originally posted by Cageman
Can I add two screamernet nodes on a dualmachine, give them uniqe numbers and they would act like two single cpu machines?

Answer: Yes.

BUT you can also change the .cfg file to make LWSN run in multi-threaded mode. The old "LWSN doesn't run in multi-threaded mode" thing is a hold-over from 5.6 days. However it's not a good idea to run it multithreaded, as many plug-ins will crash if they're not written to handle it. Also many of the pixel filters for LW will drop the screamer down to one processor - so when you watch the task manager, the system will peg out at 100%, then drop down to 50% when LWSN is doing something like a pixel filter.

Hope that helps!

prospector
08-27-2003, 06:38 PM
Why re-invent the wheel?

don't want to, just want LW to be the Craiger Super Sport Mag IN the wheel:D

And that link is just what I want !!! Cept for LW to run on it...

Cageman
08-27-2003, 06:48 PM
Thanks alot for your answers!! :) And yes... I´ve noticed that some stuff, like Hypervoxels, doesn´t work with multithread. :/

One can just hope that those who write plugins will have multithread in mind... :)

richpr
08-28-2003, 03:54 AM
LW is not a cluster aware application...

What a cluster aware application generally is able to do:

. Read/write to files with other nodes in the cluster, without worrying about them being overwritten... Like 10 copies of LW working on the same picture... Which is not the same as SN, which is use as well... With network rendering they all do strips or images in an anim, but no two nodes touch the same file.

. They are status and cluster aware. If one of the nodes creashes in a cluster, it may come back online automatically and know where to continue. Alternatively if it doesn't come back up, another node would do the job. Some SN managers are able to keep track of which frame are rendered as well by scanning directories...

In general, clusters are used for high-availability solutions for guaranteed up time...

prospector
08-28-2003, 09:14 AM
OK thanks.

So I am now looking at Multi CPU boards.

But then there are those plugins that can't use Multi CPUs.

ARRRGGGGG

bradl
09-02-2003, 03:07 PM
Originally posted by prospector
don't want to, just want LW to be the Craiger Super Sport Mag IN the wheel:D

And that link is just what I want !!! Cept for LW to run on it...

...and when you figure it out, let me know too!

BTW, I had a set of Craiger SS on my first car years ago, with 50's on the back...

prospector
09-02-2003, 06:26 PM
Weren't they the best tho??

Memories
in the corner of my mind
misty water colored memories
of the way it was:o

'68 firebird
400 cu in (no goofy liter crap)
6 pack carb
and Craiger mags all around
jacked up 8 inches in back (better to look for coins on road;) )
12 in wide road ripping tires on rear.
able to burn rubber in first 3 gears...

Oh God...I'm misting up.....:D