PDA

View Full Version : Some questions about node rendering and renderblades



Svenart
08-28-2012, 09:10 AM
Hello,

Im thinking to buy 1-2 renderblades and use them for node rendering in Lightwave.

My main problem is that I cannot find good renderblades for reasonable prices here in Germany.

Does anyone knows a good reseller for renderblades in Germany or a store where I can buy renderblade cases?

I never have used node rendering before, Is it very difficult to set it up in lw11?

Danner
08-28-2012, 10:36 AM
Can't help you with the blades but It's very easy to set up a lightwave render farm using Amletto.
http://virtualcoder.co.uk/amleto/

Cageman
08-28-2012, 01:06 PM
http://forums.newtek.com/forums/showthread.php?t=129912

:)

Hieron
08-28-2012, 02:11 PM
Why blades? That's expensive...

If these are your first nodes, perhaps best would be to get some fast consumer machines in normal smallish boxes? Sets you back anywhere between 700 and 1.050 euro depending on the system and you'd get a very high price/value result.. Either based on the intel 3770k or the 3930k.

A nice way to compare them is to calculate the price in euro per Cinebench point. About 90 to 100 euro per Cinebench point is what our nodes do. If you overclock them it may reach 75 euro per Cinebench point.

On the setup of the rendering: we have very good experiences with Butterfly Netrender. I would certainly not consider using native screamernet.

ps: that 3930k will reach 10.15 cinebench points:
http://www.bit-tech.net/hardware/cpus/2011/11/14/intel-sandy-bridge-e-review/4
and a node can be built for a price around 1.000,- euro. hence ~100 euro per CB point.

ps: if you got the money, you can push for dual Xeons. But the euros per CB point starts to rise to roughly 200,- and no option of overclocking there. Imho, if you have no space concerns etc, best to keep the nodes consumer grade. Your main workstation however could become a dual xeon sooner.. these tend to be more expensive anyway (better disk subsystem, GFX card etc) so the cost of the dual Xeons and server motherboard levels out a tad..

JonW
08-28-2012, 03:05 PM
Why blades? That's expensive...

If these are your first nodes, perhaps best would be to get some fast consumer machines in normal smallish boxes? Sets you back anywhere between 700 and 1.050 euro depending on the system and you'd get a very high price/value result.. Either based on the intel 3770k or the 3930k.

A nice way to compare them is to calculate the price in euro per Cinebench point. About 90 to 100 euro per Cinebench point is what our nodes do. If you overclock them it may reach 75 euro per Cinebench point.

On the setup of the rendering: we have very good experiences with Butterfly Netrender. I would certainly not consider using native screamernet.

ps: that 3930k will reach 10.15 cinebench points:
http://www.bit-tech.net/hardware/cpus/2011/11/14/intel-sandy-bridge-e-review/4
and a node can be built for a price around 1.000,- euro. hence ~100 euro per CB point.


Agree!

I've only got my dual Xeon boxes because they started their life as workstations where I wanted & needed as many GHz in one box.

But for render nodes it's a waste of money. If you have the space for a handful of computers as already mentioned buy some cheap Bare Bones Boxes or build your own. A 3930k seems to work out to give you the most GHz per whole box setup. Add up the cost of the box & parts including OS & divide this overall price by the Total GHz. Or better still divide by for example a Cinebench benchmark.

Also try to get a MB with built in graphics, failing that borrow an old graphics card from an old computer to set it up. Or if you do insist keeping a graphics card in the box get the cheapest most power efficient card you can otherwise you are only wasting money & electricity. Buy the cheapest cases you can find, smallest hard drive etc. The one thing I would spend a bit of money on, is if you you start collecting nodes & they are in the same room as you & or you have a challenging cooling or noise situation is get a Tower cooler for the CPU. My farm still runs relatively quietly even if the room gets to 35 plus degrees with all the nodes running with 2300 watts heat being added to the room.

Another thing about buying nodes with BBB is that you can add nodes as funds become available. I would probably start with 1 extra node, get that up & running. Once you have ironed any problems then get say another 2 nodes to double your capacity again.

JonW
08-28-2012, 03:14 PM
I never have used node rendering before, Is it very difficult to set it up in lw11?

Even if you just use Amletto or something similar. It's worthwhile setting up Sceamernet, it's a pain in the bum to get going but it does give you an insight into how it all works. Read Matt's SN tutorial it's very good.

I would get your workstation going first. Once that is set up then your first additional node. Once these 2 computers are working then getting additional nodes working is easy.

Don't forget that you need to save all your images objects scenes etc with_no_gaps-in-their-file-names!

khan973
08-28-2012, 06:12 PM
Render blades are the way to go if you plan on buying more of them.
They are easier to stack.
If you plan on just having 3 or 4 of them, don't bother just get small desktops.

I have Nec Flexpower, they are silent (kind of...) use less energy and also have memory management.

When you build a renderfarm, they are many things you need to take care of to be sure it's going to work.

1st, create a shared hardrive that is accessible by all of them and give the ame letter everywhere (W: ie).

then, install your server / client renderfarm software.
Some will copy whatever they need, some other will require you to copy the whole LightWave 3D folder (+ license + config files) somewhere accessible to every node.
Beware!!! all the config files should point to the shared folder. You can edit them with Notepad ++ wit replace all...
If you don't do that, when rendering, it will look locally for your plugins and won't find it.

Once your rendering software is setup properly, tyou're good to go.
Just copy your files to be rendered to the shared folder. Be sure to respect LightWave's subfolder system so it doesn't look for files on your desktop or hard drive while it should find it locally.

Thinking renderfarm is tricky.
Sometimes you have to bake textures.
You definitely want to break your animated textures into image sequences so it loads only 1 image for each frame.
All dynamics or special motion expressions, plug-ins or deformations should be baked (MDD).
GI also have to be baked.

If you don't do this you might not get the needed result at all or it can be different on each node.

You might also want to une semething like VLC to control the different nodes without plugging a screen.
I personnally have a rackable Screen http://www.ipctechnology.com/images/products/rack-console-kvm/smk880/smk880s17.jpg
and a rack mounted KVM to use only this screen / keyboard / mouse accross 16 nodes


Rendering on renderfarms isn't necessarely hard but it might take some time to figure out all the possible mistakes you can make that avoid it to render properly. I personnally lost a few hair on the way but now I'm fine ^_^

Hieron
08-29-2012, 03:12 PM
Render blades are the way to go if you plan on buying more of them.
They are easier to stack.
If you plan on just having 3 or 4 of them, don't bother just get small desktops.

I have Nec Flexpower, they are silent (kind of...) use less energy and also have memory management.

For sake of comparison, could you list the price per blade (including their part of the powersupply and rackspace cost etc) and the benchmark of Cinebench R11.5 multi cpu?

We have 10-15 nodes and they are still midtower cases. I'd be very very interested to know your actual euro per Cinebench point ratio to account for "easier to stack". Blades are indeed more compact and easier, but this is at a cost and I'm very interested in the actual cost per rendering unit divided by its Cinebench Points..

Many thanks for any info you can give on those accounts!

khan973
08-29-2012, 03:44 PM
We bought ours a few years ago 10k€ - you can find it for 6k€ / 8k€
NEC FlexPower Server (6 nodes)
Comput Module with Bi-E5420 CPU (8core total) / 4GB
Memory
14 x 148GB 10Krpm 2,5" SAS Hotswap HDD
4 x 1000W Power Supply Unit

So brand new it costs less than 1.5k€ per blade and second hand, about 800€ to 1k€

For Cinebench, I'll do it when I have time.

But I couldn't see myself having a stack of computers with all the wires, heat and so on.
You have to consider gettinh a Rack mount too which can be expensive, especially if you want to ones that remove noise and keep temperature low.
This setup is easy to move, unplug, cheaper if you rent a space because you have more room left for let's say 1 or 2 desks for people to work. It is also more eco-friendly.

cosbovfx
08-29-2012, 09:16 PM
I have been looking to upgrade my Farm, I have pretty descent setup consisting of

1 dual e5440

1 dual e5607

1 dual e5430

2 dual e5345

5 quad q6600 2 of which are OC'd to 3ghz

2 core2duo 2.4

2 core2 1.83, one being the render controller

I Have been looking into These

http://www.ebay.com/itm/Dell-Poweredge-XS23-SB-C6100-2U-8x-QC-2-5GHz-64GB-RAM-4-servers-in-one-chassis-/390458714572?pt=COMP_EN_Servers&hash=item5ae92729cc

These seems like a lot of processing power for the buck, and the space they take up.

Does anyone have any thoughts on these.

Or could someone point me to something with that much processing power for so little money

JonW
08-29-2012, 10:30 PM
These are the 2 main issues!

Cinebench benchmark per dollar?

Energy consumption of old computers?

Hieron
08-30-2012, 04:57 AM
...

Thanks for the info.. would be nice to see some info (price and CB points) on current/new machines..

biliousfrog
08-30-2012, 05:07 AM
I just use standard rack units...space and 'tidyness' of blades but much cheaper and easier to maintain.

Bare in mind that any rack based computers will be noisey as hell though, you'll either want them in a separate room or inside a soundproofed, air conditioned enclosure.

cosbovfx
08-30-2012, 05:28 AM
For the unit I posted they run 338watts total at full load for what is essentially 4 computers that seems descent to me , but I don't have anything to compare it with.

Hieron
08-30-2012, 05:38 AM
Standard rack units would not be an option for us, probably same as blades. Our farm is atm in the office itself. Very much doable with normal cases and tower coolers, not much noise at all..

And easy to take out a node to make into temporary workstation and back..

But maybe one day... imho blades run from 1.5x the price per CB point and up..

DogBoy
08-30-2012, 05:41 AM
Bare in mind that any rack based computers will be noisey as hell though, you'll either want them in a separate room or inside a soundproofed, air conditioned enclosure.

Quoted for emphasis. Those things can not only be loud, but extremely annoying. Keep them well away from where you work, as the constant whine can break even the strongest ;)

JonW
08-30-2012, 06:21 AM
For the unit I posted they run 338watts total at full load for what is essentially 4 computers that seems descent to me , but I don't have anything to compare it with.

My dual E5450 (80 watt each) & W5580 (130 watts each) boxes use about 520 & 560 watts with a GTX280 (idle or low usage (I've had the 280 in both boxes)) when rendering. The W5580 CPU when idling uses a lot less power than x53xx & x54xx CPUs.

The current CPUs have really good Rendering & idle power performance. I would use a 3930k as a starting point for a benchmark for: Energy, Rendering etc, & then see if you can improve on this. If you need more than a dozen computers, Boxes maybe not looking so good. In summer I do need to turn on the air conditioning when the farm is running.

E5450 3.0 GHz, W5580 3.2 GHz, typically my W5580 renders twice as quite as my E5450 (CB 7.17), & the W5580 (CB 11.39) is over 3 years old now! A new E5-26xx for dual CPU box or the 3930k single CPU box is where I would start from. For render nodes the low end E5-26xx are going to work out better price performance ratio, you just need to calculate a whole box setup for each CPU to find the crossover point, & see if it's any better than a 3930k?

Don't forget power supply, cases & other things are more expensive for server boards. But you do have less boxes, one OS etc.

JonW
08-30-2012, 06:27 AM
Quoted for emphasis. Those things can not only be loud, but extremely annoying. Keep them well away from where you work, as the constant whine can break even the strongest ;)

Tell me about it!

http://forums.newtek.com/showthread.php?t=100802

"If you are thinking of getting a pair of top end x55xx & putting them on to a SuperMicro X8DAL-3 motherboard & dont want to wear ear muffs & sit in the next room........"

cosbovfx
08-30-2012, 09:21 AM
My dual E5450 (80 watt each) & W5580 (130 watts each) boxes use about 520 & 560 watts with a GTX280 (idle or low usage (I've had the 280 in both boxes)) when rendering. The W5580 CPU when idling uses a lot less power than x53xx & x54xx CPUs.

If I understand correctly your dual 5450 system with a gtx280 when rendering uses 520-560watts.

If that is the case then these systems are pretty power efficient. I am talking about 4 dual 5420 systems, and they run 338watts at full load for all 4 systems

For $292 per dual 5420 with 16gb ram sounds really good still

Based on what I found on the web a dual 5420 system almost identical to the one I posted scored a 6.0 on cinebench. So for all 4 systems you get a score of 24 per 2U for $1168.

Based on my systems, my dual 5440 has a CB score of 6.77
My dual 5430 is CB score of 6.27

The 6.0 score for a dual 5420 sounds about right.

Now the really question for me is If I could build a new system for $1168 and it beat out the combined scores of the 4 dual 5420 systems. If I could configure something like that I would go that route.

But I really don't think that is possible because the system would have to have a CB score of like 24+

Hieron
08-30-2012, 01:54 PM
You won't be able to beat that price for 24 CP points indeed, especially compared to new. A 3930k node would be about that price, new. Though I doubt it is 338W for the whole thing...? More probably ~300W per node (thus dual Xeon), 1.2kW for all 8 Xeons thus 24 CB points? Whereas the 3930k is about 250W for 10.15 CB points. So about 2x more efficient energy wise... (they better be, 4 years newer cpu's)

In short, seems like a good deal if you want max speed for little price, can live with the noise and lower energy efficiency. Ofcourse, if you run these older cpu's constantly in a farm, at some point the extra powerdraw will negate the cheaper price... all depending on energy cost and running time..

ps: handy chart:
http://www.cbscores.com/

Cageman
08-30-2012, 03:38 PM
Hello,

Im thinking to buy 1-2 renderblades and use them for node rendering in Lightwave.

My main problem is that I cannot find good renderblades for reasonable prices here in Germany.

Does anyone knows a good reseller for renderblades in Germany or a store where I can buy renderblade cases?

I never have used node rendering before, Is it very difficult to set it up in lw11?

A couple of thoughts about renderblades vs workstations and from my 8 year+ experience with such things...

1) Renderblades are glorified workstations
2) Renderblades requires a hell of a lot more cooling compared to regular workstations since they are "designed" to fit into rack-mounts. Smaller chassi = needs more external cooling, such as a serverroom with a pro AC-setup!
3) A renderblade gets old as fast as workstations gets old. Still, they cost 3-5 times more than a similar workstation.
4) It is much, much more costeffective to purchase regular workstations for small farms (3-10 computers).
5) Renderblades are overpriced! We got six regular workstations of the price of one renderblade. The combined processing power of those six workstations compared to one of those renderblades was in the latitude of 5 to 1 (meaning that we got 5 frames out the door in the same time one of our renderblades produced 1 frame).

The combined energy consumption is larger, but on the other hand, those regular workstations are designed to work under more preassure, and one have to ask if the additional AC units needed for renderblades, will be as costeffective as adding regular workstations to the farm?

JonW
08-30-2012, 04:02 PM
You won't be able to beat that price for 24 CP points indeed, especially compared to new. A 3930k node would be about that price, new. Though I doubt it is 338W for the whole thing...? More probably ~300W per node (thus dual Xeon), 1.2kW for all 8 Xeons thus 24 CB points? Whereas the 3930k is about 250W for 10.15 CB points. So about 2x more efficient energy wise... (they better be, 4 years newer cpu's)

In short, seems like a good deal if you want max speed for little price, can live with the noise and lower energy efficiency. Ofcourse, if you run these older cpu's constantly in a farm, at some point the extra powerdraw will negate the cheaper price... all depending on energy cost and running time..

ps: handy chart:
http://www.cbscores.com/

8 x E5420 = 8 x 80 watts = 640 watts. Then you need power for the MB, ram, inefficient CPU fans & other stuff, also the power supply is probably around 85% efficient.

So around 1200 - 1300 watts would be about right. Don't forget these CPU idle is relatively power hungry compared to new CPUs. (I can't remember the exact figure but my E5450 & W5580 were about 390 & 240 watts. It was a clear difference because my EvolutionS 3.0 kVA UPS showed a very notable lower power consumption with the W5580 & at the time I was duly impressed.


There is no way one can replace the noisy fans. I can assure you that server fans scream! I had the same problem with my E5450 & E5335 boxes (The E5335 was even worse it has 2 screamers which I had to keep but worked around the problem fairly well with soundproofing, it also had a 20 watt 160mm fan which was like a leaf blower, I replaced this with 7 Noctua fans). One will need a separate room for the servers & you need to add "Rockwool" acoustic batts if you have't already got them.

JonW
08-30-2012, 04:24 PM
Aand one have to ask if the additional AC units needed for renderblades, will be as costeffective as adding regular workstations to the farm?

A few years ago over summer I was rendering frames for 3 weeks continuously through a heatwave (had no choice). At times the room was getting close to 40C with our limited air conditioning. All my boxes with Noctua heatsinks were fine, CPUs coped really well.

The only box that had an issue was the E5335 with the Intel server CPU 7000rpm fans (& had clean heatsinks as I had pulled them out cleaned them plus replaced the thermal compound. The original is a thermal pad which only covers about 60% of the CPU if you are lucky), the over heating alarm went off (severs have these) I had the case open & 2 domestic fans pushing extra air through it, & it was also in the coolest part of the room. It was touch & go but I got through.


Separate soundproof rooms with air conditioning are a must if you want server computers to survive, & your sanity!

khan973
08-30-2012, 04:45 PM
Our renderfarm used to be in an opened corridor so we didn't get that much noise and the air was flowing pretty well.
Now we moved and had to put is in a small room...man, it's so noisy, and hot! Somebody thought it would be smart to close the door for the noise. The renderfarm shut down and when I got in the room every single object was dead hot! I couldn't hold the rack mount for more than 3 sec... that's how hot it was.
We'll buy a rack that cuts the noise (-18db) right now it's 75db 1 meter away and 90 close to it... regular working environment is between 50 and 60.
The rack mount also has a cooling system.

cosbovfx
08-30-2012, 04:57 PM
They are actually L5420's. Here are some power usage pics.

JonW
08-30-2012, 05:11 PM
The L5420 are 50 watt CPUs instead of 80 watts for the E5420.



PS. I've got one of the Steplight (wattsclever) power meters it seems to be fairly accurate.

JonW
08-30-2012, 10:47 PM
They are actually L5420's. Here are some power usage pics.

I'm a bit confused here. In the Kill-A-Watt picture it says 338 Watts 2 x L5420 CPU.


If that is the case then these systems are pretty power efficient. I am talking about 4 dual 5420 systems, and they run 338watts at full load for all 4 systems

Here it says 4 systems 338 Watts. So if you just look at the CPUs & there are 8 of them. That's 8 x 50 watts = 400 Watts just to run the CPUs.

The rest of the system will use roughly the same amount of power whether is has 8 x 50 Watt L5420 (400 Watts). 8 x 80 Watt E5420 (640 Watts) or 8 x 120 Watt X5460 (960 Watts)

So with 8 x L5420 CPUs I would say it's going to chew through roughly 1000 - 1100 Watts.

cosbovfx
08-31-2012, 07:18 AM
I am not really sure, I don't own these yet, I am looking into them. The information is from a reputable seller that I have purchased systems from before. I will seek clarification. From what I understand these systems share power supply, fans and backplanes, which is supposedly why they are so efficient

Lightwolf
08-31-2012, 11:42 AM
Does anyone knows a good reseller for renderblades in Germany or a store where I can buy renderblade cases?
Did you try http://www.thomas-krenn.de ?
Froma certain sales volume onward it may make sense to call them up for a discount (the same goes for pretty much everybody else, Dell being a good example).

Cheers,
Mike

cosbovfx
08-31-2012, 12:20 PM
Alright so he made some clarifications to me.

He said that the 338watts is for one powersupply, which powers 2 nodes, so 4 CPUs. So for the whole system he said you should double that so around 780-800watts for all four nodes running at full load. Which seems about right based on the following.

You can determine how much power a Dell system will require using the ESSA app on dells site. I setup a current generation c6100 4mb system with similar specs, but using

L5630 cpus (40watts)

12gb ram per node (there is no 16gb option)

1 500gb HD per node (he said he had 80gb in for his tests (but the system does not come with any, just caddies))

1100watt psu (the system I am looking at have 500watt psu)

The software told me 794watts for the system at full load.

For the same system but will L5640 cpus (60watts) it said 945watts