Any problem with mixing P4 and Athlon64 in a render farm?

[Scott Gammans]

Hi,

I'm slowly building my own Screamernet render farm and want to know if there is a problem with mixing CPU types. My plan is to build a 3.2 GHz Pentium 4 and an Athlon64 3200+, compare the rendering performance of the two, and build future render nodes based on the performance champ... but keeping the loser in the mix as well (I'm a one-man show and don't want to waste what would still be a perfectly good P4 or Athlon64 node).

I seem to remember reading somewhere that there was a problem with mixing Macintosh and Windows machines in the same rendering network, but was the issue the operating systems or the CPUs themselves?

Thanks!

[Exper]

Once and for all: CPU!

Don't even mix AMD and Intel CPU!

Bye.

[Kurtis]

Since AMD and Intel processors do not perform the necessary calculations in the same way, you will get inconsistent results if you mix processor brands.

[Scott Gammans]

Originally posted by Exper
Once and for all: CPU!

Don't even mix AMD and Intel CPU!

Bye. Oooookaaaay... sorry if this question has been asked before but I did try to search for the answer first using "Althlon64 AND Intel" for the search terms. I waded through through almost two pages of results before I gave up and posted this inquiry.

Thanks for the reply and sorry to have asked an (apparently) obvious question that's been answered before.

[Scott Gammans]

Originally posted by Kurtis
Since AMD and Intel processors do not perform the necessary calculations in the same way, you will get inconsistent results if you mix processor brands. Thank you... that was a much more informative and definitive response. I think I will still build one of each machine, but then either sell the slower one or use it for something else.

[lots]

uh.. well coming from someone who's used a wide range of cpus in his render farms.. mixing athlons p4's p3s and p2's works perfectly fine as an end result.. at least it did for me. The biggest problem i could see arising from using different cpu's and architectures is the numbers could be calculated differently.. thus some pictures might not be exactly the same as if they were rendered by the same cpu type.

but from my results of these renders there was really nothing any different that i saw, in other words you really ahve to be looking for it.

and if you think about it all it really does is send a render instruction to the render node, the render node is running the network rendering agent (compatable with it) it produces a picture which should look basically the same as the others .. and it send the picture back to the host node.. quite technically from this view point it should work on any system with the right software for each different architecture and the ability for those computers to communicate to the host

[Kurtis]

lots is correct. It will work across the different CPU brands. The problem is in the way the brands perform the calculations.

The difference will be most evident in parts of a scene that are more particularly dependent on calculations being performed in exactly the same way, such as procedural textures.

[Scott Gammans]

Originally posted by Kurtis
The difference will be most evident in parts of a scene that are more particularly dependent on calculations being performed in exactly the same way, such as procedural textures. I can see where that would be a problem... in an animation it would make a surface appear to "crawl" or "shimmer" as the slightly-differently calculated textures from different CPUs are interleaved together from one frame to the next.

You guys must be utilizing a lot of assembly-level instructions for the brand of CPU to make a difference. I'm almost afraid to ask this, but will there also be a difference between Pentium models? For example, will I get a different result between a Pentium 4 with a 533 MHz front-side bus and a P4 with an 800 MHz FSB? Won't that play havoc when newer generations of Pentiums roll out the door (or in an AMD shop, when newer Athlon64 chips debut)?

[Kurtis]

Intel seems to stay pretty well consistent. Some users have reported some minor differences, but nothing to the level of using mixed brands.

[Freak]

There did used to be slight difference, on Procedural textures between, Mac, Intel and AMD......


I can't confirm this with the Athlon64.....
However i do remember the debate on Athlon's and P3's
And the conclusion was that Intel P3's were technically at fault...
(due to a maths bug) (possibly a deliberate one)

But again, that was just a thread, and i can't claim any validitity...
of who was correct, or if it's changed since then.

[Draven]

AFAIK this problem was largely in LW 5.6 and earlier.

[Exper]

Scott...
don't mix even different Intel models if you want be 100% sure about the rendering results!

This is expecially true when you use Radiosity, Caustics and Volumetrics (due to the specific calculations)!
Many problems was resolved from LW5 to LW6 and then LW7.

As far as I can say... every Commercial Render Farm always use the same CPU (same model) for all the nodes.

Bye,

[Scott Gammans]

^^^ Oh, I won't be mixing Pentiums or Athlon64's right away; I was planning on building all of the render nodes with identical parts (makes things much simpler)... I was just wondering whether mixing CPUs from the same family would be an issue down the road.

[Beamtracer]

It would be interesting to know whether Microsoft's new 64-bit Windows makes any difference in this regard.

My understanding is that the Athlon64 processor, when running in 32-bit mode, is based on Intel's x86/IA-32 architecture. When you load up the new 64-bit Windows (available in Beta form), the processor will switch to 64-bit mode, using AMD's own "AMD64" architecture (previously known as x86-64).

It also seems that Intel's upcoming 64-bit desktop processor will be based on the AMD64 architecture, which Intel has licensed. Maybe this will bring some standardization.

I don't know what the answer is, or whether there are any rendering differences between Win32 and Win64, but it was just an interesting thought.

[Beamtracer]

[Double post self deleted]

[Draven]

64-bit XP would likely only cause a difference in rendered images if that means that the software suddenly switched to 64-bit processing and addressing.

[Beamtracer]

I was wondering about a multi-machine rendering setup.

So if you have a P4 and an Athlon64 (both running Win32 in 32-bit mode), you'll get one result.

But if you run a 32-bit P4 together with an Athlon64, and this time you're using Win64 on the Athlon, then the Athlon is going to switch on all it's 64-bit extensions. Could the result be different?

Do the new 64-bit extensions change the way the data is computed?

[Draven]

they change the decimal prescision of that data so presumably they may change the results.

Of course, in LW's case I'd expect to see much better rendering results in 64-bit because the LW pipeline is already so much deeper than 32 bits...basically, LW's 96-bit FP pipleline would have to be run through the FPU in fewer chunks, this resulting in a greatly accelerated render.

I think we can already see some of this in the difference between SSE2 and non SSE2 processors...

[Beamtracer]

Originally posted by Draven
they [64-bit processors] change the decimal prescision of that data The greater decimal precision of 64-bit computing sounds like a very good thing.

[Draven]

oh, definitely.

[jin choung]

to elaborate on this:

keyframe magazine a while back had an article that compared using macs and pcs together in a renderfarm and it turned out that you get divergent results when calculating RANDOM NUMBERS....

so things like procedural textures and hypervoxels - these things need to be 'seeded' from the processor and it turns out that these things end up being calculated differently between pcs and macs.

if that's true, i really wouldn't bet the farm that amd and intel will return same results in such cases. in fact, i would bet that the same thing would happen.

but if you feel experimental, you could try cutting together renders that do not involve such random number calculations.

in such cases, i would imagine that even across mac and pc, you'd get the same results.

as for extra decimal precision, doesn't intel get 80bit computing for floating point out of their sse/sse2? so it's still 32bit computing for the most part but they kludged some extra precision from the special instructions....

jin

[Lynx3d]

AMD64 (x86-64) does NOT change the precision of floats. They were 64bit for doubles before (IA32) and are still 64bit.
Actually, floats are 80bit in x87, so when using the FPU they are converted to 80bit, and after calculation converted back to 64bit.

And AFAIK, when using SSE2 on a Pentium4, they are not kept as 80bit but packed as 2 64bit in one 128bit register, so you already get different results!

Random numbers is another point, when you use some random function of a C library it produces pseudo-random number AFAIK, and they may very well be calculated differently across platforms (PC<->Mac) and even compiler (library).

Oh and the P4 "Prescott" supports new instructions ("SSE3"), so you may see some differences between Northwood and Prescott once Newtek uses the latest Intel Compiler...

[Beamtracer]

Couldn't "random" numbers be generated by software rather than hardware? You'd think that would get around this problem of different processors causing different random numbers to be generated.

I mean, it can't be truly "random", otherwise each machine in the renderfarm would be generating different numbers, even if they were from the same processor class.

I think any rendering differences could be fixed in the software (Lightwave) and should be reported as bugs.

I've read previous posts in this forum from people who have used mixed Intel / PowerPC renderfarms with excellent results, so in most cases it probably works fine. Maybe something like Hypervoxels throws a spanner in the works.


re: 64-bit
If you've got these huge 80-bit numbers being calculated in Lightwave, you'd think a 64-bit computer could theoretically handle it better as it would not have to break it up into as many chunks to calculate it.

[Draven]

There was definitely a problem between processors in the 5.6 days. I believe it was patched in 6.5 or 6.5. The change probably involved using a standatd random numnber generation routine across platform instead of using CPU-specific routines. However, the optimizations done for7.x may have undone that. What I'd be interested in seeing is if there is any detectable difference between the renders ofthe following:

Pentium II (non-SSE)
Pentium 3 (SSE1)
Athlon (non-SSE)
Athlon XP (SSE1)
Pentium 4 (SSE1/2)
Athlon 64/FX51/Opteron (SSE1/2)


and, when/if LW uses SSE3 optimizations;
Pentium 4 E (SSE 1/2/3)

possibly also, for sake of completeness;

Mac G3
Mac G4
Mac G5

If anyone wants to contrubute test images from these systems, we'll work it out. Of course, they'd need to all be the same scene, same scene settings, and submitted as an uncompressed image format.

[Aegis]

Would the "Variations" benchmark scene be a good test for this? Several different hypervoxel shapes..?

[Draven]

that would probably be the best test available. I did a brief test back in the 6.5 days of a 5.6 scene (textures.lws) under 5.6 between my P2 and a Mac G3 to look for differences between the scenes and the difference was not very noticable to the eye, but matting the two images together in photoshop using 'difference' mode, then merging the layers and bootsting the brightness/contrast, showed that there were actually significant differences in the two images. At the time, I was writing for digital media net and was considering doing an article on cross-platform rendering. These days, I'm helping run a different site at www.postresource.com but I'm still interested in writing this kind of article. I'd probably use the same technique, thoroughly doumented, for this article.

After thinking about it for a bit, I've also begun to wonder that if we see significant differences between the listed processors, if we should go in a little further and look for differences between different revisions of the same processor (i.e. the Thoroughbred and Barton core Athlons, etc)

Like I said, a little research and we can answer our own questions.

As for scenes, i think both textures.lws and variations.lws should be tested. That way, we're looking for differences in both hypervoxels as well as normal textures.

[Aegis]

Here's 2 to kick it off then - rendered on a dual 3.06Ghz Xeon 533FSB - saved as 24-bit .PNG

Variations:
http://www.agmorgan.pwp.blueyonder.co.uk/Variations.png

Textures:
http://www.agmorgan.pwp.blueyonder.co.uk/Textures.png

[Draven]

got 'em, thanks

I'm assuming this is in 7.5c? (Note, we need all renders to be from the same version of LW.)

I have a P3, Athlon and Athlon XP here to render on...

[jin choung]

ya might have to OVERLAY the renders on top of each other to do a real check.... just for thoroughness.

jin

[Draven]

I find that layering them using 'difference' gives me the desired (difference) data.

[jin choung]

actually,

i just meant put em on top of each other with a transparency but yah, the difference mode is indeed the thing to do the job.

jin

[Draven]

right... well, to get the full difference, you don't set it to be transparent.

I really hope someone does those athlon64 renders :)

[Lynx3d]

Here are mine:

http://lynx.aspect-design.de/stuff/textures.png
http://lynx.aspect-design.de/stuff/variations.png

Rendered on a dual Athlon XP1800+ (8 Threads) LW7.5c
(no debate about Athlon MP vs. XP please, they are the same core-wise ;) )

[Draven]

ok, so far i have the P4 (Xeon) and Athlon XP(MP)

On textures, there is no visible difference. the 'difference' is all black.

On variations, there is also no visible difference.

Faaaaaaascinating.

now if i can get the Athlon64 results and the Mac results (G4 or G5)...

[Lynx3d]

I could render some with a Pentium-M (Centrino) too...
That contains a bit of technology from everything, P3 to P4 Prescott :D
D'oh...actually i can't, it doesn't have a parallel port...shoot.
Anyone wanna donate me an USB dongle?

[Aegis]

Had access to a 1.8 Ghz single CPU G5 today - Here's the "Variations" test (24-bit .PNG):

[Aegis]

And here's the "Textures" render:

[Lynx3d]

Hm ok with the G5 renders there varifiable differences, but i really don't see how they could get problematic in this scenario.

I think it's really only the "random"-part. I just rendered Radiosit_ReflectiveThings with 8 threads and then with one thread and look at this:

[Draven]

OK here's the results...

The P4 and Athlon had no detectable differences. when Difference matted together, they produce an image that is entirely black (no difference).

The P4 and G5 had 52 colors different in the Textures example. Oddly enough, all of these differences were on the upper left texture example, which is a simple brushed metal looking example (probably using fractal noise)

On the variations scene, the P4 and G5 had 12 shades of difference. The differences were random pixels scattered al over the image.

Maybe we shouls use some radiosity tests between the three to see if the SSE2 optimization of the radiosity routines produces different results between the P4 and Athlon XP.

[Beamtracer]

Are these tests scientific? Is there no chance that differences can be caused by other factors contaminating the results?

Everything should be rendered to a lossless codec, otherwise the codec itself may have an effect. The PNG image format can be lossy or lossless, depending on how it's set up.

On the Mac, images are sometimes created via Lightwave's internal output plug-ins. For some image formats, Lightwave interfaces directly with Quicktime at an OS level to produce the image. Sometimes you get a choice, which could affect the results.

Don't forget that Lightwave 3D is currently a 32-bit app on every platform. Newtek is not selling a 64-bit version (yet). So it's possible that a 64-bit Lightwave could produce a different result to the current 32-bit Lightwave, even on the same machine!!!!

[Draven]

>Are these tests scientific? Is there no chance that differences can be caused by other factors contaminating the results?

Not much chance, since we are using the same file format, same software revision, same scene...

If there were 'other factors contaminating the results' then even cross-platform render benchmarks would be unreliable.

Also, looking at the results and how the differences are in very specific areas (especially in textures.lws) points to something else other than random pollution.

>Everything should be rendered to a lossless codec, otherwise the codec itself may have an effect. The PNG image format can be lossy or lossless, depending on how it's set up.

I'm assuming they are saving directly out of Lightwave, so its a lossless png.

>On the Mac, images are sometimes created via Lightwave's internal output plug-ins. For some image formats, Lightwave interfaces directly with Quicktime at an OS level to produce the image. Sometimes you get a choice, which could affect the results.

Shouldn't make a difference with an uncompressed format, because the conversion of LW's internal floating-point values into integer colors is done within Lightwave. If that theory were correct, then we would get differences between any file formats we use to save images.

>Don't forget that Lightwave 3D is currently a 32-bit app on every platform. Newtek is not selling a 64-bit version (yet). So it's possible that a 64-bit Lightwave could produce a different result to the current 32-bit Lightwave, even on the same machine!!!!

Sure, a little different. enough to be visible to human eyes? doubtful. not unless you have a monitor that displays floating-point color. remember, Lightwave is calculating 96-bit (technically, 128-bit if you count the alpha, is my guess) precision images using 32-bit math. in theory, the math, if done using 64-bit math, should produce the same numbers.

[Scott Gammans]

Interesting results. Since I am not going to be mixing Apples and oranges (er, PCs), it looks like even if I choose Athlon64 over Pentium 4 (or vice-versa) I may still be able to keep the "loser" machine in my mini-farm after all.

[Draven]

Yes, but well... hrm... we don't have Athlon64 results yet :)

[Draven]

HOWEVER.....

there ARE differences between the P4, Athlon XP and (non-XP) Athlon in the textures scene. not detectable to the eye, vut several of the imahes come up different. 9 shades of difference, centered on the checkerboard and grid samples, but also a little 'random' noise across the entire image....

faaaaaascinating.

[Jalle]

If anyone successfully installed the dongle drivers under
win xp 64 beta, please tell me how!
(Im using a parallel port dongle.)

Thank You !

/Jalle

[Lynx3d]

Just use 32bit XP, will be a tad slower in XP 64bit edition anyway since it gets emulated by "WoW64" (see this article for examples (http://www.gamepc.com/labs/view_content.asp?id=xp641069&page=7) )

About the differences a 64bit version could make, i don't really think there will be a 64bit version of LW 7.5, and i also think LW8 will produce different results anyway (at least i hope so!)

So all this tests we are doing here may very well be outdated before Scott's renderfarm is ready :D

LWs Radiosity for example even yields different results when changing the number of threads om the same machine (as i pointed out above) and just isn't a stable solution for animations anyway unless you bake it. Some renderers (claim to) have solved this problem, perhaps Newtek will do too eventually.

[Draven]

Just as a note on this topic, Intel has announced their new line of Xeons... that are going yo be 'compatible' with the AMD 64 bit instruction set (plus a few of their own, of course...)ZDnet Intel article (http://zdnet.com.com/2100-1103_2-5160169.html?tag=zdfd.newsfeed)

[Scott Gammans]

Well the heck with it. There ain't gonna be any mixing of the CPUs in my basement cluster.

I got a friend of mine to execute the same LightWave and POV-Ray (yes, I still use POV-Ray now and then!) benchmarks I'm running on my newly-built Pentium 4 3.2 GHz render node on his freshly-constructed Athlon64 3200+ machine. Although I my memory timings are slightly more aggressive than his, my Pentium 4 easily outran the Athlon64, sometimes by more than 15%.

I freely admit that our little shootout was less than scientific and not exactly a comparison of "apples to apples", but the results were convincing enough to discourage me from even bothering to build an Athlon64 test node. So now the question becomes, what will happen a year from now if I try to mix 32-bit and 64-bit Intel processors...?!

[Aegis]

Radiosity Reflective Things from the "Benchmarks" folder - 24-bit .PNG saved straight out of LightWave - again, dual Xeon 3.06Ghz.

http://www.agmorgan.pwp.blueyonder.co.uk/Radiosity.png

To duplicate this you'll have to set "Resolution Multiplier" to 100% in Camera Properties as by default this scene renders at half size.

[Draven]

thanks, though i dread waiting for that one to render here....

[Aegis]

Heh! Shouldn't take too long - that's why I picked that scene instead of the "Tracer Radiosity" one :D

We'll have to do some similiar test with "FPrime" when Worley Labs releases it too...

[Exper]

Radiosity Reflective Things on Athlon XP 2400

http://xoomer.virgilio.it/hobosinfo/lw_forums/Radiosity.png

Bye.

[Draven]

Thanks! i can get P3 and (pre-XP) Athlon renders here...

[Exper]

Aegis... what LW version are you using?

Note: previously posted "Radiosity Reflective Things" rendering was done using LW7.5 plain (not b, nor c). :(

Bye.

[Aegis]

Sorry, should have said - all my renders (including the G5 ones) are with 7.5c

[Draven]

agh! oh well, I'm using 7.5c as well...