Building a Mini Render Farm

[nyvz]

My point is, are those 3 8core rack mount machines plus racks plus external cooling plus rack mount management infrastructure plus the gpu limitation really a cost savings over buying one 24core workstation that has much more gpu expandability and the flexibility to be a backup workstation? Not to mention simplifying the network bottleneck and render management problems...

Not to mention the problem of housing rack mount machines that due to their form factor tend to have small inefficient fans that run at very high speed and produce a lot of noise. You tend to want to put them in an environment that isolates their high noise level but then that usually means a small enclosed environment that itself needs cooling management to keep the nodes from overheating...

Basically for a small facility that doesn't have a machine room, rack mount render nodes might not be a great match.

Also be careful with older pc servers when trying to get the best cpu performance, what are the exact specs of the servers you may buy? If they are older architecture or low clock speed you may not be getting the performance you expect.

[Greg_E]

For the AE stuff, I'm thinking it might be faster and more effective to throw a bunch of GPU's and as much RAM as you can afford into the workstation. The render to ram makes things extremely fast, and the additional GPU's should get things done in a hurry. I did a quick search and it looks like you would need a license for AE on each "render node" to make it work because you need to open the project on each computer and tell it to skip finished frames and each one to render. Going to be some contention happening where duplicate frame could get rendered and one of them thrown out. But this was just a quick web search.

[edit] Strike that, what you need seems to be here:
https://helpx.adobe.com/after-effect...rendering.html
And it says that in CS6 and later the render engine does not need to be licensed. Then it's just a matter of determining whether it uses CPU, GPU, or both to render the files out. Nice that it can work from a watch folder, so you could set up the process and just let it sit and wait for a job to get dropped into the folder, then harvest the result later when it is finished. I'd try it on a couple of computers with no special GPU and see what happens. Biggest thing you'd need is to build a NAS storage machine, or buy one (ixsystems makes some nice NAS machines as well as free software to build your own called FreeNAS).

[Greg_E]

Also this http://www.chocofur.com/cycles-in-production.html for your other applications (maybe). The Cycles render engine seems to have come a bit farther in that I didn't know it could handle other types of files than what Blender would send.

[nyvz]

It's definitely worth considering how your approach might affect your network. Are you running all your projects and assets off a centralized server? Is your network all gigE? How big are your projects? You might need an upgrade if your network since heavy render node traffic might hinder your workstation performance.

[Greg_E]

With proper switches the workstations won't be affected much at all. Modern business class or enterprise class switches are about as good as dedicated lines. The only place that might slow down is network attached storage and it would be cheap enough to set up a second render only storage system.

[nyvz]

Yes but the need for a separate render only storage with constantly updated duplicates of all your assets and projects may not be so trivial in cost or complexity. And upgrading the current storage server's throughput and i/o performance may not be cheap either. Of course this depends a bit on the existing storage system.

[LeavingTheCandy]

Long time Mac user here, recently added a 36-core Windoze machine loaded to the hilt with memory and GPU (Tesla's) and the thing makes my maxed-out 10-core nMP look lame. (yes, 10 core...custom installed CPU...3.0 GHz) I still like OS X interface and operability better, but in rendering and editing situations the Windows machine just kills the Mac! I use it for DaVinci Resolve mostly, but when I have a big render, just punt it to that box from the Mac and watch it eat bytes like a Japanese movie monster.

As another user mentioned, speed between machines (networking) is a big consideration. We have a very efficient 10G network, but Infiniband 40G is on the upgrade list since we're currently working with Red 6K material and 8K coming soon.

Be prepared for some pain going from mac to Windows, but the speed is a big factor on intensive renders.

[Greg_E]

You can go to 40gb ethernet, the hardware is out there now just that Infiniband is more widely known right now.

If you have good hardware you should also be able to aggregate a bunch of 10gb connections into much faster. But disk speed comes into play somewhere in there too, a simple 4 disk NAS or SAN just wont keep up with those speeds.