PDA

View Full Version : Rendering Lightwave scenes on Azure Cloud.



blackmondy
11-17-2018, 03:24 AM
Is that a possibility with the following setup ?

1. Save my Lightwave scenes using Lightwave folder structure in OneDrive.
2. Spin up a Azure VM as Network Render Controller
3. Spin up multiple VMs as render nodes.
4. Azure NRC get scene files from my OneDrive Lightwave folder.
5. Azure NRC sends jobs to render node VMs for rendering.
6. Rendered files get saved to a specified directory in OneDrive.

Or is there a more elegant solution to this ?

jwiede
11-17-2018, 12:49 PM
I'd probably look at using a common-shared Azure storage volume instead of OneDrive for perf reasons, but otherwise should be a workable approach.

blackmondy
11-17-2018, 03:50 PM
Thanks for the reply.

jwiede
11-17-2018, 04:17 PM
Thanks for the reply.

If you have more specific questions, I can probably help more (I'm a bit familiar with Azure infrastructure), feel free to ask.

blackmondy
11-19-2018, 11:53 AM
If you have more specific questions, I can probably help more (I'm a bit familiar with Azure infrastructure), feel free to ask.

Really appreciate that. Will definitely have more questions coming up. :)

Now weighing the costs between a local physical renderfarm and a cloud-based one.

jwiede
11-19-2018, 12:08 PM
Really appreciate that. Will definitely have more questions coming up. :)

Now weighing the costs between a local physical renderfarm and a cloud-based one.

Glad to help!

The main benefits of the cloud-based approach stem from ability to change "scale" of farm quickly, though, which isn't necessarily well-shown in raw cost comparisons. If you want to be thorough, you should also explore net-based rental farms (GarageFarm.net and similar). Net-based rental farms offer similar benefits to using cloud-based instances, but with the advantage of having render-oriented infrastructure, plugins, etc. already done (rather than requiring you to set it all up yourself).

blackmondy
11-20-2018, 03:55 AM
Glad to help!

The main benefits of the cloud-based approach stem from ability to change "scale" of farm quickly, though, which isn't necessarily well-shown in raw cost comparisons. If you want to be thorough, you should also explore net-based rental farms (GarageFarm.net and similar). Net-based rental farms offer similar benefits to using cloud-based instances, but with the advantage of having render-oriented infrastructure, plugins, etc. already done (rather than requiring you to set it all up yourself).

Noted. I do understand the benefits of a net-based renderfarm. However, I want to use this opportunity to educate myself on cloud rendering and see how things actually work "under the hood". Eventually I will also want to have Adobe apps rendering on Azure cloud. I do have prior experiences in building local renderfarms for clients, running on both Macs and PCs. The pain-points I faced are usually related to pathing, plugins and network congestion, which in turn lead to timeout problems.

Currently my concerns for cloud rendering are :

- How do I gauge how many VMs I will need for a particular project ?
- Can a VM render-node spin up automatically when a scene gets heavy midway ?
- Do the VM NRC and render nodes shut down automatically upon completion ?
- Do I get email notification upon completion ?

In summary, how do I control costs when cloud rendering with regards to compute time and file movement ?

jwiede
11-20-2018, 08:24 PM
This will be a long one, for folks uninterested in details... TL; DR


Currently my concerns for cloud rendering are :

I'll deal with em one at a time, but understand some of these don't really have great answers, because LW simply doesn't provide good diagnostics and instrumentation about how "difficult"/"heavy" a scene will be to render, in a form that can be easily accessed. That's something that desperately needs to be fixed by Newtek, but part of a very long list of such issues as well. Until there's better resolution, some of what you want requires unavoidable approximation/guessing on your part.


- How do I gauge how many VMs I will need for a particular project ?
Lacking decent instrumentation or ability to access preview statistics programmatically, there's really not much to do here besides basic guessing. Unfortunate, but LW just doesn't provide enough information to properly estimate scene rendering times/costs.


- Can a VM render-node spin up automatically when a scene gets heavy midway ?
You can code up a timer-based external script which sits in a render node and if it's taking too long (timer expires before scene has completed rendering), take whatever action you feel is needed. Unfortunately, LW doesn't provide a good means of dynamically "splitting" the rendering of a given scene across multiple nodes once dispatched -- or, frankly, for splitting frames dynamically at all. You can respond to the results of prior rendered frames' performance for handling future frames, but you'd need to take on the infrastructure responsibility of that by yourself, LW won't be helpful for that.


- Do the VM NRC and render nodes shut down automatically upon completion ?
The answer depends on whether you're using another render controller/dispatcher, or producing your own. If you're essentially coding up your own render controller, then you would be responsible for detecting when the queue is empty and shutting down idle nodes, using scripting or otherwise. Even using LW's NRC, the nodes are presumed to remain operational while potentially working, so while NRC may provide for remote shutdown (I know BNR does, unsure about NRC), it would require manual or scripted triggering by your infrastructure (wrapped around the other infrastructure, specifically for controlling interactions with cloud systems).


- Do I get email notification upon completion ?
Again, the answer depends entirely on whether you're coding up your own render controller/dispatcher with protocol, planning to use NRC, or BNR or whatever. If doing your own, obviously you're responsible for implementing such functionality. I believe BNR can send email when jobs complete (and indicate success/failure), but check with them for details. I don't really have much experience with NRC (I only had problems when I attempted to use it, so gave up), so can't answer what it can/cannot do in this regard, because I never got it to that point of operation.


In summary, how do I control costs when cloud rendering with regards to compute time and file movement ?
Careful management of active nodes, storage resources, etc. is the best answer I can give you, a lot depends on whether you're doing your own render controller/dispatch system or using someone else's and just "wrapping around it". Either way, there will be significant implementation involved, with the DIY route requiring a very serious amount of implementation.

Hope that helps!

blackmondy
11-21-2018, 08:22 AM
Hey Jwiede,

Really appreciate the detailed reply. This will definitely help me at lot. Incidentally, my renderfarm experience was with Butterfly Network Render. It's arguably the easiest LW render controller I came across.

jwiede
11-21-2018, 10:27 AM
Hey Jwiede,

Really appreciate the detailed reply. This will definitely help me at lot. Incidentally, my renderfarm experience was with Butterfly Network Render. It's arguably the easiest LW render controller I came across.

Agreed 100%. If you're planning on using BNR (7?) with the render nodes, I can give some more precise answers to a couple of the above questions. Will reply again in a separate post.