We are happy to announce the alpha release of Flamenco, the job render management software currently used at Blender Institute for the Gooseberry project! This is the brender project we have been discussing in our recent weeklies, now renamed Flamenco. This is an alpha release, so expect rapid changes in the codebase, as well as exciting visual improvements (we get a logo and a website too).
We have been actively developing this version of the software for a few months, trying to lay a solid foundation for the future. Although not all features have been implemented yet, this is what Flamenco is designed for:
More technically, we are using a network of web servers based on the Flask framework (written in Python) and MySQL, communicating to each other via HTTP, using REST. In order to make the server accept multiple types of jobs, we rely on a compiler/parser pattern. For every job type we intend to support, we need to write two types of scripts, which define the structure of a job. When a job is submitted to the server, it gets processed and compiled into a sequence of tasks. Tasks are atomic and contain a list of commands to be executed by the worker. The Stdout and Stderr of each worker are piped into a parser, which is used to provide feedback on the progress of the task.
As you can see in the illustration, we rely on 4 types of components:
Of these components, we would like to point out the importance of the manager. This is what allows a geographically distributed cluster. In theory, anyone could set up a manager to build a cluster and provide nodes to a Flamenco server, which will then display them to the client. A practical example is to instance a manager on an Amazon or Google HPC cluster and manage multiple nodes, keeping them inaccessible from outside the LAN.
Another important aspect of having strict server-managers communications is that any rendering service provider could implement the manager protocol and offer their computing power to the server (and show it in a dashboard), while keeping the inner workings of their service private. The server has a very simple queuing and scheduling system, and relies on smart dispatching on the manager side.
Additionally, application plugins can be developed to integrate job submissions directly from within a 3D application.
Currently the project is hosted both on GitHub and developer.blender.org (repository only). We keep track of issues on GitHub. A more detailed feature list and roadmap will be published on the flamenco.io website. Many good things to come!
Special thanks to Sergey Sharybin and Keir Mierle for their early feedback and advice, to Matthieu Simon for his contributions during his time at Qarnot, and to Gabriel Caraballo, who has spent the last weeks implementing many key features.
If you have any questions/feedback, please drop us a comment!
It seems there’s no way to use this with windows. (yet?)
Hi ROB!, theoretically it works in windows, but is not tested (nor documented) yet.
Very excited about this, we’ve been weighing up building a render farm for our small production company for a while now.
Can some of the nodes double up? Just thinking that it would be a bit of a waste of processing power if the manager and the server can’t also be used as worker nodes (possibly holding back the processing power used for renders, so that they can function correctly as managers and servers). Or maybe the manager and the server can be one node?
Keep up the good work :)
Hi Stephen! Of course! you can even run Server+Manager+Worker on the same machine if its powerful enough.
Exxxxxxcellent :)
I’ve been eagerly waiting to hear news about a render farming solution from Gooseberry! Forgive my less-than-TD knowledge, but the article kinda makes it sound like the main target of Flamenco is larger-scale studio productions. Though I do have hope based on the above comment, “Of course! you can even run Server+Manager+Worker on the same machine if its powerful enough” – Can I assume Flamenco is also just as usable for small amateur productions? Say me and 5 friends want to link our laptops up one weekend?
And when you say “if it’s powerful enough”, could a raspberry pi handle the Server + manager roles?
Hey Kent!
Since Flamenco has a very scalable architecture, the minimum amount of components you need is a bit higher than you would expect (Server + manager + dashboard and then the workers). But!
One of the goals of the project is deliver a bundled version of Server, Manager and Dashboard, that you can start at once without worrying about the internals. The only thing left to do will be to connect laptops to it (which can be done very easily).
For a super small setup, maybe a Raspberry could be enough (provided enough storage), but I think I might be a better idea to host the server on one of the render nodes.
This will come in the next weeks, as it’s not our main target at the moment.
Ah ok, good to hear – Thanks for the info, Franceso!
Hi I’m Luis from Digital Rebel Animation School and Studio. I ‘ve been waiting a good solution for film production of network rendering in Blender. Something easy as the Autodesk Backburner. is the new Flamenco easy to setup? Or I need script knowledge to use it. I know how to setup a php mysql and apache server. Thanks for the work with the soft definetly I’m going to test it
Flamenco will be easy to set up. Currently you need to know a bit of git and Python to get your way around. It’s an alpha release, so things are not yet polished for the final user.
Thanks you for the answer Francesco. I will follow the evolution of the tool.
If there’s come kind of crowdfunding please let me know. For sure I would buy the definitive tool for network render with Blender.
https://cloud.blender.org is probably the best place to go to support the development of Flamenco I’d wager
So did I get it right that it is split in a frontend (browser) and a backend (Linux machine )? That would be really great cuz then it would be rock solid reliable and usable from any client. (which would be Linux in my case too :)
Yes, that is pretty much correct.
What’s different with “bRender”?
We simply changed the name of the project. It was a bit too confusing and similar to a common family name.
Guys, this seems as an amazing work.
The concept of a “megafarm” where can be more local clusters e.t.c. is very nice.
Are you considering running a server accessible from blendercloud, where users could join with their managers nodes, and possibly get some score for provided renders to exchange the computing power?
something like sheeply, just more advanced?
We plan to make this available on the Blender Cloud, allowing users to contribute resources for themselves. Probably sharing resources across users in a way similar to sheeply will not be possible at first. Our primary focus is to allow full control over the nodes connected, in order to get the best performance, and this is not possible if you don’t “own” the node. I hope this clarifies a bit.
So your starting at v2.0 when you haven’t made a first release yet?
My first impression here is that this isn’t a blender related project. Will this be an “official Blender Institute” project or a third party project recommended by BI? Is the github or the developer.blender repo to be considered the official repo?
Is there a real need to separate this from blender development, even if it will support more than blender tasks when done?
Apart from that the project looks promising.
If you check out the repo, you can see that the software has been around for a while already. This is a Blender Institute project (you can see a link on the website footer), we are using it currently to render the film. The repo where we are actively pushing code and keeping track of issues is on GitHub just because it’s more easily accessible, but the one on developer.blender.org will be up to date as well.
This is not blender development, it is pipeline (different developers, different requirements, technologies and so on), so it makes sense to keep it a bit separate.
Thank you for your encouraging words.
Congratulations for the alpha release! I have been watching Gooseberry weeklies and observing this farming development. I’m definitely going to use it at some level.
I was wondering, if Flamenco could be extended a bit. I mean, the manager entity could also be something else, when appearing to the outside of cluster being the same. And I’m thinking of Brenda “– Blender render farm software for Amazon Web Services”.
It should be possible to integrate other render farm as a part of Flamenco, when the manager is specialiced to handle that other farming. It could work with slice strategy, where that one manager hogs a bunch of frames/tasks, and delivers them to the workers of that other cluster.
The great benefit could be the transparency; all own machines could be used to work and some other farm (like Amazon with Brenda) could be utilized too.
It all depends how well the manager is “inheritable”, because it is acts as an interface for the server.
I’d never been aware of this project before, and looking at the screenshots I was really hoping it was going to allow much more simple render managing: queuing up multiple render jobs from multiple .blend files on just one computer for batch rendering (overnight, etc) with a real GUI. After reading the article I realize that this is not what this is designed for. Any chance of something much, much more basic like this coming out of this project or in Blender at all? I realize this is a little off topic.
Do you know Render+? I think this shoud work fine for batch rendering. I havn’t really tried it yet, but it seems to do exactly what you want… http://cgcookiemarkets.com/blender/all-products/render/
Check out Afanasy, part of the CGRU package.
Open Source render management software, works quite well.
http://www.cgru.info
What operating system are you using? In linux. ( ubuntu, centOs ..etc).
how many PC need to run this flamenco , attract , blender cloud thing ?
like minimum recommend is 9 PC ?
1 server, 1 dashboard, 2 manager , 4 workers node, 1 NAS ?
I’m about using supermicro Intel Xeon D-1541 with high performance and low voltage will be better performance per watt