Pipeline,Tools

Announcing Flamenco 2.0 alpha release

25 February, 2015 |  25 Comments | by Francesco Siddi

We are happy to announce the alpha release of Flamenco, the job render management software currently used at Blender Institute for the Gooseberry project! This is the brender project we have been discussing in our recent weeklies, now renamed Flamenco. This is an alpha release, so expect rapid changes in the codebase, as well as exciting visual improvements (we get a logo and a website too).

flamenco_screenshot_02

We have been actively developing this version of the software for a few months, trying to lay a solid foundation for the future. Although not all features have been implemented yet, this is what Flamenco is designed for:

  • image sequence rendering (both linear and sparse)
  • tiled rendering (supporting the compositor)
  • simulations
  • node-based jobs
  • job distribution on multiple clusters of workers, geographically separated (via managers)

More technically, we are using a network of web servers based on the Flask framework (written in Python) and MySQL, communicating to each other via HTTP, using REST. In order to make the server accept multiple types of jobs, we rely on a compiler/parser pattern. For every job type we intend to support, we need to write two types of scripts, which define the structure of a job. When a job is submitted to the server, it gets processed and compiled into a sequence of tasks. Tasks are atomic and contain a list of commands to be executed by the worker. The Stdout and Stderr of each worker are piped into a parser, which is used to provide feedback on the progress of the task.

architecture_diagram

As you can see in the illustration, we rely on 4 types of components:

  • server (only one per network)
  • manager (one or more per network)
  • worker (one or more per manager)
  • dashboard (one or more, web-based, or OS native)

Of these components, we would like to point out the importance of the manager. This is what allows a geographically distributed cluster. In theory, anyone could set up a manager to build a cluster and provide nodes to a Flamenco server, which will then display them to the client. A practical example is to instance a manager on an Amazon or Google HPC cluster and manage multiple nodes, keeping them inaccessible from outside the LAN.

Another important aspect of having strict server-managers communications is that any rendering service provider could implement the manager protocol and offer their computing power to the server (and show it in a dashboard), while keeping the inner workings of their service private. The server has a very simple queuing and scheduling system, and relies on smart dispatching on the manager side.

Additionally, application plugins can be developed to integrate job submissions directly from within a 3D application.

Currently the project is hosted both on GitHub and developer.blender.org (repository only). We keep track of issues on GitHub. A more detailed feature list and roadmap will be published on the flamenco.io website. Many good things to come!

Special thanks to Sergey Sharybin and Keir Mierle for their early feedback and advice, to Matthieu Simon for his contributions during his time at Qarnot, and to Gabriel Caraballo, who has spent the last weeks implementing many key features.

If you have any questions/feedback, please drop us a comment!


25 Responses

  1. Rob says:

    It seems there’s no way to use this with windows. (yet?)

  2. Very excited about this, we’ve been weighing up building a render farm for our small production company for a while now.

    Can some of the nodes double up? Just thinking that it would be a bit of a waste of processing power if the manager and the server can’t also be used as worker nodes (possibly holding back the processing power used for renders, so that they can function correctly as managers and servers). Or maybe the manager and the server can be one node?

    Keep up the good work :)

  3. I’ve been eagerly waiting to hear news about a render farming solution from Gooseberry! Forgive my less-than-TD knowledge, but the article kinda makes it sound like the main target of Flamenco is larger-scale studio productions. Though I do have hope based on the above comment, “Of course! you can even run Server+Manager+Worker on the same machine if its powerful enough” – Can I assume Flamenco is also just as usable for small amateur productions? Say me and 5 friends want to link our laptops up one weekend?

    And when you say “if it’s powerful enough”, could a raspberry pi handle the Server + manager roles?

    • Hey Kent!
      Since Flamenco has a very scalable architecture, the minimum amount of components you need is a bit higher than you would expect (Server + manager + dashboard and then the workers). But!
      One of the goals of the project is deliver a bundled version of Server, Manager and Dashboard, that you can start at once without worrying about the internals. The only thing left to do will be to connect laptops to it (which can be done very easily).

      For a super small setup, maybe a Raspberry could be enough (provided enough storage), but I think I might be a better idea to host the server on one of the render nodes.

      This will come in the next weeks, as it’s not our main target at the moment.

  4. Luis Arizaga says:

    Hi I’m Luis from Digital Rebel Animation School and Studio. I ‘ve been waiting a good solution for film production of network rendering in Blender. Something easy as the Autodesk Backburner. is the new Flamenco easy to setup? Or I need script knowledge to use it. I know how to setup a php mysql and apache server. Thanks for the work with the soft definetly I’m going to test it

  5. peterling says:

    So did I get it right that it is split in a frontend (browser) and a backend (Linux machine )? That would be really great cuz then it would be rock solid reliable and usable from any client. (which would be Linux in my case too :)

  6. Min-Su Kim says:

    What’s different with “bRender”?

  7. Vilem Novak says:

    Guys, this seems as an amazing work.
    The concept of a “megafarm” where can be more local clusters e.t.c. is very nice.
    Are you considering running a server accessible from blendercloud, where users could join with their managers nodes, and possibly get some score for provided renders to exchange the computing power?
    something like sheeply, just more advanced?

    • We plan to make this available on the Blender Cloud, allowing users to contribute resources for themselves. Probably sharing resources across users in a way similar to sheeply will not be possible at first. Our primary focus is to allow full control over the nodes connected, in order to get the best performance, and this is not possible if you don’t “own” the node. I hope this clarifies a bit.

  8. Shane says:

    So your starting at v2.0 when you haven’t made a first release yet?

    My first impression here is that this isn’t a blender related project. Will this be an “official Blender Institute” project or a third party project recommended by BI? Is the github or the developer.blender repo to be considered the official repo?

    Is there a real need to separate this from blender development, even if it will support more than blender tasks when done?

    Apart from that the project looks promising.

    • If you check out the repo, you can see that the software has been around for a while already. This is a Blender Institute project (you can see a link on the website footer), we are using it currently to render the film. The repo where we are actively pushing code and keeping track of issues is on GitHub just because it’s more easily accessible, but the one on developer.blender.org will be up to date as well.

      This is not blender development, it is pipeline (different developers, different requirements, technologies and so on), so it makes sense to keep it a bit separate.

      Thank you for your encouraging words.

  9. Ville Lukka says:

    Congratulations for the alpha release! I have been watching Gooseberry weeklies and observing this farming development. I’m definitely going to use it at some level.

    I was wondering, if Flamenco could be extended a bit. I mean, the manager entity could also be something else, when appearing to the outside of cluster being the same. And I’m thinking of Brenda “– Blender render farm software for Amazon Web Services”.

    It should be possible to integrate other render farm as a part of Flamenco, when the manager is specialiced to handle that other farming. It could work with slice strategy, where that one manager hogs a bunch of frames/tasks, and delivers them to the workers of that other cluster.

    The great benefit could be the transparency; all own machines could be used to work and some other farm (like Amazon with Brenda) could be utilized too.

    It all depends how well the manager is “inheritable”, because it is acts as an interface for the server.

  10. Kevin says:

    I’d never been aware of this project before, and looking at the screenshots I was really hoping it was going to allow much more simple render managing: queuing up multiple render jobs from multiple .blend files on just one computer for batch rendering (overnight, etc) with a real GUI. After reading the article I realize that this is not what this is designed for. Any chance of something much, much more basic like this coming out of this project or in Blender at all? I realize this is a little off topic.

  11. Aravindkumar says:

    What operating system are you using? In linux. ( ubuntu, centOs ..etc).

Leave a Reply

Your email address will not be published. Required fields are marked *