We are happy to announce the alpha release of Flamenco, the job render management software currently used at Blender Institute for the Gooseberry project! This is the brender project we have been discussing in our recent weeklies, now renamed Flamenco. This is an alpha release, so expect rapid changes in the codebase, as well as exciting visual improvements (we get a logo and a website too).
We have been actively developing this version of the software for a few months, trying to lay a solid foundation for the future. Although not all features have been implemented yet, this is what Flamenco is designed for:
More technically, we are using a network of web servers based on the Flask framework (written in Python) and MySQL, communicating to each other via HTTP, using REST. In order to make the server accept multiple types of jobs, we rely on a compiler/parser pattern. For every job type we intend to support, we need to write two types of scripts, which define the structure of a job. When a job is submitted to the server, it gets processed and compiled into a sequence of tasks. Tasks are atomic and contain a list of commands to be executed by the worker. The Stdout and Stderr of each worker are piped into a parser, which is used to provide feedback on the progress of the task.
As you can see in the illustration, we rely on 4 types of components:
Of these components, we would like to point out the importance of the manager. This is what allows a geographically distributed cluster. In theory, anyone could set up a manager to build a cluster and provide nodes to a Flamenco server, which will then display them to the client. A practical example is to instance a manager on an Amazon or Google HPC cluster and manage multiple nodes, keeping them inaccessible from outside the LAN.
Another important aspect of having strict server-managers communications is that any rendering service provider could implement the manager protocol and offer their computing power to the server (and show it in a dashboard), while keeping the inner workings of their service private. The server has a very simple queuing and scheduling system, and relies on smart dispatching on the manager side.
Additionally, application plugins can be developed to integrate job submissions directly from within a 3D application.
Currently the project is hosted both on GitHub and developer.blender.org (repository only). We keep track of issues on GitHub. A more detailed feature list and roadmap will be published on the flamenco.io website. Many good things to come!
Special thanks to Sergey Sharybin and Keir Mierle for their early feedback and advice, to Matthieu Simon for his contributions during his time at Qarnot, and to Gabriel Caraballo, who has spent the last weeks implementing many key features.
If you have any questions/feedback, please drop us a comment!