This is the first progress report on the asset manager project (originally known as Redcurrant). Going forward, though, the plan is to report all updates on a weekly basis.
After Bart’s initial investigation at the beginning of the year, we spent the following months looking further into existing pipeline solutions and designs, still unsure whether to pursue our own project or adopt an existing one. This summer, in particular during SIGGRAPH 2014 in Vancouver, we had the chance to meet with a few pipeline architects, and also attend some relevant sessions on topics such as asset management and render management. We also got some demos from the TACTIC team as well as Damas software, Shotgun, and Thinkbox – with some of them extremely keen to support our project. But after these meetings and demos it was pretty clear that we had to develop our own solution. The main rationale behind this is that we need:
Relating to the last point, our pipeline is a little peculiar compared to most film-production scenarios. In general, pipelines and pipeline tools focus on data flow across departments as well as consistent data representation (depending on the context in which the data is required). In this production, we will mostly use the .blend file format (possibly with some Alembic for caching), so some of the main pipeline issues can be addressed later, although we are keeping them in the front of our minds as we design.
Building a feature-film-ready pipeline is a massive task, which we will not attempt to solve in one go. Based on past film experience and early feedback, we believe the best solution is to subdivide the pipeline into 3 main components:
We could write several paragraphs about each one of the components (and we will, in future posts), but for the moment let’s just focus on asset management. This is the component we need most urgently. Also, starting here gives us several benefits:
The simplified design for our asset manager (BAM) looks like this:
The idea is to allow selective “checkout” of a large project structure, including time and bandwidth and efficiency. At the same time, we want to maintain compatibility with the existing SVN-based storage and versioning system (used in all our previous open movie projects). It’s reasonable to expect that SVN will not scale well to the feature film level, but for the moment it’s a valuable safety net for the pilot production. (More detailed documentation will be available here.)
Currently the core team for this project is composed of Campbell Barton and yours truly (Francesco Siddi). We’ve received a lot of feedback and offers to help, which have been extremely valuable. At the moment there are no plans to expand the team inside the studio, but external contribution to the project will be seriously evaluated and possibly integrated. In order to make the project successful we need a number of experienced and motivated stakeholders, willing and interested in adopting the project in their own pipelines.
We have already spent several weeks collecting feedback, investigating, designing, and discussing the big picture. We have also started work on a proof-of-concept implementation, which is working well and could be the foundation for future work on the asset management module.
During the past week we have wrapped up 80% of the prototype functionality, and we are confident that during this week we will be able to deliver a basic command-line utility, allowing communication with a remote server via HTTP, abstracting SVN checkouts and commit operations. During the week we plan structure this and formalize the timeline even further.
If you’re interested to see the code and checkout the git repository, check the asset manager prototype here. (Keep in mind that this is a work in progress, not a production-ready tool.) More planning and design docs are available on wiki.blender.org.
We are still in the early development stage, but one of our top priorities (as highlighted in the time-line) is to have a working tool for the current production, allowing artists inside and outside the studio to work together. That being said, we are always open to feedback, suggestions, and the fixing of any issues you might encounter while working with the pipeline tools.
Let us know in the comments your early thoughts.
Wow, it’s very interesting ! We will look at that in detail soon.
Thanks gooseberry !
Sounds nice.
Becarefull with the “Something that will fit into the current Blender workflow” statement. AM system is larger than just what you do or can do with Blender. Many stakeholders who are not ONLY working with Blender could be interested about this.
I have been working on pipeline management system for the past 4 years with in-house system, shotgun, tactic, … I would be glad to contribute.
cheers,
F.
We are keeping that in mind! Thanks :)
Finally it’s here. Im also responsible for this task in our studio. Hopefully I can contribute well for this stuff.
Cheers
Hi! I’ve some experience with TACTIC, SVN and recently testing Shotgun a little bit.
What I like from each one?
– TACTIC: Customizable (a bit too much)
– SVN: Powerfull, on movie production scales well as Server, but not as Client
– Shotgun: Beautifull, pre configured, integration with software (but is hard to add a new software/engine)
Please keep in mind easy integration with other software. Add Blender integration as an addon to BAM (Shotgun was very dissapointing on this)
SVN clients are too much complicated, working copy is big, a nightmare for artists, is a great idea add a Layer on top of it.
When I see it on the weekly video I think “Man! I wish I had it 12 monts ago!”.
Congratulations!!
I’ll be glad to help too!
So awesome! There is not much open pipeline stuff out there and I have always been interested in this sort of thing. I understand the goal is to Blender centric but this project has the ability to open Blender up to talk to other packages more efficiently as well. Hope I can contribute in some way.
Hey, this is looking really awesome! Im curious though as to how you specifically plan on handling blend files as well as other files that they may depend on (textures for example). Firstly, when checking out a .blend file that has external dependencies on other files, will it also check them out too, or will it require that the user embeds all the data for the file?
Secondly, will BAM be able to handle exported model files like FBX files? I’m currently interested in this from a game development perspective, and was wondering how well BAM could be used to manage art assets, including final exports to be used in a game engine.
Lastly, what kind of user interface adjustments do you see being made for Blender to integrate the asset system? Will it integrate with or change the way data is handled inside Blender? :D
Hello there!
BAM is designed to help blender managing external dependencies (linked libraries or assets). It can be expanded in the future to support more software as well.
The export model files like FBX fits the “automated tasks” part, which allows the asset manager to create multiple procedurale representation of an asset. It is foreseen, but not implemented yet.
Regarding Blender UI integration, we are starting to brainstorm and design ideas for it. More infos about this coming soon!
Thanks for the reply. I figured BAM would be used for Blender initially, I was just curious how it handles assets and dependancies :D
The automated tasks part sounds extremely interesting, it sounds like a pretty nice way of streamlining exports, although I presume it’s use in game-related applications is currently outside the scope of BAM, as it would need to account for low-poly/high-poly asset versions and baking to be useful in a development pipeline. Sounds like the potential is there for scripting or adding it in the future though, which is awesome, and it’s a pretty cool feature in itself.
I look forward to seeing this develop!
hey guys did you already know about
eyeon Production Pipeline (EPP)
https://github.com/eyeonSoftware/epp
http://forums.cgsociety.org/showthread.php?t=1117468
I’ve had a look. It does not look like a very active project unfortunately…
Hi all
We’re the developers of SaAM (Shot and Assets Manager), an online CG pipeline tool to manage and monitor broadcast/movie projects and facilitate collaboration between artists who works remotely.
We’re interested in BAM and we’d be very happy to share the knowledge we gathered on this project so it benefits BAM.
Our early thoughts were about SVN and versioning. We think that it’d be too restrictive to force the use of SVN (or any other versioning tool) because each studio has its own proper solutions, sometimes developed by themselves…
The trick we used for SaAM is that we simply don’t care about data itself, but we use a kind of “metadata”. It works as follows:
_ Local script (part of asset manager as an addon, or standalone CLI) that checks, in a directory, every asset file and its dependencies, and create an XML descriptor file, called “masterfile_assets.xml”.
_ SaAM parse this XML and create / update assets rows in server database. It’s also possible to create assets directly from database; SaAM will reconstruct the XML masterfile, and offers to download it.
_ For every artists to make sure they have the last updated XML file, it can be treated like any asset, in the assets folder, and handled by the chosen versioning system.
This way, we can discuss an asset on SaAM regardless of the version. But it’s also possible to create a talk for a specific version !
There is also a check of who’s working on the asset’s file (we call it “handling”). If someone handle the asset, it’s file can’t be overwritten by other artists, until it’s “released”.
An other thought is about Scenes; it can be treated like assets in most cases, the only specificity is that they’re related to shots and sequences.
We have a lot of ideas about managing scenes and the way they interact with assets, cameras, and shots…
We keep a close eye on your work, and hope this information will be helpful.
Cheers,
.Polo & Charles
Thanks for posting your feedback here as well. Locking assets is definitely and important topic, as well as having an abstraction layer between the asset manager and the storage/versioning system. We are taking that into account :)
Hi guys!
Awesome project!
I’m starting an educative short film (CC Zero) completely based on Blender . At the beginning I started using Tactic but as soon as I knew about your project I stopped.
Since my project has no delivery date and it’s not money related I can take as much time as I need so I would like to be one of your “Guinea Pigs ” ;P if you need any.
Also I have experience pipeline and all that stuff.
So if I can be of any help, just tell me ;)
Vicente
I have a animation script called dragonstone I ‘d like to submit that might be of use to everyone how do I do that .it has good revue from steven king.
The Project Management end of this project is very exciting!
Would help with juggling multiple projects and people working on them,
Hey.
What server do you recommend for SVN?
Work soon to start on a short film for which we want to work at 2K.
Thanks
B.