Starting your first project and figuring out how to successfully deliver it over the next few months can be daunting. A clear workflow is essential for seamless collaboration and communication between departments. The term "pipeline" is often used in this context, but it can sound technical and complex. This article simplifies how to build a pipeline, with or without coding.
What is a Pipeline?
Pipelines are essential to a production, connecting departments to work as a unit. Many artists don't understand how pipelines work or how to build them. Let's change that!
The term "pipeline" combines "pipe" and "line" to describe a process where something goes in one end and comes out the other process. It's commonly used in visual effects, animation, and games, but "workflow" is a good synonym as well. Pipelines can range from a 3D software rendering pipeline to a production pipeline that manages the workflow and data output at each stage.
Every skilled artist faces a new challenge in a project: Teamwork. Specialization requires collaboration with other specialists, which poses challenges of communication and focus. Knowing our work well doesn't mean we can easily apply those skills in a team environment. Creating workflows and standards unifies efforts, allowing artists to work freely within project expectations while delivering the right package to the next department. A pipeline reduces friction and lets artists focus on their work while ensuring smooth handoffs to other departments.
Essentially, a pipeline defines the input and output of each department, focusing on a common result. In the day-to-day of a production, there are many moments of uncreative work. Things that need to be done, sometimes without us even realizing it. Over the years, we have become accustomed to the way we do certain tasks: Be it modeling, rigging, animation, lighting, fx, or compositing. We open certain menus, navigate to certain folders, run command chains, or wait for processes to complete. These workflows are important to us, they give us security in our daily work or in the realization of our creative ideas.
To build our own pipeline, we need to look at the four elements it's built on:
Folder Structure
Every pipeline starts with the folder structure, because files are the essential glue between departments and the representations of each department's work. The first questions we have when working with our project files are
Where do I store my files?
The folder structure defines the folder names, their hierarchy, and what goes in each folder. Depending on your project and studio, this may change to suit your needs.
task-based: The task-based approach works best for specialized teams where tasks are strictly divided into departments so that artists have a simple "everything in one place" structure. Assets and shots are organized under tasks. (ASSETS/RIG/Shots)
asset-based (most common): The asset-based approach allows for an easier switching when each artist is working on multiple tasks at once. A great generalist approach for smaller projects. (ASSETS/Mike/RIG)
Additional planning folders give space for production data such as asset/shot spreadsheets and budgets, while Pre-Production stores artwork and mood boards along with footage, where we find general project-independent resources to help with the asset and shot production.
A project folder structure can be broken down into the following categories:
Naming Convention
Now that we've organized our files into different project folders, we need to answer another question:
How do we name the texture files, each task, an asset, a shot, or a node in the scene?
These questions are answered by the naming convention, which defines general and specific rules for certain types and stages of the project. These rules must answer all naming questions to eliminate any confusion and to help in the later stages when we're using scripting to handle our nodes and files.
General naming conventions might be
English
camelCase (ragingBull)
no space "" instead underline "_"
tasks are in UPPERCASE (RIG)
versions are a "v" with 3 digits (v001)
no names and comments in the file name (will be saved as metadata)
Tasks can be shortened for the use in asset and shot files:
short | long |
GEO | Modeling |
SHD | Shading |
TEX | Texturing |
RIG | Rigging |
ANIM | Animating |
FX | Effects |
LGT | Lighting |
RENDER | Rendering |
- | - |
PRE | PREVIZ |
SLAP | SlapComp |
COMP | Compositing |
- | - |
EDIT | Editing |
GRAD | Grading |
Here is the final result for our project files using the naming conventions:
Work files: name_TASK_version.extension
mother_RIG_v001.mb
Render file: shotNr_AOVs_version.####.extension
s010_diffuse_v001.1001.exr
Software Pipeline
Now that we've structured our project and naming conventions inside and outside the scene, it's time to look at the software and plugins we use. The software pipeline defines the software, its versions, the data files, and how they connect to each department. A simple way to summarize its purpose is to define what goes in, what goes out, and what software is used in the middle of the process. The result is that every artist uses the same tools and plug-ins, while data flows clearly to and from each department.
TASK | INPUT | OUTPUT |
GEO | .abc | .mb |
TEX | .abc | .tiff (8 bit) / .exr (16 bit) |
SHD | .mb | .mb |
RIG | .mb | .mb |
ANIM | .mb | .mb / .abc |
LGT | .mb | .mb |
RENDER | .mb | .exr (16 bit) |
SOFTWARE
software version (Maya 2025, Houdini 20)
file format per task (in & out)
EXTRAS
Start Frame : 1001 (prerun time for simulation)
Frame Handles : 5 (pre- and postrun time for the edit)
Script Pipeline
We have reached the final stage of our pipeline: The script pipeline. This is what most people think a pipeline is. In fact, it's not. The previous sections are far more important and essential to creating a pipeline than the scripts that support it. The purpose of the script pipeline is really just to automate what has been defined by the folder structure, naming convention, and software pipeline.
The script pipeline can help our pipeline to
Folder Structure: Create folders and organize project files accordingly.
Naming Conventions: Save, load, import, and export files using a consistent naming convention inside and outside the scene.
Software Pipeline: Opens the appropriate project software and loads the necessary plug-ins, scripts, and files through import and publish applications.
The script pipeline automates the previous steps to make them smooth and frictionless. Even the best folder structure, naming convention, and software pipeline can be ignored by the individual artist. Automating these processes with scripting eliminates manual errors, additional training, and saves time, allowing the artist to focus on their craft instead of data handling.
The key to a successful script pipeline is to automate these 3 elements:
Software setup at startup
Save & Load
Export & Import
Launching the right software with the necessary plug-ins, settings, and scripts ensures that everyone on the team is working with the same tools. Save, load, export, and import enforce the pipeline structure while simplifying the process of working and publishing to other departments.
“Why spend 1 minute to do a task manually when you can spend a day automating it.” - Dilbert
On the other hand, we tend to see certain automations as the only way to get things done, not realizing that we can achieve the same results with little preparation and a simple manual workflow. Studios are mostly plagued by the same problems, but they remedy themselves with pipeline systems that automate as much of the "idle time" as possible to make artists more productive. The best time to create a script pipeline is after rigorous testing of the previous 3 stages to determine which part needs and benefits from automation, as automation is time consuming and can bring its own challenges. (How long can you work on making a routine task more efficient before you’re spending more time than you save?)
Here are a few other free and paid pipelines to help your project with its workflow:
Autodesk Flow Production Tracking (previously ShotGrid)
Extras
We have a few points that are important when looking into the pipeline of a project:
Onboarding: Introducing new people to a project should always be a priority in the beginning. Besides giving them time to adjust, it's important to show them how the pipeline functions specifically for their work, while allowing them to read about it in the pipeline documentation. (see Open Source Pipeline Plex Wiki)
Backup: It's always wise to keep multiple backups of our project in case we lose it or need to revert to a particular file or state. Every operating system offers file backups with timestamps, or simply uses Google Drive or Dropbox to store a copy in a remote location. For the script pipeline, the best backup system is git, as it allows a tremendous amount of control over backups, changes, and rollbacks.
Simple automation: A variety of free and paid external tools can handle certain tasks creating a simple script pipeline. Example: Tools like Total Commander or Adobe Bridge allow you to rename tons of files according to a set logic.
Resume
Pipelines are a key element of any visual effects, animation or game production. A good pipeline guides the artist through the complexities of a multi-step project and can mean the difference between meeting or missing a deadline. A bad one adds complexity, confusion, and unnecessary steps that stifle creativity, teamwork, and progress.
To build a great pipeline, it's important to have clarity about the project and enough experience from previous projects to plan a workflow in advance that helps guide everyone from script to final cut. But it's not all about automation and scripts, as a smart and clear workflow can achieve similar results without over-reliance on programming. In the end it’s all about arriving at the result as frictionless as possible.
To learn more about pipelines check out my Open Source Pipeline Plex.
Comments