What Exactly to TOPs Do?

​

When I first heard about TOPs, I wrongfully assumed that they had something to do with materials, or they were some advanced TD level thing that I wouldn't learn until I was handed that crown.

​

Well they aren't that scary or that complex. TOPs are relatively new to Houdini. They were implemented in version 17.5, and they are known as Task Operators. 

​

Task operators use something called a Procedural Dependency Graph(PDG) to help automate tasks. TOPs Automate and scale pipeline tasks. So if you are a Houdini pipeline developer, or studio TD, these are probably something that are worth knowing.

​

TOPs also allow for you to set up nested wedging, Automation for Houdini and Maya, Scripting and managing of scheduled work items. Plus a lot more. TOPs operate with something called work items. These are generated from the processing units from each TOP node. These work items are passed downstream through each node till each item/job is completed.

​

You can find the TOPs network under the network menu drop down tab.

​

Static and Dynamic Items

​

When working with TOPs, you may notice that it works with two different types of work items. These items are either static or dynamic.

​

Static Work Items

​

These items describe the information that is needed for TOPs to start running. Static items deal with the input information going into TOPs. Naturally, the TOPs system will take these static items and sort them according to the number of frames to render as set in the parameters. This will help the system know the number of work items is known ahead of time. 

 

These items work well in Film industry based workflows where everything is frame based. This is because the number of frames to render is known ahead of time, which also the static items(or frames in this case) to be sorted preemptively. Therefore, the TOPs system can work with the static items faster.

​

Dynamic Work Items

​

These items describe the information that is needed for TOPs to start running.These items are generated while the work in TOPs are being done.  These items are mainly used in games and effect workflows, as they need to handle layered simulation effects. For example; crowd simulations. When working with, or handling crowd simulations, you have to first set up the agent and then the action that will be layered on-top of them. These simulations are harder for a TOPs system to handle, and so they are divided into dynamic items where the system can deal with the different simulation layers individually. 

​

Some rules of thumb for static and dynamic objects:

​

• Always try to chose static work items over dynamic. It is easier to count how many work items your TOPs setup will have to complete with static items. This will also help Houdini give a better sense on how long a project task will take to complete.

​

• In a games workflow you most likely will have to use dynamic items for all of your TOPs setup. Simply because game engines operate differently than a traditional film pipeline.

​

• Houdini might not be able to automatically recognize if you have added dynamic or static items into the system. In this case it will ask you to specify which one is which.

​

• Some nodes have an option called dynamic partitioning. In this case, you would turn this option on for dynamic items so that the information can be generated dynamically. Turning this option off computes the the information statically.

​

What's a PDG and How Does It Work?

​

As mentioned above the PDG is known as the Procedural Dependency Graph. This is basically a node's flags and display in the TOPs network. This allows you to see the information inside the node, and if it is passing data through correctly. This works in conjunction with the other TOPs viewers. 

​

There are three TOPs viewers you can use. The first one is the Task Graph Table. This viewer allows for the user to see how the information is passing through your TOP nodes. You can visualize the node name, name, index, state, cook time, and output size of your node.

​

The second viewer is the PDG Services Manager. This interface allows for you to see the current status of your current cooking work items. It operates as a server for submitted jobs so it can control and decide which order to complete jobs in.

​

The final one is the PDG State Viewer. This viewer allows for you to host severs to handle multiple amounts of tasks. You can chose between a Basic Houdini Binary Server, or a Data Layer Web-sockets Server.

​

Now that we understand our available PDG tools, let's get right back into how they work. As we discussed above, PDG displays the information being passed through your selected node. If there is no information being passed through the node, the node is not connected to anything, or the tasks are not active, you most likely won't see the PDG interface.

​

On an active interface you will see dots appear, these dots can be different colors, and they represent the active tasks inside the node. The active node will also display the number of completed , current, and non-active tasks on the left hand side of itself. The PDG interface is great for keeping track how Houdini is handling your task list per node.

​

Event Handling

​

The PDG interface runs on a custom python script to handle the cooking of your items. Event handling describes the process of how the PDG handles the changes and events that happen during this cooking period. 

​

In order for an event handler to work, it needs to reference a  node, work item, or graph context. You can have more than one event handler working at the same time.

​

When your tasks are finished, you may want to remove the event handlers that are no longer doing anything. You can do this by calling the event handler with this script: pdg_node.removeEventHandler(node_handler).

​

Callbacks

​

In order to understand what a callback is, you have to understand what a possessor does. Processors are one of the main types of nodes in a PDG graph. They create new work items, parameters , and attributes on work items. You can think of them as a node that is both a merge and an attribute create node merged together, or even a ROP Fetch node. All processors can be either static or dynamic. Setting this option will decide which type of work item the processor will create.

​

In these processors  there is an option for callbacks. This function can control how the node operates. There are only three different callback functions you need to use at all time. The onGenerate function is one of them. This is responsible for producing work items. Without work items your TOPs network won't operate. By providing a onCookTask callback you can cook the work items. Lastly any work items need a inProcess callback in order to start their cooking or progress in the processor. 

​

Attributes and Nodes

​

TOPs have similar nodes and attributes to the other networks inside of Houdini. Here are some that you can familiarize yourself with. 

​

Nodes

​

Map by Expression

​

This node is similar to the Map by range node. But instead of remapping upstream(input) items with their downstream(output) counterparts with a range of numbers, it uses expressions. However, using a Python Mapper TOPs node can do similar results.

​

Map by Range

​

This node maps the number of input items coming into it, with the number of items downstream of it, or ones it has already outputted. For example, if an item being inputted into the node has a value of 3, and the one it has just sent downstream has a value of 1, the upstream(input) items will be remapped and set down stream with a similar value. 

​

Invoke

​

This node operates with something called complied blocks. In geometry networks, you can put a part of the network inside a compiled block. This will make all the nodes inside of it act like one node. What the Invoke node is designed to is take the chosen complied block and run it over incoming input geometry. This can allow for data from the complied block to be added to the input geometry.

​

Error Handler

​

This node will output copies of any failed work items from elsewhere in the network. If any of your items fail, this node will mark them, and tell you which ones failed.

​

Nuke Server Begin

​

This node a Nuke command server. Similar to the Maya Server Begin node below, It is a modified version of the Python Server TOPs node. However, it creates Nuke Server work items.

Maya Server Begin

​

This node starts a Maya command server. It is a modified version of the Python Server TOPs node. However, this one can be used to create Maya Server work items. 

​

Partition by Range

​

This node sorts work items based on their ranged values. The partitioner in this node divides the work into different sections so the computer can decide what to render next , or which work items can have a higher priority. In this case the node is given a list of work items to handle, and based on their number in the sequence it will decide which item to handle first.

Send Email

​

This node sends an email to whichever person or organization you desire. However, it requires the The host-name of the SMTP server to send the email. (For example somename@gmail.com) You also have settings to add an encryption method for the email, and includes custom file path to the item you are sending in the email.

​

Wedge

​

This node creates work items with varying attribute values. Basically, what this means is that the node stores data of multiple versions of an image or piece of geometry, with different settings. This allows you to create multiple versions of whatever you are making without forgetting the changing values, or creating errors.

​

Built in Attributes

​

In TOPs these are some of the main attributes you will use while creating your setup. If you'd like to reference these attributes at anytime, you have to use @pdg_ in-front of it. For example: @pdg_name.

​

• Frame: This attribute references the frame number this item is working on.

​

• Name: This attribute is the name/descriptor of individual work items.

​

• Index: This attribute helps sorting work items within a node.

​

• Output: This attribute references all the work items being passed downstream or out of the nodes. It helps produce a list of filenames as the output of for the work.

​

• Input: This attribute references all the work items being passed upstream or  into the nodes. It helps produce a list of filenames as the input of for the work.

​

References

​

Introduction to TOPs: https://www.sidefx.com/docs/houdini/tops/intro.html

​

Using attributes: https://www.sidefx.com/docs/houdini/tops/attributes.html

​

TOP nodes: https://www.sidefx.com/docs/houdini/nodes/top/index.html

​

Wedging attributes: https://www.sidefx.com/docs/houdini/tops/wedge.html

​

Tips and tricks: https://www.sidefx.com/docs/houdini/tops/tips.html

​

HOW TO INGEST AN ASSET FROM A TO AROUND Z WITH TOPS ( PART I ): https://projectjulien.space/houdini-tutorials/2019/06/21/ingest-asset-from-a-to-around-z-part-i

​

TOPs video mosaic for simulations: https://forums.odforce.net/topic/44183-tops-video-mosaic-for-simulations/

​

Mantra is 3x -7x faster in PDG/TOPs...Is this normal?: https://www.reddit.com/r/Houdini/comments/elghpn/mantra_is_3x_7x_faster_in_pdgtopsis_this_normal/

​

Houdini PDG – The Procedural Dependency Graph event horizon: https://www.teotigraphix3d.com/2019/04/07/houdini-pdg-the-procedural-dependency-graph-event-horizon/

​

Houdini - TOP / PDG pyro wedging: https://realtimevfx.com/t/houdini-top-pdg-pyro-wedging/9141

​

PDG and TOPs: https://prism-pipeline.com/forum/topic/pdg-and-tops/