Rendering Action Scripts

Presentation

A rendering action in Puli is a python class that control the execution of a command on a dedicated rendernode, also called a Runner.
It is often associated with a Decomposer: utility class in charge of creating several command for a particular job when it is a repetition.

For instance when rendering a sequence of frames with MtoA:

• a decomposer will create several commands attached the a task (generally with identical arguments)
• a runner will be executed on each render node to actually launch the "kick" exec

Several runners and a default decomposer are available to fill common use case but other needs can arise.
In most cases, a TD will only create new runners for specific execution and use the DefaultTaskDecomposer available.

The Runner

It's a class that defines the execution workflow for each command. Runner characteristics:

• starts execute method when started on a rendernode
• can define several arguments requirements
• can raise CommandError or CommandDone exception (with a message) to inform the worker of a specific ending
• can use PuliActionHelper module to access several useful tools, for instance:
  • getting a command environment
  • printing log with default formatting
  • using a timeout on its execution
  • more information below

During the execution, the runner has several ways to communicate with the server, 4 callbacks are passed to its execute method : execute(self, arguments, updateCompletionCallback, updateMessageCallback, updateLicenseCallback, updateStatsCallback)

These will be used to update:

Callback	Description
updateCompletion	the completion level, a float in range [0-1]
updateMessage	a custom message string
updateLicense	a dict to release a specific license token
updateStats	a custom dict to use for statistics

The runner (i.e. the command) has a very limited knowledge of the graph it is executed from. This minimal information is set in its environment:

ENV	Description
PULI_COMMAND_ID	id of the command
PULI_TASK_ID	id of the parent task
PULI_TASK_NAME	name of the parent task
PULI_LOG	full path to the command log file

Here is a commented runner example :

class TestRunner(CommandRunner):
    def execute(self, arguments, updateCompletion, updateMessage, updateStats):
        helper = PuliActionHelper()

        # arguments corresponds to the dict that we added to the command in the Decomposer (cmdArgs)

        # if necessary, we can map path (for cross platform rendering)
        prodPath = helper.mapPath(arguments[PROJECT])

        # here we get the appropriate env for our rendering process
        env = helper.getEnv(am_version=arguments[ARNOLD_VERSION],       
                            maya_version=arguments[MAYA_VERSION],       
                            shave_version=arguments[SHAVE_VERSION],
                            crowd_version=’’,
                            home=os.environ["HOME"], 
                            job=os.path.basename(prodPath), 
                            jobdrive=os.path.dirname(prodPath), 
                            applis=helper.mapPath("/s/apps/lin"),
                            use_shave=0)

        # tells Puli that we are at 5% completion :
        updateCompletion(0.05)
        updateMessage("About to actually start the process.")

        # At this moment we can release a license token
        licInfo = { "licenseName":"shave", "action":"release" }
        updateLicense( licInfo )

        # Do important work here
        try:
            # Proper action
            # ...
        except Exception, e:
            # Use specific exception to inform the worker of an error
            raise CommandError("error message")

        # Post-process actions like updating stats
        updateCompletion(0.9)
        updateMessage("We are close to the end.")
        updateStats( {'useful_data': "xyz" } )

        # Completion will automatically be updated to 100% when ending a command

Default Decomposer

With Puli, tasks are split in one or more commands that will be executed concurrently on several machines. A Decomposer is a python class, attached to a Task that generates these commands.

A default decomposer is shipped with the API and should cover most of your needs. It has useful facilities to create one or several commands given specific fields.

When creating a task a user usually uses the following call:

name = "MyTask"
args = {"start": 1, "end": 10, "packetSize": 3}
task = Task( name, arguments=args )

Such task, defined with default decomposer and defaut runner will use the "arguments" dictionnary to:

• create one or several command
• run a process given as a string

Here are the standard fields of "arguments":

• start: the frame to render
• end: the last frame to render
• packetSize: the number of frame to pack together in a single command
• framesList: a string indicating space-separated frame numbers
• timeout: [optionnal] the max number of seconds allowed for the job execution. After this delay, the process will end and raise a TimeoutException.

If frameRange or framesList field exists in arguments, commands will be created accordingly. If not, but the 3 fields: "start, end, packetSize" exist, commands will be created accordingly. If none of those fields exist, a single command is created.

The name of a command is modified to represent the frame range on wich it will work: For instance given the previous script, the result hierarchy will be:

MyTask 
  |- MyTask_1_3     (a command with 3 frames) 
  |- MyTask_4_6     (a command with 3 frames) 
  |- MyTask_7_9     (a command with 3 frames) 
  `- MyTask_10_10   (last command with 1 remaining frame) 

Another example:

name = "MyTask"
args = {"start": 1, "end": 10, "packetSize": 3, "cmd": "my_program -f $i"}
task = Task( name, arguments=args )

For example, say we want to split a Task that has a START and END into chunks of PACKET_SIZE, here's our Decomposer :

# coding: utf-8
from puliclient.jobs import TaskDecomposer
from puliclient.contrib.helper.helper import PuliActionHelper

class TestDecomposer(TaskDecomposer):
   def __init__(self, task):
       self.task = task

       # The decompose method will split the task from start to end in packet_size and call the addCommand method below for each chunk
       PuliActionHelper().decompose(task.arguments["START"], task.arguments["END"],
                                    task.arguments["PACKET_SIZE"], self)

   def addCommand(self, packetStart, packetEnd):
       # get all arguments from the Task
       cmdArgs = self.task.arguments.copy()

       # change values of start and end
       cmdArgs["START"] = packetStart
       cmdArgs["END"] = packetEnd
       cmdName = "%s_%s_%s" % (self.task.name, str(packetStart), str(packetEnd))

       # Add the command to the Task
       self.task.addCommand(cmdName, cmdArgs)

Now that we have our Decomposer, we will set it on the Task like this :

name = "25 commands with 4 frames per command"
args = {"START": 1, "END": 100, "PACKET_SIZE": 4}
task = Task(name, arguments=args, decomposer="test.TestDecomposer")

The helper module

PuliActionHelper.decompose(start, end, packetSize, callback)

Cette méthode sert à décomposer une tâche en plusieurs commandes en se basant sur les paramètres start, end et packetSize et en faisant appel à la méthode addCommand du callback.

PuliActionHelper.mapPath(path)

Mappe le chemin path en fonction de la plateforme actuelle. (utilise Luigi)

PuliActionHelper.getEnv(am_version, maya_version, shave_version, crowd_version, home, job, jobdrive, applis, use_shave, nuke_rep)

Retourne un dictionnaire contenant les variables d’environnement créées à partir des différentes versions passées en arguments. (utilise Luigi)

PuliActionHelper.isLinux()

Retourne True si on est sur une plateforme linux.

PuliActionHelper.printStartLog(name, version)

Affiche un header contenant le nom de l’action fourni en argument, sa version, le nom de la machine actuelle ainsi que la date et l’heure.

PuliActionHelper.checkExistenceOrCreateDir(path, name)

Vérifie l’existence du chemin path et crée les dossiers le cas échéant. name est une description du path (par exemple ‘render dir’).

PuliActionHelper.execute(cmdArgs, env) PuliActionHelper.executeWithTimeout(cmdArgs, env, timeout)

Lance un nouveau process avec le tableau d’arguments cmdArgs et en utilisant le dictionnaire env contenant les variables d’environnement. Retourne le code de retour de la commande, s’il existe.

PuliActionHelper.buildMayaCommand(mikserActionScript, arguments, additionalArguments, env)

Retourne un tableau contenant les arguments pour une commande de rendu maya. mikserActionScript correspond au nom du script python se trouvant dans /s/apps/lin/maya/scripts/ et permettant de setter les attributs sur la scène maya. arguments est le dictionnaire d’arguments pour le script python et additionalArguments est un tableau contenant les chemins du projet et de la scène maya.

PuliActionHelper.buildNukeCommand(arguments, localNukeScene)

Retourne un tableau contenant les arguments pour une commande de rendu nuke.