Manager

The manager is a service that sits between the middleware and the backend, and it is customized for a particular combination of simulator, compute infrastructure, and cloud storage infrastructure.

The following API endpoints are provided, which are called by the middleware:

POST

/job/<job_id>/start

payload = {"fields_to_patch": [
                {
                "name" : <field_name>,
                "value": <val>
                },
                ...
        ],
        "scripts" : [
                {
                "name" : <script_name>,
                "location" : <script_location>
                },
                ...
        ]
}

return = {"data": <message>,
          "status": <status_code>
}

Start a new job with id <job_id>.

GET

/job/<job_id>/output

return = [
    {"job_id": <job_id>,
     "output_type": <file_extension>,
     "destination_path": <URL>
    },
    ...
]

When a job is finished, a call to this endpoint will yield the URLs needed to access the job outputs. In many cases, these will include temporary access tokens generated by the job manager.

The job manager is also responsible for notifying the middleware of various, occurences, via the following API calls:

PUT  request to <middleware_url>/job/<job_id>/status
payload = {"status": <job_status>}

where the job_status must be one of "QUEUED", "RUNNING", "FINALIZING", "COMPLETED" or "FAILED".

POST request to <middleware_url>/job/<job_id>/output
payload = {"job_id": <job_id>,
           "output_type": <output_type>,
   "destination_path": <URL>
   }

This API call is made as soon as the job manager is aware that the job has successfully completed, in order to notify the middleware that the outputs are available. If some sort of temporary access token is needed to access the data, it will generally not be appended to the destination_path URL here - instead, the middleware will make a GET request to the output endpoint of the manager, at which point the manager will obtain the token.

OpenFOAM Job Manager

At present, the only fully-implemented manager is for the OpenFOAM simulator, running on a machine that can be ssh-ed to, and storing the output on Azure blob storage.

The service is written in Python 3, and uses the Flask framework. Calls to the middleware API are made using the requests package. Communication with the machine (or Docker container) running the OpenFOAM simulator is via ssh.

The following API endpoint on the job manager is called by the backend:

PATCH

/job/<job_id>/status

payload = {"status": <job_status>}

return = {"status": <status_code>,
          "message": <message>}

OR (if job_status is “FINALIZING”):

return = {"status": <job_status>,
           "data": {"token": <SAS token>,
                "container": <Azure container>,
        "account": <Azure account name>,
        "blob": <Azure blob name>
        }
      }

The backend is able to update the status of a job by calling this endpoint, which in turn triggers the manager to call the job status endpoint of the middleware.

Starting a job

When the job start endpoint is hit, the manager performs the following steps:

  • Retrieve the scripts from the specified location (on Azure blob storage in the currently implemented demo).
  • Patch the “fields_to_patch” parameters in the scripts with the specified values, using Mako.
  • Copy the scripts to the backend over ssh.
  • For scripts with specified “actions”, execute those actions on the backend. The primary example for this is the “RUN” action, which will trigger the manager to run that script on the backend, in order to launch the job.

Finishing a job

When the backend hits the job status endpoint with a status of “FINALIZING”, the manager will call the prepare_output_storage method which will:

  • Use the Azure credentials stored in config.json to generate a Shared Access Signature (SAS) token, with “write” permissions, valid for one hour.
  • Create a container on Azure blob storage, with the name specified in config.json.
  • Define the name of the blob that will be uploaded to Azure. The blob name is constructed from a base-name defined in config.py and the job_id.

The Azure container name, blob name, and SAS token are returned to the backend, as described in the API endpoint description above.

When the backend sends a status of “COMPLETED”, the manager calls the get_outputs function, which finds the URL of the blobs on Azure blob storage. It then calls the middleware’s output API endpoint with this information, as detailed above. Note that there is no SAS token appended to the output URLs at this point.

Retrieving output

When the job output endpoint is hit, the manager will generate a SAS token with “read” access valid for one hour, and append this to the output blob’s URL. The file-type and full URL are then returned to the middleware, as detailed in the API endpoint description above.