QueueUtilities

Utilities to help Computing Element Queues manipulation

class DIRAC.WorkloadManagementSystem.Utilities.QueueUtilities.QueueCECache(backing=None)

Bases: object

A cache of ComputingElement instances keyed by queue.

CEs – and, for connection-based CEs such as the SSHComputingElement, their underlying connections – are reused across cycles instead of being re-created on every use. A CE is rebuilt only when its queue parameters change, detected through a hash of the parameters dictionary: the same invalidation strategy used by the SiteDirector (see getQueuesResolved()).

__init__(backing=None)

Initialise the cache, optionally adopting an existing backing store.

Parameters:

backing (dict) – optional pre-existing queueKey -> {"Hash", "CE"} dict to adopt as the cache store. Used to stay backward compatible with callers that historically passed (and held onto) a plain dict cache; mutating it in place preserves their cross-cycle reuse. A fresh dict is used when omitted.

drop(queueKey)

Evict a cached CE so it is rebuilt on the next request.

The evicted CE is shut down so that any underlying connection (e.g. the SSHComputingElement’s SSH/gateway connections) is closed immediately rather than left to non-deterministic garbage collection, which Fabric documents as unsafe to rely on.

getCE(queueKey, ceType, ceName, ceParametersDict)

Return a cached CE for queueKey, (re)building it when needed.

Parameters:

  • queueKey (str) – unique identifier of the queue, used as cache key

  • ceType (str) – CE type passed to the ComputingElementFactory

  • ceName (str) – CE name passed to the ComputingElementFactory

  • ceParametersDict (dict) – queue/CE parameters; a change triggers a rebuild

Returns:

S_OK(ce)/S_ERROR

DIRAC.WorkloadManagementSystem.Utilities.QueueUtilities.computeQueueCPULimit(queueDict)

Evaluate the CPU limit of the queue according to the Glue convention

DIRAC.WorkloadManagementSystem.Utilities.QueueUtilities.generateQueueHash(queueDict)

Generate a hash of the queue description

DIRAC.WorkloadManagementSystem.Utilities.QueueUtilities.getQueuesResolved(siteDict, queueCECache=None, vo=None, checkPlatform=False, instantiateCEs=False)

Get the list of relevant CEs (what is in siteDict) and their descriptions. The main goal of this method is to return a dictionary of queues

Parameters:

  • siteDict (dict) – sites/CEs/queues structure as returned by getQueues

  • queueCECache – a QueueCECache used to reuse CE instances across cycles when instantiateCEs is set. For backward compatibility a plain dict (the legacy cache format) is also accepted and adopted as the cache backing store; if omitted, a fresh cache is used.

  • vo (str) – VO name

  • checkPlatform (bool) – resolve the queue platform

  • instantiateCEs (bool) – instantiate (and cache) a CE object per queue

Returns:

S_OK(queueDict)/S_ERROR

DIRAC.WorkloadManagementSystem.Utilities.QueueUtilities.matchQueue(jobJDL, queueDict, fullMatch=False)

Match the job description to the queue definition

Parameters:

  • job (str) – JDL job description

  • fullMatch (bool) – test matching on all the criteria

  • queueDict (dict) – queue parameters dictionary

Returns:

S_OK/S_ERROR, Value - result of matching, S_OK if matched or S_ERROR with the reason for no match

DIRAC.WorkloadManagementSystem.Utilities.QueueUtilities.resolveTags(ceDict, queueDict)

Tags & RequiredTags defined on the Queue level and on the CE level are concatenated. This also converts them from a string to a list if required.

DIRAC.WorkloadManagementSystem.Utilities.QueueUtilities.setAdditionalParams(ceDict, queueDict)

Some parameters can be defined on the CE level and are inherited by all Queues

DIRAC.WorkloadManagementSystem.Utilities.QueueUtilities.setPlatform(ceDict, queueDict)

Set platform according to CE parameters if not defined