.. _scalingLimitations:

=========================
Dirac Service Limitations
=========================


DataManagement
==============

----------
LFN length
----------

Because they are stored in a database, the LFNs are limited in size. The standard size is 255 characters. It is enforced in the following database:

* JobDB
* TransformationTB
* StorageManagementDB
* DataIntegrityDB
* FTS3DB
* RequestDB

Shall you want to have longer LFN, then you would need to update your database manually.

A special case is the DFC. The limitations depend on the Directory and File managers you use.

In the DirectoryLevelTree and FileManager (defaults one) managers, the LFNs are split by '/', yielding other limitations:

* 128 char for the filename
* 255 char for each directory level

In case of the Managers WithPkAndPs (LHCb):

* 128 char for the filename
* 255 for the base path



========================
Performance Optimisation
========================


Servers
=======

When you servers are heavily loaded, you may want to tune some kernel parameters. Internet is full or resources to explain you what you should do, but a few parameters of interests certainly are the number of file descriptors allowed, as well as a few kernel tcp parameters that should be increased (https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)::

   net.nf_conntrack_max
   net.ipv4.tcp_max_syn_backlog
   net.core.somaxconn
   net.core.netdev_max_backlog

You can also adjust the limit of opened files descriptors in the ``Service`` section of the ``/usr/lib/systemd/system/runsvdir-start.service`` file::

   LimitNOFILE=500000


Databases
=========

Every now and then, it is interesting to look at the fragmentation status of your database. This is done by using the ``analyze table`` statement (https://dev.mysql.com/doc/refman/8.4/en/analyze-table.html) possibly followed by the ``optimize table`` statement (https://dev.mysql.com/doc/refman/8.4/en/optimize-table.html).

To know whether your tables are fragmented::

   select table_schema,table_name, sys.format_bytes(data_length) table_size, sys.format_bytes(data_free) empty_space from information_schema.tables where data_length >= (1024*1024*1024) order by data_length desc;


The fragmented space should be very small with respect to the overall table size.




Duplications
============

The WMS **Executors** can be duplicated.

In some cases, it is possible to run several instances of the same agent/service in order to scale.

NB: in the tables below, for those components where the **Duplicate** column is not filled,
it simply means that we have not tried to duplicate them yet.
For services, this probably means that they can be duplicated.
For agents, instead, this probably means that they can not be duplicated.

Services
========

+--------------------+---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
| **System**         | **Component**                                                                                     |**Duplicate**| **Remarks**                                                               | **HTTPs** +
+--------------------+---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
| Accounting         | :mod:`DataStore <DIRAC.AccountingSystem.Service.DataStoreHandler>`                                | PARTIAL     | One controller and helpers (See :ref:`datastorehelpers`)                  |           +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`ReportGenerator <DIRAC.AccountingSystem.Service.ReportGeneratorHandler>`                    |             |                                                                           |           +
+--------------------+---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
| Configuration      | :mod:`Configuration <DIRAC.ConfigurationSystem.Service.ConfigurationHandler>`                     | PARTIAL     | One controller (rw) and workers (ro). Should have several CS workers      | YES       +
+--------------------+---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
| DataManagement     | :mod:`DataIntegrity <DIRAC.DataManagementSystem.Service.DataIntegrityHandler>`                    | YES         |                                                                           | YES       +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`FileCatalog <DIRAC.DataManagementSystem.Service.FileCatalogHandler>`                        | YES         |                                                                           | YES       +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`FTS3Manager <DIRAC.DataManagementSystem.Service.FTS3ManagerHandler>`                        | YES         |                                                                           | YES       +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`S3Gateway <DIRAC.DataManagementSystem.Service.S3GatewayHandler>`                            | YES         |                                                                           | YES       +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`StorageElement <DIRAC.DataManagementSystem.Service.StorageElementHandler>`                  |             |                                                                           |           +
+--------------------+---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
| Framework          | :mod:`BundleDelivery <DIRAC.FrameworkSystem.Service.BundleDeliveryHandler>`                       | YES         |                                                                           | YES       +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`ComponentMonitoring <DIRAC.FrameworkSystem.Service.ComponentMonitoringHandler>`             | YES         |                                                                           | YES       +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`Notification <DIRAC.FrameworkSystem.Service.NotificationHandler>`                           | YES         |                                                                           | YES       +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`ProxyManager <DIRAC.FrameworkSystem.Service.ProxyManagerHandler>`                           | YES         |                                                                           | YES       +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`SecurityLogging <DIRAC.FrameworkSystem.Service.SecurityLoggingHandler>`                     | **NO**      |                                                                           |           +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`SystemAdministrator <DIRAC.FrameworkSystem.Service.SystemAdministratorHandler>`             | **MUST**    | There should be one on each and every machine                             |           +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`TokenManager <DIRAC.FrameworkSystem.Service.TornadoTokenManagerHandler>`                    | YES         |                                                                           | YES(only) +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`UserProfileManager <DIRAC.FrameworkSystem.Service.UserProfileManagerHandler>`               | YES         |                                                                           | YES       +
+--------------------+---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
| Monitoring         | :mod:`Monitoring <DIRAC.MonitoringSystem.Service.MonitoringHandler>`                              | YES         |                                                                           | YES       +
+--------------------+---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
| RequestManagement  | :mod:`ReqManager <DIRAC.RequestManagementSystem.Service.ReqManagerHandler>`                       | YES         |                                                                           | YES       +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`ReqProxy <DIRAC.RequestManagementSystem.Service.ReqProxyHandler>`                           | PARTIAL     | Relies on local storage                                                   |           +
+--------------------+---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
| ResourcesStatus    | :mod:`Publisher <DIRAC.ResourceStatusSystem.Service.PublisherHandler>`                            | YES         |                                                                           | YES       +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`ResourceManagement <DIRAC.ResourceStatusSystem.Service.ResourceManagementHandler>`          | YES         |                                                                           | YES       +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`ResourceStatus <DIRAC.ResourceStatusSystem.Service.ResourceStatusHandler>`                  | YES         |                                                                           | YES       +
+--------------------+---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
| StorageManager     | :mod:`StorageManager <DIRAC.StorageManagementSystem.Service.StorageManagerHandler>`               |             |                                                                           |           +
+--------------------+---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
| Transformation     | :mod:`TransformationManager <DIRAC.TransformationSystem.Service.TransformationManagerHandler>`    | YES         |                                                                           | YES       +
+--------------------+---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
| WorkloadManagement | :mod:`JobManager <DIRAC.WorkloadManagementSystem.Service.JobManagerHandler>`                      | YES         |                                                                           | YES       +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`JobMonitoring <DIRAC.WorkloadManagementSystem.Service.JobMonitoringHandler>`                | YES         |                                                                           | YES       +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`JobStateUpdate <DIRAC.WorkloadManagementSystem.Service.JobStateUpdateHandler>`              | YES         |                                                                           | YES       +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`Matcher <DIRAC.WorkloadManagementSystem.Service.MatcherHandler>`                            | **NO**      |                                                                           |           +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`OptimizationMind <DIRAC.WorkloadManagementSystem.Service.OptimizationMindHandler>`          | **NO**      |                                                                           |           +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`PilotManager <DIRAC.WorkloadManagementSystem.Service.PilotManagerHandler>`                  | PARTIAL     | In case there are HTCondor CEs to deal with, the HTCondor                 |           +
|                    |                                                                                                   |             | WorkingDirectory should exist and be accessible in each and every machine |           +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`SandboxStore <DIRAC.WorkloadManagementSystem.Service.SandboxStoreHandler>`                  |             |                                                                           |           +
+                    +---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+
|                    | :mod:`WMSAdministrator <DIRAC.WorkloadManagementSystem.Service.WMSAdministratorHandler>`          | YES         |                                                                           | YES       +
+--------------------+---------------------------------------------------------------------------------------------------+-------------+---------------------------------------------------------------------------+-----------+


Agents
======

+--------------------+---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
| **System**         | **Component**                                                                                     | **Duplicate** | **Remarks**                                                                       |
+--------------------+---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
| Accounting         | :mod:`~DIRAC.AccountingSystem.Agent.NetworkAgent`                                                 |               |                                                                                   |
+--------------------+---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
| Configuration      | :mod:`~DIRAC.ConfigurationSystem.Agent.Bdii2CSAgent`                                              | PARTIAL       | Select sites for different agent instances                                        |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.ConfigurationSystem.Agent.GOCDB2CSAgent`                                             |               |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.ConfigurationSystem.Agent.VOMS2CSAgent`                                              | PARTIAL       | Split by VOs                                                                      |
+--------------------+---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
| DataManagement     | :mod:`~DIRAC.DataManagementSystem.Agent.FTS3Agent`                                                | YES           |                                                                                   |
+--------------------+---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
| RequestManagement  | :mod:`~DIRAC.RequestManagementSystem.Agent.CleanReqDBAgent`                                       | NO            |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.RequestManagementSystem.Agent.RequestExecutingAgent`                                 | YES           |                                                                                   |
+--------------------+---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
| ResourceStatus     | :mod:`~DIRAC.ResourceStatusSystem.Agent.CacheFeederAgent`                                         | YES           |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.ResourceStatusSystem.Agent.ElementInspectorAgent`                                    |               |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.ResourceStatusSystem.Agent.EmailAgent`                                               | NO            |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.ResourceStatusSystem.Agent.SiteInspectorAgent`                                       |               |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.ResourceStatusSystem.Agent.SummarizeLogsAgent`                                       |               |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.ResourceStatusSystem.Agent.TokenAgent`                                               |               |                                                                                   |
+--------------------+---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
| StorageManagement  | :mod:`~DIRAC.StorageManagementSystem.Agent.RequestFinalizationAgent`                              | NO            |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.StorageManagementSystem.Agent.RequestPreparationAgent`                               | NO            |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.StorageManagementSystem.Agent.StageMonitorAgent`                                     | NO            |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.StorageManagementSystem.Agent.StageRequestAgent`                                     | NO            |                                                                                   |
+--------------------+---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
| Transformation     | :mod:`~DIRAC.TransformationSystem.Agent.InputDataAgent`                                           | PARTIAL       | Split by TransformationTypes                                                      |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.TransformationSystem.Agent.MCExtensionAgent`                                         | PARTIAL       | Split by TransformationTypes                                                      |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.TransformationSystem.Agent.RequestTaskAgent`                                         | PARTIAL       | Split by TransformationTypes                                                      |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.TransformationSystem.Agent.TransformationAgent`                                      | PARTIAL       | Split by TransformationTypes                                                      |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.TransformationSystem.Agent.TransformationCleaningAgent`                              | PARTIAL       | Split by TransformationTypes                                                      |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.TransformationSystem.Agent.ValidateOutputDataAgent`                                  | PARTIAL       | Split by TransformationTypes                                                      |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.TransformationSystem.Agent.WorkflowTaskAgent`                                        | PARTIAL       | Split by TransformationTypes                                                      |
+--------------------+---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
| WorkloadManagement | :mod:`~DIRAC.WorkloadManagementSystem.Agent.SiteDirector`                                         | PARTIAL       | Split by Sites, CETypes                                                           |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.WorkloadManagementSystem.Agent.JobAgent`                                             |               | Installed by Pilots on Worker Nodes, not for server installations                 |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.WorkloadManagementSystem.Agent.JobCleaningAgent`                                     | YES           |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.WorkloadManagementSystem.Agent.PilotSyncAgent`                                       | YES           |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.WorkloadManagementSystem.Agent.PilotStatusAgent`                                     | YES           |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.WorkloadManagementSystem.Agent.PushJobAgent`                                         | YES           | Split by Sites                                                                    |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.WorkloadManagementSystem.Agent.StalledJobAgent`                                      | YES           |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.WorkloadManagementSystem.Agent.StatesAccountingAgent`                                | NO            |                                                                                   |
+                    +---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+
|                    | :mod:`~DIRAC.WorkloadManagementSystem.Agent.TaskQueuesAgent`                                      | YES           |                                                                                   |
+--------------------+---------------------------------------------------------------------------------------------------+---------------+-----------------------------------------------------------------------------------+