- Getting Started
- Requirements
- Best Practices
- Installation
- Updating
- Identity Server
- Troubleshooting startup errors
Performance Best Practices
The following page describes the recommended requirements for large-scale production environments and provides a set of best practices aimed at improving their performance.
To run 10K Attended Robots or 1K Unattended Robots, you need:
- An F5 load balancer
- Orchestrator - at least 6 nodes that run on machines with 8 CPU Cores and 16 GB RAM
- Robots - machines with 4 CPU Cores and 16 GB RAM
-
SQL Server - machines with 4 sockets / 16 CPU Cores (default 8 sockets/16 CPU Cores) and 64 GB RAM
Note: To reduce SQL Server allocation contention in a highly concurrent environment, make sure you employ an optimal number of tempdb data files that have equal sizing.
UiPathOrchestrator.dll.config
file in order to improve performance in a large scale production environment.
<appSettings>
<add key="LoadBalancer.UseRedis" value="true"/>
<add key="LoadBalancer.Redis.ConnectionString" value="your.redis.cache.windows.net:6379,password=***"/>
</appSettings>
<appSettings>
<add key="LoadBalancer.UseRedis" value="true"/>
<add key="LoadBalancer.Redis.ConnectionString" value="your.redis.cache.windows.net:6379,password=***"/>
</appSettings>
<connectionStrings>
<add name="Default" providerName="System.Data.SqlClient" connectionString="Data Source=<SQLServer>;Initial Catalog=<dbName>;User ID=<username>;Password=***;Max Pool Size=1000" />
</connectionStrings>
<connectionStrings>
<add name="Default" providerName="System.Data.SqlClient" connectionString="Data Source=<SQLServer>;Initial Catalog=<dbName>;User ID=<username>;Password=***;Max Pool Size=1000" />
</connectionStrings>
- Add or modify the logging target for Elasticsearch using the following parameters.
xml <nlog> <targets> <target name="robotElasticBuffer" xsi:type="AsyncWrapper" overflowAction="Grow" queueLimit="10000" batchSize="1000" timeToSleepBetweenBatches="1"> <target xsi:type="ElasticSearch" name="robotElastic" uri="elastic.example.com:9200" requireAuth="true" username="***" password="***" index="${event-properties:item=indexName}-${date:format=yyyy.MM}" documentType="logEvent" includeAllProperties="true" layout="${message}" excludedProperties="agentSessionId,tenantId,indexName" /> </target> </targets> </nlog>
xml <nlog> <targets> <target name="robotElasticBuffer" xsi:type="AsyncWrapper" overflowAction="Grow" queueLimit="10000" batchSize="1000" timeToSleepBetweenBatches="1"> <target xsi:type="ElasticSearch" name="robotElastic" uri="elastic.example.com:9200" requireAuth="true" username="***" password="***" index="${event-properties:item=indexName}-${date:format=yyyy.MM}" documentType="logEvent" includeAllProperties="true" layout="${message}" excludedProperties="agentSessionId,tenantId,indexName" /> </target> </targets> </nlog> - Add or modify the robot logs rule to write logs only to the target shown in the previous example. This automatically disables
the default Database target.
xml <nlog> <rules> <logger name="Robot.*" ruleName="primaryRobotLogsTarget" final="true" writeTo="robotElasticBuffer" /> </rules> </nlog>
xml <nlog> <rules> <logger name="Robot.*" ruleName="primaryRobotLogsTarget" final="true" writeTo="robotElasticBuffer" /> </rules> </nlog> - Configure Orchestrator to read logs from ElasticSearch.
<appSettings> <add key="Logs.RobotLogs.ReadTarget" value="robotElasticBuffer" /> </appSettings>
<appSettings> <add key="Logs.RobotLogs.ReadTarget" value="robotElasticBuffer" /> </appSettings>
You can opt for Azure Blob Storage for this purpose.
<appSettings>
<add key="Storage.Type" value="Azure" />
<add key="Storage.Location" value="DefaultEndpointsProtocol=https;AccountName=yourBlob;AccountKey=yourKey;EndpointSuffix=core.windows.net" />
</appSettings>
<appSettings>
<add key="Storage.Type" value="Azure" />
<add key="Storage.Location" value="DefaultEndpointsProtocol=https;AccountName=yourBlob;AccountKey=yourKey;EndpointSuffix=core.windows.net" />
</appSettings>
If you don’t use the Webhooks feature, you can disable it.
<appSettings>
<add key="Webhooks.Enabled" value="false" />
<add key="Ledger.Enabled" value="false"/>
<add key="Webhooks.LedgerIntegration.Enabled" value="false" />
</appSettings>
<appSettings>
<add key="Webhooks.Enabled" value="false" />
<add key="Ledger.Enabled" value="false"/>
<add key="Webhooks.LedgerIntegration.Enabled" value="false" />
</appSettings>
If you don’t use Maintenance Mode, you can disable it.
<appSettings>
<add key="MaintenanceMode.Enabled" value="false" />
</appSettings>
<appSettings>
<add key="MaintenanceMode.Enabled" value="false" />
</appSettings>
You can configure Orchestrator to use some in-memory caches that reduce the number of calls to Redis. For instance, configuring them to expire every 30 minutes means that, once you make a change to an application-level setting, it may not be reflected until after the 30-minute interval.
The expiry time you set here can vary depending on what delay you can tolerate, but even a 5-minute caching time can greatly improve performance.
<appSettings>
<add key="Caching.MultiTier[AbpApplicationSettingsCache]" value="Memory" />
<add key="Caching.MultiTier[AbpApplicationSettingsCache].ExpiresAfter" value="0:30:00" />
<add key="Caching.MultiTier[AbpTenantSettingsCache]" value="Memory"/>
<add key="Caching.MultiTier[AbpTenantSettingsCache].ExpiresAfter" value="0:30:00"/>
<add key="Caching.MultiTier[AbpZeroTenantFeatures]" value="Memory"/>
<add key="Caching.MultiTier[AbpZeroTenantFeatures].ExpiresAfter" value="0:30:00"/>
<add key="Caching.MultiTier[AbpZeroRolePermissions]" value="Memory"/>
<add key="Caching.MultiTier[AbpZeroRolePermissions].ExpiresAfter" value="0:30:00"/>
<add key="Caching.MultiTier[LicenseKeys]" value="Memory"/>
<add key="Caching.MultiTier[LicenseKeys].ExpiresAfter" value="0:30:00"/>
<add key="Caching.MultiTier[TenantLicense]" value="Memory"/>
<add key="Caching.MultiTier[TenantLicense].ExpiresAfter" value="0:30:00"/>
</appSettings>
<appSettings>
<add key="Caching.MultiTier[AbpApplicationSettingsCache]" value="Memory" />
<add key="Caching.MultiTier[AbpApplicationSettingsCache].ExpiresAfter" value="0:30:00" />
<add key="Caching.MultiTier[AbpTenantSettingsCache]" value="Memory"/>
<add key="Caching.MultiTier[AbpTenantSettingsCache].ExpiresAfter" value="0:30:00"/>
<add key="Caching.MultiTier[AbpZeroTenantFeatures]" value="Memory"/>
<add key="Caching.MultiTier[AbpZeroTenantFeatures].ExpiresAfter" value="0:30:00"/>
<add key="Caching.MultiTier[AbpZeroRolePermissions]" value="Memory"/>
<add key="Caching.MultiTier[AbpZeroRolePermissions].ExpiresAfter" value="0:30:00"/>
<add key="Caching.MultiTier[LicenseKeys]" value="Memory"/>
<add key="Caching.MultiTier[LicenseKeys].ExpiresAfter" value="0:30:00"/>
<add key="Caching.MultiTier[TenantLicense]" value="Memory"/>
<add key="Caching.MultiTier[TenantLicense].ExpiresAfter" value="0:30:00"/>
</appSettings>
MinWorkerThreads
helps with ramp-up scenarios.
<appSettings>
<add key="ThreadPool.MinWorkerThreads" value="50" />
</appSettings>
<appSettings>
<add key="ThreadPool.MinWorkerThreads" value="50" />
</appSettings>
This step is necessary only if you have a very high number of triggers firing, e.g. on the order of 2,000 per minute.
<quartz>
<add key="quartz.scheduler.batchTriggerAcquisitionMaxCount" value="15" />
<add key="quartz.threadPool.threadCount" value="15" />
<add key="quartz.jobStore.misfireThreshold" value="3600000" />
<add key="quartz.jobStore.clusterCheckinInterval" value="60000" />
<add key="quartz.jobStore.clustered" value="true" />
</quartz>
<quartz>
<add key="quartz.scheduler.batchTriggerAcquisitionMaxCount" value="15" />
<add key="quartz.threadPool.threadCount" value="15" />
<add key="quartz.jobStore.misfireThreshold" value="3600000" />
<add key="quartz.jobStore.clusterCheckinInterval" value="60000" />
<add key="quartz.jobStore.clustered" value="true" />
</quartz>
If you are not in a corporate network or do not need to see the DNS names of IPs recorded in Audit, you can disable the resolving.
<appSettings>
<add key="Audit.UseDnsResolving" value="false" />
</appSettings>
<appSettings>
<add key="Audit.UseDnsResolving" value="false" />
</appSettings>
Directory.ActiveDirectory.GroupMembershipFetchStrategy
and Directory.ActiveDirectory.AuthorizationGroupsCacheExpirationHours
only apply to the legacy AD adapter. You should use these settings only if you upgraded from an Orchestrator version that
had AD enabled to 2021.10 or later.
Directory.ActiveDirectory.GroupMembershipFetchStrategy
to TokenGroups
in the [identity].[Settings]
table, run:
SELECT * FROM [identity].[Settings] WHERE [Key]='Directory.ActiveDirectory.GroupMembershipFetchStrategy' and [PartitionId]=1
IF @@ROWCOUNT>0
UPDATE [identity].[Settings] SET [Value]='TokenGroups' WHERE [Key]='Directory.ActiveDirectory.GroupMembershipFetchStrategy' and [PartitionId]=1
ELSE
INSERT INTO [identity].[Settings] ([Key], [Value], [PartitionId]) VALUES ('Directory.ActiveDirectory.GroupMembershipFetchStrategy', 'TokenGroups', 1)
SELECT * FROM [identity].[Settings] WHERE [Key]='Directory.ActiveDirectory.GroupMembershipFetchStrategy' and [PartitionId]=1
IF @@ROWCOUNT>0
UPDATE [identity].[Settings] SET [Value]='TokenGroups' WHERE [Key]='Directory.ActiveDirectory.GroupMembershipFetchStrategy' and [PartitionId]=1
ELSE
INSERT INTO [identity].[Settings] ([Key], [Value], [PartitionId]) VALUES ('Directory.ActiveDirectory.GroupMembershipFetchStrategy', 'TokenGroups', 1)
Directory.ActiveDirectory.AuthorizationGroupsCacheExpirationHours
to 4 hours in the [identity].[Settings]
table, run:
SELECT * FROM [identity].[Settings] WHERE [Key]='Directory.ActiveDirectory.AuthorizationGroupsCacheExpirationHours' and [PartitionId]=1
IF @@ROWCOUNT>0
UPDATE [identity].[Settings] SET [Value]='4' WHERE [Key]='Directory.ActiveDirectory.AuthorizationGroupsCacheExpirationHours' and [PartitionId]=1
ELSE
INSERT INTO [identity].[Settings] ([Key], [Value], [PartitionId]) VALUES ('Directory.ActiveDirectory.AuthorizationGroupsCacheExpirationHours', '4', 1)
SELECT * FROM [identity].[Settings] WHERE [Key]='Directory.ActiveDirectory.AuthorizationGroupsCacheExpirationHours' and [PartitionId]=1
IF @@ROWCOUNT>0
UPDATE [identity].[Settings] SET [Value]='4' WHERE [Key]='Directory.ActiveDirectory.AuthorizationGroupsCacheExpirationHours' and [PartitionId]=1
ELSE
INSERT INTO [identity].[Settings] ([Key], [Value], [PartitionId]) VALUES ('Directory.ActiveDirectory.AuthorizationGroupsCacheExpirationHours', '4', 1)
Directory.ActiveDirectory.CacheExpirationSeconds
only applies to the legacy AD adapter. You should use this setting only if you upgraded from an Orchestrator version that
had AD enabled to 2021.10 or later.
Directory.ActiveDirectory.CacheExpirationSeconds
setting in the [identity].[Settings]
table. The default value of this setting is 43200 seconds (12 hours). To disable AD domain caching, you must set this setting
to 0.
To set this setting to 7 days, run:
SELECT * FROM [identity].[Settings] WHERE [Key]='Directory.ActiveDirectory.CacheExpirationSeconds' and [PartitionId]=1
IF @@ROWCOUNT>0
UPDATE [identity].[Settings] SET [Value]='604800' WHERE [Key]='Directory.ActiveDirectory.CacheExpirationSeconds' and [PartitionId]=1
ELSE
INSERT INTO [identity].[Settings] ([Key], [Value], [PartitionId]) VALUES ('Directory.ActiveDirectory.CacheExpirationSeconds', '604800', 1)
SELECT * FROM [identity].[Settings] WHERE [Key]='Directory.ActiveDirectory.CacheExpirationSeconds' and [PartitionId]=1
IF @@ROWCOUNT>0
UPDATE [identity].[Settings] SET [Value]='604800' WHERE [Key]='Directory.ActiveDirectory.CacheExpirationSeconds' and [PartitionId]=1
ELSE
INSERT INTO [identity].[Settings] ([Key], [Value], [PartitionId]) VALUES ('Directory.ActiveDirectory.CacheExpirationSeconds', '604800', 1)
To disable AD domain caching, run:
SELECT * FROM [identity].[Settings] WHERE [Key]='Directory.ActiveDirectory.CacheExpirationSeconds' and [PartitionId]=1
IF @@ROWCOUNT>0
UPDATE [identity].[Settings] SET [Value]='0' WHERE [Key]='Directory.ActiveDirectory.CacheExpirationSeconds' and [PartitionId]=1
ELSE
INSERT INTO [identity].[Settings] ([Key], [Value], [PartitionId]) VALUES ('Directory.ActiveDirectory.CacheExpirationSeconds', '0', 1)
SELECT * FROM [identity].[Settings] WHERE [Key]='Directory.ActiveDirectory.CacheExpirationSeconds' and [PartitionId]=1
IF @@ROWCOUNT>0
UPDATE [identity].[Settings] SET [Value]='0' WHERE [Key]='Directory.ActiveDirectory.CacheExpirationSeconds' and [PartitionId]=1
ELSE
INSERT INTO [identity].[Settings] ([Key], [Value], [PartitionId]) VALUES ('Directory.ActiveDirectory.CacheExpirationSeconds', '0', 1)
During an aggressive ramp-up, the license acquire events would put significant pressure on the SignalR Redis backplane. To mitigate that, the event can be disabled, which results in the licensing UI no longer being updated in real time. However, when you have 100k robots, that screen updating in real time is of little value.
<appSettings>
<add key="Scalability.SignalR.Browser.RobotLicenseChangeEventEnabled" value="false" />
</appSettings>
<appSettings>
<add key="Scalability.SignalR.Browser.RobotLicenseChangeEventEnabled" value="false" />
</appSettings>
This change ensures the server-side caching of the package contents, and it can prove useful when the file share cannot keep up with the package download throughput.
<<add key="NuGet.Caching.Enabled" value="true" />
<add key="NuGet.Caching.MaxCacheSize" value="104857600" /><!-- (100 MiB) -->
<add key="NuGet.Caching.MaxPackageSize" value="4194304" /><!-- (4 MiB) -->
<add key="NuGet.Caching.AbsoluteExpirationRelativeToNow" value="00:10:00" /><!-- (1 min) -->
<<add key="NuGet.Caching.Enabled" value="true" />
<add key="NuGet.Caching.MaxCacheSize" value="104857600" /><!-- (100 MiB) -->
<add key="NuGet.Caching.MaxPackageSize" value="4194304" /><!-- (4 MiB) -->
<add key="NuGet.Caching.AbsoluteExpirationRelativeToNow" value="00:10:00" /><!-- (1 min) -->
NuGet.Caching.Enabled
setting should be enough to enable the feature. You only need to adjust the rest of the settings displayed in the example
above if you want to change the defaults, e.g. increase the size of the cache or package size limits.
Set the following configuration as slot sticky on the production slot of the Orchestrator App Service.
WEBSITE_LOCAL_CACHE_OPTION = Always
WEBSITE_LOCAL_CACHE_SIZEINMB = 1000
WEBSITE_LOCAL_CACHE_OPTION = Always
WEBSITE_LOCAL_CACHE_SIZEINMB = 1000
Set the following configuration on both the production and hotswap deployment slots of the Orchestrator App Service. During deployment, this will warm up the application, minimizing the performance hit caused by redirecting traffic to a cold app.
WEBSITE_SWAP_WARMUP_PING_PATH = /api/status
WEBSITE_SWAP_WARMUP_PING_STATUSES = 200
WEBSITE_SWAP_WARMUP_PING_PATH = /api/status
WEBSITE_SWAP_WARMUP_PING_STATUSES = 200
Using Azure SignalR Service in an Azure deployment can significantly reduce the load on the web servers. See Scalability.AzureSignalR.ConnectionString for more details on the setting.
<appSettings>
<add key="LoadBalancer.UseRedis" value="true"/>
<add key="Scalability.AzureSignalR.ConnectionString" value="**myAzureSignalRConnectionString" />
</appSettings>
<appSettings>
<add key="LoadBalancer.UseRedis" value="true"/>
<add key="Scalability.AzureSignalR.ConnectionString" value="**myAzureSignalRConnectionString" />
</appSettings>
UiPath.Orchestrator.dll.config
and configure them according to your needs:
-
RobotsLogs.Flush.Interval
<add key="RobotsLogs.Flush.Interval" value="0" />
<add key="RobotsLogs.Flush.Interval" value="0" />
For more on this setting, see RobotsLogs.Flush.Interval.
-
RobotsLogs.Flush.BatchSize
<add key="RobotsLogs.Flush.BatchSize" value="100" />
<add key="RobotsLogs.Flush.BatchSize" value="100" />
For more on this setting, see RobotsLogs.Flush.BatchSize.
AppSettings.Production.json
file should result in improved performance in large scale production environments.
"ConnectionStrings": {
"DefaultConnection": "Data Source=dbServer;Initial Catalog=UiPath_is;User ID=username;Password=****;Max Pool Size=1000"
}
"ConnectionStrings": {
"DefaultConnection": "Data Source=dbServer;Initial Catalog=UiPath_is;User ID=username;Password=****;Max Pool Size=1000"
}
Orchestrator's UI allows you to take the following steps in order to improve performance in large scale production environments:
- Disable personal workspace;
- Disable Allow web login for robot users;
- Assign Robot and Automation User roles to robot users.
- Recommended Requirements for Large Scale Production Environments
- Orchestrator Changes for Improved Performance
- Enable Redis
- Specify a Large Max Pool Size in the Connection String
- Write Robot Logs to Elasticsearch and Use AsyncWrapper
- Use a High Throughput Storage
- Disable Webhooks and Ledger
- Disable Maintenance Mode
- Increase the Caching Time of Application Settings
- Increase MinWorkerThreads Value
- Tune Quartz for Very High Throughput
- Disable IP to DNS Resolving in Audit
- Switch to a Faster Strategy for Fetching Groups Membership
- Adjust AD Domain Cache Expiration Time
- Disable SignalR for Robot License Change Events
- Enable NuGet Package Caching
- Enable Local Cache in Azure PaaS Deployments
- Configure Slot Warm-Up in Azure PaaS Deployments
- Use Azure SignalR Service in Azure PaaS Deployments
- Adjust the Flush Interval and Size for the SubmitLogs Endpoint
- Identity Server Changes for Improved Performance
- Specify a Large Max Pool Size in the Connection String
- Enable Redis
- UI Configuration Changes for Improved Performance