- Overview
- Requirements
- Recommended: Deployment templates
- Manual: Preparing the installation
- Manual: Preparing the installation
- Step 1: Configuring the OCI-compliant registry for offline installations
- Step 2: Configuring the external objectstore
- Step 3: Configuring High Availability Add-on
- Step 4: Configuring Microsoft SQL Server
- Step 5: Configuring the load balancer
- Step 6: Configuring the DNS
- Step 7: Configuring the disks
- Step 8: Configuring kernel and OS level settings
- Step 9: Configuring the node ports
- Step 10: Applying miscellaneous settings
- Step 12: Validating and installing the required RPM packages
- Step 13: Generating cluster_config.json
- Cluster_config.json Sample
- General configuration
- Profile configuration
- Certificate configuration
- Database configuration
- External Objectstore configuration
- Pre-signed URL configuration
- ArgoCD configuration
- External OCI-compliant registry configuration
- Disaster recovery: Active/Passive and Active/Active configurations
- High Availability Add-on configuration
- Orchestrator-specific configuration
- Insights-specific configuration
- Process Mining-specific configuration
- Document Understanding-specific configuration
- Automation Suite Robots-specific configuration
- AI Center-specific configuration
- Monitoring configuration
- Optional: Configuring the proxy server
- Optional: Enabling resilience to zonal failures in a multi-node HA-ready production cluster
- Optional: Passing custom resolv.conf
- Optional: Increasing fault tolerance
- Adding a dedicated agent node with GPU support
- Adding a dedicated agent Node for Task Mining
- Connecting Task Mining application
- Adding a Dedicated Agent Node for Automation Suite Robots
- Step 15: Configuring the temporary Docker registry for offline installations
- Step 16: Validating the prerequisites for the installation
- Manual: Performing the installation
- Post-installation
- Cluster administration
- Managing products
- Getting Started with the Cluster Administration portal
- Migrating objectstore from persistent volume to raw disks
- Migrating from in-cluster to external High Availability Add-on
- Migrating data between objectstores
- Migrating in-cluster objectstore to external objectstore
- Migrating to an external OCI-compliant registry
- Switching to the secondary cluster manually in an Active/Passive setup
- Disaster Recovery: Performing post-installation operations
- Converting an existing installation to multi-site setup
- Guidelines on upgrading an Active/Passive or Active/Active deployment
- Guidelines on backing up and restoring an Active/Passive or Active/Active deployment
- Monitoring and alerting
- Migration and upgrade
- Migrating between Automation Suite clusters
- Upgrading Automation Suite
- Downloading the installation packages and getting all the files on the first server node
- Retrieving the latest applied configuration from the cluster
- Updating the cluster configuration
- Configuring the OCI-compliant registry for offline installations
- Executing the upgrade
- Performing post-upgrade operations
- Applying a patch
- Product-specific configuration
- Best practices and maintenance
- Troubleshooting
- How to troubleshoot services during installation
- How to uninstall the cluster
- How to clean up offline artifacts to improve disk space
- How to clear Redis data
- How to enable Istio logging
- How to manually clean up logs
- How to clean up old logs stored in the sf-logs bucket
- How to disable streaming logs for AI Center
- How to debug failed Automation Suite installations
- How to delete images from the old installer after upgrade
- How to disable TX checksum offloading
- How to manually set the ArgoCD log level to Info
- How to expand AI Center storage
- How to generate the encoded pull_secret_value for external registries
- How to address weak ciphers in TLS 1.2
- How to check the TLS version
- Unable to run an offline installation on RHEL 8.4 OS
- Error in downloading the bundle
- Offline installation fails because of missing binary
- Certificate issue in offline installation
- SQL connection string validation error
- Prerequisite check for selinux iscsid module fails
- Azure disk not marked as SSD
- Failure after certificate update
- Antivirus causes installation issues
- Automation Suite not working after OS upgrade
- Automation Suite requires backlog_wait_time to be set to 0
- Volume unable to mount due to not being ready for workloads
- Support bundle log collection failure
- Single-node upgrade fails at the fabric stage
- Upgrade fails due to unhealthy Ceph
- RKE2 not getting started due to space issue
- Volume unable to mount and remains in attach/detach loop state
- Upgrade fails due to classic objects in the Orchestrator database
- Ceph cluster found in a degraded state after side-by-side upgrade
- Unhealthy Insights component causes the migration to fail
- Service upgrade fails for Apps
- In-place upgrade timeouts
- Docker registry migration stuck in PVC deletion stage
- AI Center provisioning failure after upgrading to 2023.10 or later
- Upgrade fails in offline environments
- SQL validation fails during upgrade
- snapshot-controller-crds pod in CrashLoopBackOff state after upgrade
- Setting a timeout interval for the management portals
- Authentication not working after migration
- Kinit: Cannot find KDC for realm <AD Domain> while getting initial credentials
- Kinit: Keytab contains no suitable keys for *** while getting initial credentials
- GSSAPI operation failed due to invalid status code
- Alarm received for failed Kerberos-tgt-update job
- SSPI provider: Server not found in Kerberos database
- Login failed for AD user due to disabled account
- ArgoCD login failed
- Update the underlying directory connections
- Failure to get the sandbox image
- Pods not showing in ArgoCD UI
- Redis probe failure
- RKE2 server fails to start
- Secret not found in UiPath namespace
- ArgoCD goes into progressing state after first installation
- MongoDB pods in CrashLoopBackOff or pending PVC provisioning after deletion
- Pods stuck in Init:0/X
- Missing Ceph-rook metrics from monitoring dashboards
- Running High Availability with Process Mining
- Process Mining ingestion failed when logged in using Kerberos
- After Disaster Recovery Dapr is not working properly for Process Mining and Task Mining
- Unable to connect to AutomationSuite_ProcessMining_Warehouse database using a pyodbc format connection string
- Airflow installation fails with sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string ''
- How to add an IP table rule to use SQL Server port 1433
- Automation Suite certificate is not trusted from the server where CData Sync is running
- Running the diagnostics tool
- Using the Automation Suite support bundle
- Exploring Logs
Managing the certificates
The installation process generates self-signed certificates on your behalf. These certificates are FIPS-compliant and will expire in 90 days. You must replace them with certificates signed by a trusted Certificate Authority (CA) as soon as installation completes. If you do not update the certificates, the installation will stop working in 90 days.
If you installed Automation Suite on a FIPS-enabled host and want to update the certificates, make sure they are FIPS-compliant.
The installation bundle provides a cluster management tool that enables you to update certificates post-installation. To access the tool, navigate to the location of the installer bundle:
cd /opt/UiPathAutomationSuite/
cd /opt/UiPathAutomationSuite/
To generate the CSR and the private key, run the following command:
# copy the machine openssl configuration locally
cp /etc/pki/tls/openssl.cnf ./openssl.tmp.cnf
# Replace the [AUTOMATION_SUITE_FQDN] value. For example, "automationsuite.corp.com"
AS_FQDN=[AUTOMATION_SUITE_FQDN]
cat >> ./openssl.tmp.cnf <<EOF
[SAN]
subjectAltName=DNS:$AS_FQDN,DNS:alm.$AS_FQDN,DNS:monitoring.$AS_FQDN,DNS:registry.$AS_FQDN,DNS:objectstore.$AS_FQDN,DNS:insights.$AS_FQDN
EOF
# create the certificate request
openssl req -new -sha256 -newkey rsa:2048 -nodes -keyout server.key -subj "/C=xx/ST=xx/O=xx/OU=xx/CN=$AS_FQDN" -reqexts SAN -config openssl.tmp.cnf -out ${AS_FQDN}.csr
# copy the machine openssl configuration locally
cp /etc/pki/tls/openssl.cnf ./openssl.tmp.cnf
# Replace the [AUTOMATION_SUITE_FQDN] value. For example, "automationsuite.corp.com"
AS_FQDN=[AUTOMATION_SUITE_FQDN]
cat >> ./openssl.tmp.cnf <<EOF
[SAN]
subjectAltName=DNS:$AS_FQDN,DNS:alm.$AS_FQDN,DNS:monitoring.$AS_FQDN,DNS:registry.$AS_FQDN,DNS:objectstore.$AS_FQDN,DNS:insights.$AS_FQDN
EOF
# create the certificate request
openssl req -new -sha256 -newkey rsa:2048 -nodes -keyout server.key -subj "/C=xx/ST=xx/O=xx/OU=xx/CN=$AS_FQDN" -reqexts SAN -config openssl.tmp.cnf -out ${AS_FQDN}.csr
Your IT team uses the obtained values to generate a signed certificate. The generated private key remains local.
To view more information about server certificates, run the following command:
./bin/uipathctl config tls-certificates --help
./bin/uipathctl config tls-certificates --help
Output:
************************************************************************************
Manage tls certificates
Usage:
uipathctl config tls-certificates [flags]
uipathctl config tls-certificates [command]
Available Commands:
get Get the current tls certificates
update Update tls certificates
Flags:
-h, --help help for tls-certificates
Global Flags:
--context string name of the kubeconfig context to use
--kubeconfig string kubectl configuration file (default: ~/.kube/config)
--log-format string log format. one of [text,json] (default "text")
--log-level string set log level. one of [trace,debug,info,error] (default "error")
-q, --quiet suppress all output except for errors and warnings
--timeout duration timeout of the command (default 1h0m0s)
************************************************************************************
************************************************************************************
Manage tls certificates
Usage:
uipathctl config tls-certificates [flags]
uipathctl config tls-certificates [command]
Available Commands:
get Get the current tls certificates
update Update tls certificates
Flags:
-h, --help help for tls-certificates
Global Flags:
--context string name of the kubeconfig context to use
--kubeconfig string kubectl configuration file (default: ~/.kube/config)
--log-format string log format. one of [text,json] (default "text")
--log-level string set log level. one of [trace,debug,info,error] (default "error")
-q, --quiet suppress all output except for errors and warnings
--timeout duration timeout of the command (default 1h0m0s)
************************************************************************************
uipathctl config tls-certificates
command.
Online installation: How to find the server certificate
istio-ingressgateway-certs
name in the istio-system
namespace.
See the certificate files in the following list:
- Server TLS certificate is stored as
tls.crt
- Server TLS private key as
tls.key
- CA bundle is stored as
ca.crt
You can verify the secrets using the following command :
kubectl -n istio-system get secrets istio-ingressgateway-certs -o yaml
kubectl -n istio-system get secrets istio-ingressgateway-certs -o yaml
Certificates are also stored in the UiPath namespace. This is applicable to every UiPath® product that needs certificate information to trust incoming calls. For details, see Understanding the container architecture related to certificates.
Offline installation: How to find the server certificate
rootCA.crt
and tls.crt
: ArgoCD and Docker Registry. Certificates are then stored in both Docker and ArgoCD namespaces.
You can verify the secrets using the following command :
# For docker registry
kubectl -n docker-registry get secrets docker-registry-tls -o yaml
# For Argocd
argocd login alm.cluster_fqnd --username argocd_username --password argocd_password
argocd cert list --cert-type https
# For docker registry
kubectl -n docker-registry get secrets docker-registry-tls -o yaml
# For Argocd
argocd login alm.cluster_fqnd --username argocd_username --password argocd_password
argocd cert list --cert-type https
How to update server certificates
To decrypt the certificate key, run the following command:
# replace /path/to/encrypted/cert/key to absolute file path of key
# replace /path/to/decrypt/cert/key to store decrypt key
# Once prompted, please entry the passphrase or password to decrypt the key
openssl rsa -in /path/to/encrypted/cert/key -out /path/to/decrypt/cert/key
# replace /path/to/encrypted/cert/key to absolute file path of key
# replace /path/to/decrypt/cert/key to store decrypt key
# Once prompted, please entry the passphrase or password to decrypt the key
openssl rsa -in /path/to/encrypted/cert/key -out /path/to/decrypt/cert/key
uipathctl
to update the certificate as shown below. You need the path to each of the three certificate files. All the certificate file
should be in PEM
format.
- Certificate Authority Bundle - This bundle should contain only the chain certificates used to sign the TLS server certificate. The chain limit is up to nine certificates.
-
Server Certificate - Public server certificate
Note:Theserver.crt
file must contain the entire chain, as shown in the following example:-----server cert----- -----root ca chain-----
-----server cert----- -----root ca chain----- -
Private key - Private key for server certificate
./bin/uipathctl config tls-certificates update --cert server.crt --cacert ca.crt --key server.key
./bin/uipathctl config tls-certificates update --cert server.crt --cacert ca.crt --key server.key
/directory/path/to/store/certificate
location.
To print out the certificate files, run the following command, specifying the directory where certificates are stored.
./bin/uipathctl config tls-certificates get --show-details
./bin/uipathctl config tls-certificates get --show-details
You are responsible for making sure the generated certificates are trusted.
To add the certificate to the host VM trust store, run the following commands on all the nodes in the cluster:
# 1. Copy the certificate file to the /usr/share/pki/ca-trust-source/anchors/ or the /etc/pki/ca-trust/source/anchors/ directory
cp /path/to/the/ca-cert /usr/share/pki/ca-trust-source/anchors/
# 2. Update the trust store configuration
update-ca-trust
# 1. Copy the certificate file to the /usr/share/pki/ca-trust-source/anchors/ or the /etc/pki/ca-trust/source/anchors/ directory
cp /path/to/the/ca-cert /usr/share/pki/ca-trust-source/anchors/
# 2. Update the trust store configuration
update-ca-trust
To view more information about additional CA certificates, run the following command:
./bin/uipathctl config additional-ca-certificates --help
./bin/uipathctl config additional-ca-certificates --help
Output:
***************************************************************************************
Manage additional ca certificates
Usage:
uipathctl config additional-ca-certificates [flags]
uipathctl config additional-ca-certificates [command]
Available Commands:
get Get the current additional ca certificates
update Update additional ca certificates
Flags:
-h, --help help for additional-ca-certificates
Global Flags:
--context string name of the kubeconfig context to use
--kubeconfig string kubectl configuration file (default: ~/.kube/config)
--log-format string log format. one of [text,json] (default "text")
--log-level string set log level. one of [trace,debug,info,error] (default "error")
-q, --quiet suppress all output except for errors and warnings
--timeout duration timeout of the command (default 1h0m0s)
***************************************************************************************
***************************************************************************************
Manage additional ca certificates
Usage:
uipathctl config additional-ca-certificates [flags]
uipathctl config additional-ca-certificates [command]
Available Commands:
get Get the current additional ca certificates
update Update additional ca certificates
Flags:
-h, --help help for additional-ca-certificates
Global Flags:
--context string name of the kubeconfig context to use
--kubeconfig string kubectl configuration file (default: ~/.kube/config)
--log-format string log format. one of [text,json] (default "text")
--log-level string set log level. one of [trace,debug,info,error] (default "error")
-q, --quiet suppress all output except for errors and warnings
--timeout duration timeout of the command (default 1h0m0s)
***************************************************************************************
uipathctl config additional-ca-certificates
command.
This command helps you update or replace the existing configured CA certificates.
./bin/uipathctl config additional-ca-certificates update --cacert additional_ca.crt
./bin/uipathctl config additional-ca-certificates update --cacert additional_ca.crt
--replace
at the end.
.pem
format and can have more than one certificate present in it.
To download the already configured CA certificates, run the following command:
./bin/uipathctl config additional-ca-certificates get
./bin/uipathctl config additional-ca-certificates get
You are responsible for making sure the generated certificates are trusted.
To add the certificate to the host VM trust store, run the following commands on all the nodes in the cluster:
# 1. Copy the certificate file to the /usr/share/pki/ca-trust-source/anchors/ or the /etc/pki/ca-trust/source/anchors/ directory
cp /path/to/the/ca-cert /usr/share/pki/ca-trust-source/anchors/
# 2. Update the trust store configuration
update-ca-trust
# 1. Copy the certificate file to the /usr/share/pki/ca-trust-source/anchors/ or the /etc/pki/ca-trust/source/anchors/ directory
cp /path/to/the/ca-cert /usr/share/pki/ca-trust-source/anchors/
# 2. Update the trust store configuration
update-ca-trust
Automation Suite offers two methods to manage the rotation of identity token-signing certificates: automatic and manual.
To view more information about identity token-signing certificates, run the following command:
./bin/uipathctl config token-signing-certificates --help
./bin/uipathctl config token-signing-certificates --help
Output:
************************************************************************************
Manage token signing certificates
Usage:
uipathctl config token-signing-certificates [flags]
uipathctl config token-signing-certificates [command]
Available Commands:
get Get the current token signing certificate
rotate Rotate token signing certificates
update Update future token signing certificate
Flags:
-h, --help help for token-signing-certificates
Global Flags:
--context string name of the kubeconfig context to use
--kubeconfig string kubectl configuration file (default: ~/.kube/config)
--log-format string log format. one of [text,json] (default "text")
--log-level string set log level. one of [trace,debug,info,error] (default "error")
-q, --quiet suppress all output except for errors and warnings
--timeout duration timeout of the command (default 1h0m0s)
************************************************************************************
************************************************************************************
Manage token signing certificates
Usage:
uipathctl config token-signing-certificates [flags]
uipathctl config token-signing-certificates [command]
Available Commands:
get Get the current token signing certificate
rotate Rotate token signing certificates
update Update future token signing certificate
Flags:
-h, --help help for token-signing-certificates
Global Flags:
--context string name of the kubeconfig context to use
--kubeconfig string kubectl configuration file (default: ~/.kube/config)
--log-format string log format. one of [text,json] (default "text")
--log-level string set log level. one of [trace,debug,info,error] (default "error")
-q, --quiet suppress all output except for errors and warnings
--timeout duration timeout of the command (default 1h0m0s)
************************************************************************************
You can use a maximum key length of 4096 bits for signing certificates. We highly recommend you to use a key length of at least 512 bits (64 bytes) as a best practice.
uipathctl config token-signing-certificates
command.
Automatic certificate rotation means Automation Suite manages the lifecycle of signing keys. This includes rotating keys every 90 days, announcing new keys 14 days prior to rotation, retaining old keys for 14 days post-rotation, and then deleting them when the 14-day period ends.
If you're upgrading from an older version to 2024.10, automatic certificate rotation is disabled by default. To enable automatic key management, use the following command:
uipathctl config token-signing-certificates automatic-key-management enable
uipathctl config token-signing-certificates automatic-key-management enable
Enabling automatic certificate rotation may result in a downtime of up to one hour.
Automatic certificate rotation is enabled by default for clean Automation Suite installations. To disable automatic key management, use the following command:
uipathctl config token-signing-certificates automatic-key-management disable
uipathctl config token-signing-certificates automatic-key-management disable
If the automatic management feature is disabled, signing certificates need to be updated and rotated manually. For details on manual key management, see the documentation on manually updating and rotating the certificate.
To upload the new certificate to sign the token, run the following command:
The following command does not replace the existing token signing certificate.
.pem
format.
server.crt
file must contain the entire chain, as shown in the
following example:
-----server cert-----
-----root ca chain-----
-----server cert-----
-----root ca chain-----
./bin/uipathctl config token-signing-certificates update --cert server.crt --key server.key
./bin/uipathctl config token-signing-certificates update --cert server.crt --key server.key
To rotate or replace the old certificate with the new one, run the following command:
./bin/uipathctl config token-signing-certificates rotate
./bin/uipathctl config token-signing-certificates rotate
There should be the lead time of about 24 - 48 hours between certificate update and rotate.
We need this lead time to keep supporting the authentication for cached token signed by old certificate.
If you rotate the certificate too soon before the expiry of cache token can result in downtime. And you may have to restart all your robots.
To perform an emergency certificate update, take the following steps:
By default, RKE2 certificates expire in 12 months. In the 90 days prior to their expiration date, certificates are rotated when you restart RKE2.
For more details, refer to RKE2 - Advanced Options - Certificate rotation.
# Directory path to check
if [[ -d "/var/lib/rancher/rke2/server/tls" ]];
then
dir="/var/lib/rancher/rke2/server/tls"
else
dir="/var/lib/rancher/rke2/agent/tls"
fi
# Loop through each .crt file in the directory
for file in "$dir"/*.crt;
# Extract the expiry date from the certificate
do
expiry=$(openssl x509 -enddate -noout -in "$file" | cut -d= -f 2-)
# Get the file name without the path
filename=$(basename "$file")
# Print the filename and expiry date in a pretty format
printf "%-30s %s\n" "$filename:" "$expiry"
done
# Directory path to check
if [[ -d "/var/lib/rancher/rke2/server/tls" ]];
then
dir="/var/lib/rancher/rke2/server/tls"
else
dir="/var/lib/rancher/rke2/agent/tls"
fi
# Loop through each .crt file in the directory
for file in "$dir"/*.crt;
# Extract the expiry date from the certificate
do
expiry=$(openssl x509 -enddate -noout -in "$file" | cut -d= -f 2-)
# Get the file name without the path
filename=$(basename "$file")
# Print the filename and expiry date in a pretty format
printf "%-30s %s\n" "$filename:" "$expiry"
done
The output you get should be similar to the one shown in the following image:
To rotate the RKE2 certificates, you must first execute a series of action on the server nodes, then proceed with some steps on the agent nodes.
- Stop the RKE2 server:
systemctl stop rke2-server.service
systemctl stop rke2-server.service - Clear any remaining RKE2 processes:
rke2-killall.sh
rke2-killall.sh - Delete the
dynamic-cert.json
file located at/var/lib/rancher/rke2/server/tls/
. - Restart the RKE2 server:
systemctl start rke2-server.service
systemctl start rke2-server.serviceNote: If the cluster has more than one server node, steps 1-4 might not wholly execute as etcd may be unable to complete the leader election. Should this happen, repeat steps from 1-4 on other server nodes. - Delete the
rke2-serving
secret from thekube-system
namespace:kubectl delete secret -n kube-system rke2-serving
kubectl delete secret -n kube-system rke2-servingNote:In a multi-node deployment, you may not be able to run thekubectl
commands until you have completed the first four operations on the necessary number of server nodes. This is to fulfill the etcd quorum requirement. You can remove therke2-serving
secret immediately after the RKE2 server starts up.
kubectl get nodes
command. When your server nodes are ready, you can proceed to the agent nodes to regenerate the certificates.
Take the following steps on the agent nodes:
- Stop the RKE2 server:
systemctl stop rke2-agent.service
systemctl stop rke2-agent.service - Clear any remaining RKE2 processes:
rke2-killall.sh
rke2-killall.sh - Restart the RKE2 server:
systemctl start rke2-agent.service
systemctl start rke2-agent.service
- Generating a Certificate Signing Request (CSR) and a private key
- Managing server certificates
- Updating the server certificate
- Accessing the TLS certificate
- Adding the CA certificate to the host trust store
- Managing additional CA certificates
- Updating the CA certificates
- Accessing the CA certificates
- Adding the CA certificate to the host trust store
- Managing identity token-signing certificates
- Automatic certificate rotation
- Manually updating the certificate
- Manually rotating the certificate
- Emergency certificate rotation
- Accessing the certificate
- Managing RKE2 certificates
- Checking the RKE2 certificate expiration date
- Rotating the RKE2 certificate
- Managing the external OCI-compliant registry certificate