- Release Notes
- Requirements
- Installation
- Getting Started
- Projects
- Datasets
- ML Packages
- Pipelines
- ML Skills
- ML Logs
- Document Understanding in AI Fabric
- Basic Troubleshooting Guide
5. Run the AI Fabric Application Installer
Run the application installer. This will install AI Fabric and Out of the Box models in case of airgapped installations.
<machine-ip>:8800
. This is where the configuration for the
application installer will take place. On navigating to that address, the below page is
displayed.
Click Continue to Setup. The below window is displayed.
If the Linux machine has DNS configured and you with to use your own certificate when accessing AI Fabric app based on a fully qualified domain name, this configuration screen will allow you to do so.
Enter the hostname and upload both the private key and the SSL certificate. Make sure that the domain is reachable from the network where you want to access AI Fabric over the ports 8800, 31443, 31390. In the case of domain certificates, please make sure the certificate bundle file includes all the chain elements in this specific order: root, intermediate and server certificate. Also, if the public key certificate was not issued with a public RSA key of at least 2048-bit size, the validation will fail.
.pem
file, and the private key is a
.key
file. The files can be obtained from the .pfx
certificate.
Logging in to the Admin Console
After configuring the DNS in the step above (or clicking **Skip & Continue" to skip that step), the following page is displayed:
Enter your password from Step 4: Run the AI Fabric Infrastructure Installer. After successfully logging in, you will be prompted to upload a license file:
Upload the yaml file that has your AI Fabric license (contact your AI Fabric representative if you do not have an AI Fabric license).
If you are following the airgapped installation, in this step you must upload the installation bundle.
Click on Choose a bundle to upload button and select aif_services file extracted on previous step.
Next step is to configure installer using following screen:
The fields in this page need to be completed. Please refer to the explanations below
- Host Enter the IP address of the SQL database where you created the databases in Step 2. SQL Configuration.
- Username Enter the username for the SQL database owner
- Password Enter the
password for the SQL database.
Note: If you are using Azure databases please make sure to use FQDN to connect to the databases. It can be public or private domain name (as long as DNS is configured) but it won't work if you are using the private IP is documented here: https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-dns
<domain-name>:443
.
See the examples below for pitfalls to avoid.
Orchestrator endpoint entered | Correct |
---|---|
aifabric.orchestrator.cloudapp.azure.com:443 | ✓ |
https://aifabric.orchestrator.cloudapp.azure.com | ✗ |
https://23.96.154:443 | ✗ |
23.96.154:443 | ✗ |
This is only necessary if UiPath Identity Server is different from Orchestrator Endpoint.
<orchestrator-address>/identity
in a browser. Make sure
you are logged in to the host tenant (as opposed to the "default" tenant). The below
page is displayed:
If you see a page that does not have the left navigation as above, you are likely logged in to the default tenant. Make sure you log in as the host tenant.
Now click Installation Access Token, click Generate Token and use the two card icon to copy this token to your clipboard.
Paste this token into the field Identity access token.
Click Continue in the configuration page. You will be navigated to a page with title Preflight checks. If all your preflight checks pass, you will see a page like this:
Total
Memory
. Conservatively, the machine should have at least 52GB
RAM
, to see how provisioning less memory limits the capacities of AI
Fabric see the hardware requirements page.
If you do not see a green checkmark on the preflight checks use this guide to fix your errors.
Preflight Check | |
---|---|
Orchestrator Check | Verifies that:
|
Identity Server Check | Verifies that we can connect to identity server on
path /identity.
Solution: Make sure the network rules are set so the linux machine can connect to Identity Server (if different from Orchestrator). |
Identity Server Access Token Validation | Verifies that the access token is valid.
Solution: If you pass the Identity Server Check, make sure that you have a fresh token as the token validity is 1 hour. |
Orchestrator and Identity Server checks fail AND you do not have an external DNS |
If you do not have an external DNS through which the Orchestrator domain name or the Identity Server domain name can be resolved, you need an extra application of a configuration file. See Advanced Troubleshooting - DNS Resolution |
DB Checks | Verifies that:
|
All checks below Disk Space
fail.
| If you have verified the network/firewall rules for
Orchestrator and SQL Server but are still seeing all the checks
related to connections fail it may be due to a more nuanced network
configuration issue called IP Masquerading. This is usually
caused when the linux machine and Orchestrator/SQL Server are on
different subnets. If this happens, run:
On the linux machine, and retry the preflight checks. |
If you are unable to resolve preflight check failure please contact support and send them a Support Bundle. See Advanced Troubleshooting - Support Bundle for instructions on how to create one.
Continue
. You will
see the following page:
This page means that the AI Fabric application installer has started. If all configurations where set correctly, this should execute within 20-30 mins. See below to output the log that will show whether the installation was successful.
At a high-level troubleshooting the application install by yourself (note you can also just send UiPath technical support your Support Bundle, see below) involves these steps:
- Looking at the provision log
to determine what happened. Run
kubectl logs -f provision-*
to see the latest provision log. - Fix/Edit your configuration after you learn of the error from the logs.
- Re-trigger a provision by
saving the configuration, clicking
Go to new new version
and then clickingDeploy
To see running logs, you can use the kubernetes commadline interface since the application installer runs on top of kubernetes.
bash -l
.
Every time a configuration is changed and deployed (for the first configuration the deploy is done automatically, for subsequent one you must save the configuration and click Deploy), a new job will be executed to install the application.
To see the running logs on the linux machine run:
bash -l
kubectl get pods
bash -l
kubectl get pods
you will see something like this:
aif-admin@aifabric-onprem-int0:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
...
...
provision-rmvfg 0/1 Running 0 1m
aif-admin@aifabric-onprem-int0:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
...
...
provision-rmvfg 0/1 Running 0 1m
The output shows a name formatted like provision-<identifier>. To see the running application installer do:
kubectl logs -f provision-rmvfg
kubectl logs -f provision-rmvfg
kubectl logs -f provision
and hit TAB,
this will autocomplete the identifier).
This will show running logs (if the process has not ended) as well as the logs when the process has succeeded or terminated. Most, if not all users can troubleshoot without having to use anything other than the command above.
Successful Install
A successful install will show the screenshot below 15-20 minutes after passing the preflight checks:
If you see this screen, you can proceed to the step 6. Verify Installation .
Known Issues
Contingent on triggering an installation only after you have passed the preflight checks, there is only one known issue that can happen in the AI Fabric installer. If this is the case, you will see the following output from the log:
...
Starting ai-helper deployment ...Release "ai-helper" does not exist.
Installing it now.
Error: etcdserver: request timed outHelm installation failed for ai-helper in namespace aifabric.
Exiting !!!onebox provisioning failed.
Exiting !!!
...
Starting ai-helper deployment ...Release "ai-helper" does not exist.
Installing it now.
Error: etcdserver: request timed outHelm installation failed for ai-helper in namespace aifabric.
Exiting !!!onebox provisioning failed.
Exiting !!!
<ip>:8800
, clicking on
Config
, pasting a new Identity server token, hitting
Continue to new version
, and finally clicking
Deploy
.
Error: etcdserver: request timed out
) and retrying does
not resolve the issue please contact support and create a support bundle.
For airgapped installation you need to manually download OOB models and then "install" them on your AI Fabric machine to be able to use them. For each model that you want to add you will have a tar file that you need to move to your AI Fabric machine. Once on AI Fabric machine just run following commands for each file:
tar -zxvf formextractor-1.tar.gz
cd formextractor
nohup sudo ./setup.sh > formextractor.out 2>&1
tar -zxvf formextractor-1.tar.gz
cd formextractor
nohup sudo ./setup.sh > formextractor.out 2>&1
Logs will be accessible in formextract.out file. We recommend using nohup for this command because installation can take up to an hour and this would avoid any issue due to losing connection to the machine.
<machine-ip>:8800
) and click on Troubleshoot on
the top navigation bar. Click the button to generate a new support bundle, and then
download that bundle. When you contact support, include in the ticket that file
(support-bundle.tar.gz
).
As mentioned above, this issue will surface during preflight checks. As a reminder, this issue occurs when there is no external DNS that will resolve the Orchestrator domain or the Identity Server domain.
/etc/hosts
),
we will need to edit cluster's configmap so the cluster is aware of this DNS. To do
do so you need to run the following command:
kubectl -n kube-system edit cm coredns
kubectl -n kube-system edit cm coredns
It will open a vi editor with file looking as follow
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
}
kind: ConfigMap
metadata:
creationTimestamp: "2020-11-30T12:25:28Z"
name: coredns
namespace: kube-system
resourceVersion: "17667708"
selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
uid: 2bde7049-eda6-46eb-b523-beb8c421085f
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
}
kind: ConfigMap
metadata:
creationTimestamp: "2020-11-30T12:25:28Z"
name: coredns
namespace: kube-system
resourceVersion: "17667708"
selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
uid: 2bde7049-eda6-46eb-b523-beb8c421085f
A new section hosts needs to be added after loadbalance in Corefile section (line 19), you can list there as many dns you want to and add fallthrough at the end:
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
hosts example.hosts orchestrator-dns.com {
1.2.3.4 example.hosts
5.6.7.8 orchestrator-dns.com
fallthrough
}
}
kind: ConfigMap
metadata:
creationTimestamp: "2020-11-30T12:25:28Z"
name: coredns
namespace: kube-system
resourceVersion: "17667708"
selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
uid: 2bde7049-eda6-46eb-b523-beb8c421085f
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
hosts example.hosts orchestrator-dns.com {
1.2.3.4 example.hosts
5.6.7.8 orchestrator-dns.com
fallthrough
}
}
kind: ConfigMap
metadata:
creationTimestamp: "2020-11-30T12:25:28Z"
name: coredns
namespace: kube-system
resourceVersion: "17667708"
selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
uid: 2bde7049-eda6-46eb-b523-beb8c421085f
After that the DNS will be configured and working inside your cluster.
- Accessing admin console
- Configure DNS (optional)
- Choose installation type
- Airgapped Installation
- Online Installation
- Configure the installer
- Ingress
- Single Database Vs Multiple Databases
- SQL
- Orchestrator
- Identity Server
- Identity Server Access Token
- Preflight checks
- Troubleshooting Preflight Checks
- Running the application installer
- Troubleshooting the Application Installer
- Running Logs
- Install OOB models (airgapped only)
- Advanced troubleshooting
- Creating a Support Bundle
- DNS Resolution