• What is this
• How it works
• Hardware requirements
• Network communication
• Firewall requirements
• Software requirements
• DNS requirements
• SSL requirements
• Manual installation
• Manual upgrade
• SMTP settings
• Troubleshooting

Endpoint Manager by ITarian (EM) for stand alone installation. Basic features described on official site https://dm.comodo.com/

Os types support:

• Windows
• OS X
• Linux
• iOS
• Android

Features

• Device management
• Remote packages installation
• Antivirus protection
• Management groups and users
• Management configuration templates
• File logs / activity / verdict management
• Remote tools (file / process explorer)
• Remote control
• Remote procedure and monitoring
• Wipe Apple and Android devices
• Update CCC and CCS agents from internal cache in order to optimize internet bandwidth and accelerate updates in large networks.

ITarian on-premise solution is distributed as set of docker images and can be run with docker. Therefore it is main requirement to have operation system with support and pre-installed docker

To make it works together for numerous docker-container it is used docker-compose tool

It allows easily setup all components together and maintain already run environment

All required action from system administrator of on-premise solution is:

• Install docker
• Install docker-compose
• Setup configuration (basic domain name)
• Setup certificates (for domain above)
• Run all together with single docker-compose command

Scheme of docker-compose usage:

Docker-compose supports stand alone configuration for single server. It is applicable for limited count of endpoints (up to 1000).
For scalable solution to manage multiple servers and horizontally scalable and reliable system need to use more complicated docker management system like a Kubernetes

At the moment support of kubernetes for on-premise solution is under the development and not supported

Minimum 2 servers for docker-compose configuration

Hardware requirements / recommendations (1000 endpoints)
By statistics: 1 endpoint produces 0.015 requests per second
It means that we can handle about 65 sequential requests from different endpoints per seconds.
Server could handle 50-100 connections simultaneously.
Therefore average endpoint count that can be handled is 50 * 65 > 3000.

ITSM-server
8 Cores Cpu | 8 Gb Ram | 100 Gb Hdd

Xmpp-server
4 Cores Cpu | 4 Gb Ram | 40 Gb Hdd

On-premise installation consists of multiple services and components which are communicated between themselves via network

Public listen ports:

ITSM Server

• 80 HTTP - web port (by default redirect to https 443 port) used only for making available default browser connection without https
• 443 HTTP(S) - common port which handle all incoming connections with TLS encryption

Xmpp Server

• 443 TCP - secured tcp connection for endpoints and remote control tools
• 5222 TCP - default xmpp port with the same purposes but not used. Might be used as a fallback option for 443
• 8080 HTTP - service port for sending push messages. It used only by ITSM server and could be closed for external connections

Turn server

• 49152 - 65535 UDP - range of ports dynamically allocated for remote control connections to endpoints which located behind the NAT

Private network

Besides public ports most services expose specific ports to internal network which is closed to external world. These ports could be exposed just for debug purposes, but by default all service ports are closed including databases, message brokers and microservices which are the part of all system.

ITSM system designed for most restricted environments with corporate networks which has almost fully closed network

Therefore it only expose 443 port as main secure channel

Port 80 used only for convenient redirects as most popular default web port for each domain

For these purposes it was discovered also 443 port as default for xmpp connections to XMPP-server, but with difference to use TCP traffic instead of HTTP

Summarize it we need to have open 443 port on firewall as a minimum requirement. Port 80 also is recommended to open as it might confusing users and reduce usability

General tests held on Ubuntu Desktop and Ubuntu Server (Ubuntu 16.04.4 LTS).
Other versions of ubuntu not tested, but should work starting from Ubuntu 14 (maybe even Ubuntu 12)
Another versions of linux also should work (Debian, CentOS, etc). Only difference is how to install docker. All scripts below prepared and tested on Ubuntu 16.
For docker-compose configuration it doesn't matter which hostname is specified for each server.

Itsm requires several domain names which should be resolved by different components.
Expected that there are exists domain name and it could be resolved everywhere. Otherwise we have to specify each required subdomain on every endpoint according to infrastructure.
Basic dns domain should be set by customer. But there are few requirements for existing domains / subdomains.
List of required domains:

• Base domain
• Itsm-domain
• Xmpp-domain
• Rmm-domain
• Patch-Management-domain
• Audit-log-domain
• Download-domain
• RealtimeDeviceCommunication-API-domain
• RealtimeDeviceCommunication-Relay-domain
• BulkInstallationPackage-domain

Base domain is just a pointer for all another subdomains.

Small example:

                ITSM-server IP    10.0.5.1
                XMPP-server IP  10.0.5.2
Assume we have itsm domain on-prem.company.local on IP 10.0.5.1 (itsm-server).
It means that base domain is company.local (doesn't matter which ip it has. This entry not used in the system).
Next subdomains must be related to base domain.
Rmm-domain - rmm-api.company.local (IP 10.0.5.1 same as itsm-server)
Patch-Management-domain - plugins-api.company.local (IP 10.0.5.1 same as itsm-server)
Audit-log-domain - auditlogs-api.company.local (IP 10.0.5.1 same as itsm-server)
Download-domain - dl.company.local (IP 10.0.5.1 same as itsm-server)
Xmpp-domain - xmpp.company.local (IP 10.0.5.2 XMPP-server)
RealtimeDeviceCommunication-API-domain - rtdc-api.company.local (IP 10.0.5.1 same as itsm-server)
RealtimeDeviceCommunication-Relay-domain - rtdc-relay-01.company.local (IP 10.0.5.1 same as itsm-server)
BulkInstallationPackage-domain - bip.company.local (IP 10.0.5.1 same as itsm-server)

Required subdomain list which should be resolved:

From ITSM-server

• xmpp (to XMPP-server)

From XMPP-server

• <itsm> - customer specified ITSM_DOMAIN (to itsm-server)

From administrator endpoint (web access)

• <itsm> - customer specified ITSM_DOMAIN (to itsm-server)
• rtdc-api (to itsm-server)
• rtdc-relay-01 (to itsm-server)

From enrolled device endpoint

• <itsm> - customer specified ITSM_DOMAIN (to itsm-server)
• bip - bulk installation package download host (to itsm-server)
• rmm-api - rmm logs reporting (to itsm-server)
• auditlogs-api - audit logs reporting (to itsm-server)
• rtdc-api - remote tools configuration (to itsm-server)
• rtdc-relay-01 - remote tools download file (to itsm-server)
• plugins - alerts, patch management, software inventory logs reporting (to itsm-server)
• xmpp - persistent connection for receiving push messages and remote control commands (to XMPP-server)

From remote control tool

• <itsm> - customer specified ITSM_DOMAIN (to itsm-server)
• dl - check and download updates (to itsm-server)
• xmpp - remote control communication (to XMPP-server)

Scheme example:

We recommend to have wildcard certificate. You may to have certificates for each subdomain specified above.
Currently required to have wildcard ssl certificate for itsm-server *.company.local and the same certificate or specific one for XMPP xmpp.company.local.
If you don’t have real domain and trusted SSL certificate you can generate self-signed certificates by yourself.
Note: endpoints couldn’t work with self-signed certificate on XMPP. In this case you have never get green online status on itsm-server for endpoints also remote control will not work.

For minimal configuration it is required to have set of certificates and keys for each server (itsm, xmpp)

Setup SSL certificates for ITSM

Place valid ssl certificate and key into /opt/itsm/web/certs under the names cert.crt and cert.key.

Note: private key must be without passphrase as web server could not work with those

# create directory
sudo mkdir -p /opt/itsm/web/certs

# copy prepared certificate and key to destination
cp /path/to/your/certificate.crt /opt/itsm/web/certs/cert.crt
cp /path/to/your/certificate.key /opt/itsm/web/certs/cert.key

If you don’t have valid certificates. It is possible to issue self-signed certificate key-pair. But in this case you need to allow unsecured access in the browser and some features will be dropped. Next commands create self-signed certificates:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /opt/itsm/web/certs/cert.key -out /opt/itsm/web/certs/cert.crt

NOTE: your certificate CN (common name) domain should be the same as ITSM_DOMAIN variable value specified in .env

Setup SSL certificates for XMPP

Place valid pem certificate for domain specified in XMPP_DOMAIN into /opt/ejabberd/certs

Filename should be in following pattern {XMPP_DOMAIN}.pem

NOTE:

• certificate name should be exactly as XMPP_DOMAIN value specified in .env file.
  For example above certificate filename should be yourdomain.com.pem without prefix “xmpp.” .
• Certificate bundle must contain root CA certificate. For creation valid certificate need to concatenate private.key + certificate.crt + chain.crt + root.crt

sudo mkdir -p /opt/ejabberd/certs
cat cert.key cert.crt chain.crt root.crt > your.domain.pem
sudo mv your.domain.pem /opt/ejabberd/certs/

1. Prerequisites: install docker and docker-compose

   1. Login to remote server
      ssh username@ip-or-hostname

   2. Get installation script (for ubuntu)
      wget http://get-compose.on-premise.itarian.com/install-docker-compose.sh

   3. Make file executable
      chmod +x install-docker-compose.sh

   4. Run script
      sudo ./install-docker-compose.sh

   5. Setup local user permissions
      sudo usermod -a -G docker $USER

   6. Logout from current session and login again to apply local user group changes
      exit
ssh username@ip-or-hostname

   7. Perform docker login
      Note: your must have created account on CAM Comodo Accounts Management and order AEP license
      docker login registry.on-premise.itarian.com
      Enter CAM login and CAM password
      

2. Extra server setup

   Only for itsm server it need to tune system settings:
   sudo sysctl -w vm.max_map_count=262144
echo vm.max_map_count=262144 | sudo tee -a /etc/sysctl.conf

3. Logging Configurations

   In order to apply log rotation and limiting the size of logs add below command under /etc/logrotate.conf:
   

   /var/lib/docker/containers/*/*.log {
     rotate 5
     copytruncate
     missingok
     notifempty
     compress
     maxsize 200M
     daily
   }

   Or create file /etc/docker/daemon.json with below commands:
   

   {
     "log-driver": "json-file",
     "log-opts": {
       "max-size": "200m",
       "max-file": "5",
       "compress": "true"
     }
   }

   Above configurations will rotate logs large than 200m and compress rotated logs. The system will keep original and 4 rotated log files. Configurations can be changed according to needs.
   Note: For existing installations, second option requires on-premise server to be upgraded to latest version manually.

4. Get docker-compose.yml and configure settings

   1. Create and navigate to itsm dir
      mkdir ~/itsm
cd ~/itsm

   2. Get docker-compose.yml for specific server
      for itsm server
      wget https://get-compose.on-premise.itarian.com/version/release/registry-docker-compose.yml -O docker-compose.yml
      for XMPP-server
      wget https://get-compose.on-premise.itarian.com/version/release/registry-ejabberd-docker-compose.yml -O docker-compose.yml

   3. create file with name .env and fill it according to your server requirements

      For itsm server

      ITSM_DOMAIN=on-premise.itsm.local
ITSM_TURN_SERVERS=ip of turn server
ITSM_XMPP_HOST=xmpp.itsm.local
ITSM_XMPP_IP=ip of xmpp server
ITSM_WEB_HOST=same as ITSM_DOMAIN on-premise.itsm.local
ITSM_WEB_IP=ip of this host

      Where:
      ITSM_DOMAIN - domain name which must be the same as the certificate domain used in setup
      ITSM_TURN_SERVERS - list of ips where turn server is running separated by comma or space ( if turn servers has been setup )
      ITSM_XMPP_HOST - domain for xmpp server ( if xmpp has been setup )
      ITSM_XMPP_IP - ip for host specified in ITSM_XMPP_HOST if dns record cannot be resolved ( if xmpp has been setup without dns )
      ITSM_WEB_HOST - domain name which used by rmm microservices and points to itsm-server.
      ITSM_WEB_IP - ip for host specified in ITSM_WEB_HOST if dns record cannot be resolved
      

      For XMPP-server

      XMPP_DOMAIN=itsm.local
ITSM_WEB_HOST=on-premise.itsm.local
ITSM_WEB_IP=ip of itsm server

5. Prepare and save SSL certificates

   1. Prepare ASCII encoded certificate files:

      • cert.crt - wildcard certificate for domain which specified as ITSM_DOMAIN
      • cert.key - private key without password for cert.crt
      • chain.crt - chaining certificate for cert.crt
      • root.crt - CA root certificate for chain

   2. Compose and save certificate files

      For itsm server

      Save cert.crt and cert.key

      sudo mkdir -p /opt/itsm/web/certs
cat cert.crt chain.crt | sudo tee /opt/itsm/web/certs/cert.crt
sudo cp cert.key /opt/itsm/web/certs/cert.key

      For XMPP-server

      Save single certificate bundle under the name specified in XMPP_DOMAIN and .pem extension

      sudo mkdir -p /opt/ejabberd/certs
cat cert.key cert.crt chain.crt root.crt | sudo tee /opt/ejabberd/certs/itsm.local.pem

6. Starting up and basic management

   • Starting system
     cd ~/itsm
docker-compose up -d

   • Stopping system
     docker-compose down

   • View logs / debugging
     docker-compose logs

To perform upgrade it need to go on next steps:

1. Go to itsm directory which contains docker-compose.yml and .env files
   cd ~/itsm

2. Stop the system
   docker-compose down

3. Get latest docker-compose.yml file for regarding to your server type
   for itsm server
   wget https://get-compose.on-premise.itarian.com/version/release/registry-docker-compose-with-turn.yml -O docker-compose.yml
   for XMPP-server
   wget https://get-compose.on-premise.itarian.com/version/release/registry-ejabberd-docker-compose.yml -O docker-compose.yml

4. Run the system again
   docker-compose up -d

After complete installation need to setup SMTP to have ability to receive email from ITSM server

1. Login to portal as admin
2. Open link https://itsm.yourcompany.com/email/settings where itsm.yourcompany.com is your ITSM server domain specified during installation
3. Click edit button
4. Fill all fields according to your corporate smtp settings and save it

• Problem: ITSM server not responding

  Suggestion: Check docker logs for service "web"

  Command
  docker-compose logs -f web --tail 20
  Output should not have constantly appended rows like
  
  nginx: [emerg] host not found in upstream "service_name" in /etc/nginx/conf.d/website.conf:277
  
  In case you find those rows, it mean that your service_name is not started
  Check container logs with that service_name
  docker-compose logs -f --tail 20 service_name
  and check all containers that have not status Running or Exit 0

• Problem: any docker command (example: docker ps) not works from your current user (Error like permission denied).

  Suggestion: Check your current user belongs to group docker

  Command
  groups
  Should contains docker

  If user not belongs to group docker run this command
  sudo usermod -a -G docker $USER

  After that need to logout and login again with current user

• Problem: any docker command (example: docker ps) not works. (Error like docker daemon is not running).

  Suggestion: Check docker daemon status

  Command
  sudo service docker status
  Output should have Active: active (running)

  If output is Active: inactive (dead) Start docker daemon with command
  sudo service docker start

• Problem: at least one container from list docker-compose ps has state Exit 1

  Suggestion: Try to start this container.

  Command
  docker start <container name from column Name>
  If problem persist - need to view container logs Command
  docker logs <container name from column Name> --tail 100
  Output need to send to ITSM team for analyzing.

• Problem: server data consumes lot of disk space (/var/lib/docker/overlay2)

  Suggestion: Try to cleanup old docker images.
  Every on-premise update pull several new images and keep both old and new ones

  Command
  docker image prune -af

• Problem: server data consumes lot of disk space (/opt/itsm/elk)

  Suggestion: Try to cleanup old elasticsearch indexes
  

  On-premise provided with built-in elasticsearch management system kibana
  Follow kibana setup guide to setup it
  Remove indexes with kibana GUI

• Problem: EM CCS clients updates consume lot of internet traffic

  Suggestion: Setup local antivirus database updates
  On-premise provided with av-db offline package
  Follow antivirus local updates guide to setup CCS update preferences