Version: 1.1.5

What is this

Endpoint Manager by ITarian (EM) for stand alone installation. Basic features described on official site https://dm.comodo.com/

Os types support:
  • Windows
  • OS X
  • Linux
  • iOS
  • Android
Features
  • Device management
  • Remote packages installation
  • Antivirus protection
  • Management groups and users
  • Management configuration templates
  • File logs / activity / verdict management
  • Remote tools (file / process explorer)
  • Remote control
  • Remote procedure and monitoring
  • Wipe Apple and Android devices
  • Update CCC and CCS agents from internal cache in order to optimize internet bandwidth and accelerate updates in large networks.

How it works

ITarian on-premise solution is distributed as set of docker images and can be run with docker. Therefore it is main requirement to have operation system with support and pre-installed docker

To make it works together for numerous docker-container it is used docker-compose tool

It allows easily setup all components together and maintain already run environment

All required action from system administrator of on-premise solution is:

  • Install docker
  • Install docker-compose
  • Setup configuration (basic domain name)
  • Setup certificates (for domain above)
  • Run all together with single docker-compose command

Scheme of docker-compose usage:
Docker compose topology

Docker-compose supports stand alone configuration for single server. It is applicable for limited count of endpoints (up to 1000).
For scalable solution to manage multiple servers and horizontally scalable and reliable system need to use more complicated docker management system like a Kubernetes

At the moment support of kubernetes for on-premise solution is under the development and not supported

Hardware requirements

Minimum 2 servers for docker-compose configuration

Hardware requirements / recommendations (1000 endpoints)
By statistics: 1 endpoint produces 0.015 requests per second
It means that we can handle about 65 sequential requests from different endpoints per seconds.
Server could handle 50-100 connections simultaneously.
Therefore average endpoint count that can be handled is 50 * 65 > 3000.

ITSM-server
8 Cores Cpu | 8 Gb Ram | 100 Gb Hdd

Xmpp-server
4 Cores Cpu | 4 Gb Ram | 40 Gb Hdd

Network communications

On-premise installation consists of multiple services and components which are communicated between themselves via network

Public listen ports:

ITSM Server
  • 80 HTTP - web port (by default redirect to https 443 port) used only for making available default browser connection without https
  • 443 HTTP(S) - common port which handle all incoming connections with TLS encryption
Xmpp Server
  • 443 TCP - secured tcp connection for endpoints and remote control tools
  • 5222 TCP - default xmpp port with the same purposes but not used. Might be used as a fallback option for 443
  • 8080 HTTP - service port for sending push messages. It used only by ITSM server and could be closed for external connections
Turn server
  • 49152 - 65535 UDP - range of ports dynamically allocated for remote control connections to endpoints which located behind the NAT

Private network

Besides public ports most services expose specific ports to internal network which is closed to external world. These ports could be exposed just for debug purposes, but by default all service ports are closed including databases, message brokers and microservices which are the part of all system.

Firewall requirements

ITSM system designed for most restricted environments with corporate networks which has almost fully closed network

Therefore it only expose 443 port as main secure channel

Port 80 used only for convenient redirects as most popular default web port for each domain

For these purposes it was discovered also 443 port as default for xmpp connections to XMPP-server, but with difference to use TCP traffic instead of HTTP

Summarize it we need to have open 443 port on firewall as a minimum requirement. Port 80 also is recommended to open as it might confusing users and reduce usability

Software requirements

General tests held on Ubuntu Desktop and Ubuntu Server (Ubuntu 16.04.4 LTS).
Other versions of ubuntu not tested, but should work starting from Ubuntu 14 (maybe even Ubuntu 12)
Another versions of linux also should work (Debian, CentOS, etc). Only difference is how to install docker. All scripts below prepared and tested on Ubuntu 16.
For docker-compose configuration it doesn't matter which hostname is specified for each server.

DNS requirements

Itsm requires several domain names which should be resolved by different components.
Expected that there are exists domain name and it could be resolved everywhere. Otherwise we have to specify each required subdomain on every endpoint according to infrastructure.
Basic dns domain should be set by customer. But there are few requirements for existing domains / subdomains.
List of required domains:

  • Base domain
  • Itsm-domain
  • Xmpp-domain
  • Rmm-domain
  • Patch-Management-domain
  • Audit-log-domain
  • Download-domain
  • RealtimeDeviceCommunication-API-domain
  • RealtimeDeviceCommunication-Relay-domain
  • BulkInstallationPackage-domain

Base domain is just a pointer for all another subdomains.

Small example:
                ITSM-server IP    10.0.5.1
                XMPP-server IP  10.0.5.2
Assume we have itsm domain on-prem.company.local on IP 10.0.5.1 (itsm-server).
It means that base domain is company.local (doesn't matter which ip it has. This entry not used in the system).
Next subdomains must be related to base domain.
Rmm-domain - rmm-api.company.local (IP 10.0.5.1 same as itsm-server)
Patch-Management-domain - plugins-api.company.local (IP 10.0.5.1 same as itsm-server)
Audit-log-domain - auditlogs-api.company.local (IP 10.0.5.1 same as itsm-server)
Download-domain - dl.company.local (IP 10.0.5.1 same as itsm-server)
Xmpp-domain - xmpp.company.local (IP 10.0.5.2 XMPP-server)
RealtimeDeviceCommunication-API-domain - rtdc-api.company.local (IP 10.0.5.1 same as itsm-server)
RealtimeDeviceCommunication-Relay-domain - rtdc-relay-01.company.local (IP 10.0.5.1 same as itsm-server)
BulkInstallationPackage-domain - bip.company.local (IP 10.0.5.1 same as itsm-server)

Required subdomain list which should be resolved:

From ITSM-server
  • xmpp (to XMPP-server)
From XMPP-server
  • <itsm> - customer specified ITSM_DOMAIN (to itsm-server)
From administrator endpoint (web access)
  • <itsm> - customer specified ITSM_DOMAIN (to itsm-server)
  • rtdc-api (to itsm-server)
  • rtdc-relay-01 (to itsm-server)
From enrolled device endpoint
  • <itsm> - customer specified ITSM_DOMAIN (to itsm-server)
  • bip - bulk installation package download host (to itsm-server)
  • rmm-api - rmm logs reporting (to itsm-server)
  • auditlogs-api - audit logs reporting (to itsm-server)
  • rtdc-api - remote tools configuration (to itsm-server)
  • rtdc-relay-01 - remote tools download file (to itsm-server)
  • plugins - alerts, patch management, software inventory logs reporting (to itsm-server)
  • xmpp - persistent connection for receiving push messages and remote control commands (to XMPP-server)
From remote control tool
  • <itsm> - customer specified ITSM_DOMAIN (to itsm-server)
  • dl - check and download updates (to itsm-server)
  • xmpp - remote control communication (to XMPP-server)
Scheme example:

Domain communication

SSL requirements

We recommend to have wildcard certificate. You may to have certificates for each subdomain specified above.
Currently required to have wildcard ssl certificate for itsm-server *.company.local and the same certificate or specific one for XMPP xmpp.company.local.
If you don’t have real domain and trusted SSL certificate you can generate self-signed certificates by yourself.
Note: endpoints couldn’t work with self-signed certificate on XMPP. In this case you have never get green online status on itsm-server for endpoints also remote control will not work.

For minimal configuration it is required to have set of certificates and keys for each server (itsm, xmpp)

Setup SSL certificates for ITSM

Place valid ssl certificate and key into /opt/itsm/web/certs under the names cert.crt and cert.key.

Note: private key must be without passphrase as web server could not work with those

# create directory
sudo mkdir -p /opt/itsm/web/certs

# copy prepared certificate and key to destination
cp /path/to/your/certificate.crt /opt/itsm/web/certs/cert.crt
cp /path/to/your/certificate.key /opt/itsm/web/certs/cert.key

If you don’t have valid certificates. It is possible to issue self-signed certificate key-pair. But in this case you need to allow unsecured access in the browser and some features will be dropped. Next commands create self-signed certificates:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /opt/itsm/web/certs/cert.key -out /opt/itsm/web/certs/cert.crt

NOTE: your certificate CN (common name) domain should be the same as ITSM_DOMAIN variable value specified in .env

Setup SSL certificates for XMPP

Place valid pem certificate for domain specified in XMPP_DOMAIN into /opt/ejabberd/certs

Filename should be in following pattern {XMPP_DOMAIN}.pem

NOTE:

  • certificate name should be exactly as XMPP_DOMAIN value specified in .env file.
    For example above certificate filename should be yourdomain.com.pem without prefix “xmpp.” .
  • Certificate bundle must contain root CA certificate. For creation valid certificate need to concatenate private.key + certificate.crt + chain.crt + root.crt
sudo mkdir -p /opt/ejabberd/certs
cat cert.key cert.crt chain.crt root.crt > your.domain.pem
sudo mv your.domain.pem /opt/ejabberd/certs/

Manual installation

  1. Prerequisites: install docker and docker-compose

    1. Login to remote server
      ssh username@ip-or-hostname

    2. Get installation script (for ubuntu)
      wget http://get-compose.on-premise.itarian.com/install-docker-compose.sh

    3. Make file executable
      chmod +x install-docker-compose.sh

    4. Run script
      sudo ./install-docker-compose.sh

    5. Setup local user permissions
      sudo usermod -a -G docker $USER

    6. Logout from current session and login again to apply local user group changes
      exit
      ssh username@ip-or-hostname

    7. Perform docker login
      Note: your must have created account on CAM Comodo Accounts Management and order AEP license
      docker login registry.on-premise.itarian.com
      Enter CAM login and CAM password

  2. Extra server setup

    Only for itsm server it need to tune system settings:
    sudo sysctl -w vm.max_map_count=262144
    echo vm.max_map_count=262144 | sudo tee -a /etc/sysctl.conf

  3. Logging Configurations

    In order to apply log rotation and limiting the size of logs add below command under /etc/logrotate.conf:

    /var/lib/docker/containers/*/*.log {
      rotate 5
      copytruncate
      missingok
      notifempty
      compress
      maxsize 200M
      daily
    }
    Or create file /etc/docker/daemon.json with below commands:
    {
      "log-driver": "json-file",
      "log-opts": {
        "max-size": "200m",
        "max-file": "5",
        "compress": "true"
      }
    }
    Above configurations will rotate logs large than 200m and compress rotated logs. The system will keep original and 4 rotated log files. Configurations can be changed according to needs.
    Note: For existing installations, second option requires on-premise server to be upgraded to latest version manually.

  4. Get docker-compose.yml and configure settings

    1. Create and navigate to itsm dir
      mkdir ~/itsm
      cd ~/itsm

    2. Get docker-compose.yml for specific server
      for itsm server
      wget https://get-compose.on-premise.itarian.com/version/release/registry-docker-compose.yml -O docker-compose.yml
      for XMPP-server
      wget https://get-compose.on-premise.itarian.com/version/release/registry-ejabberd-docker-compose.yml -O docker-compose.yml

    3. create file with name .env and fill it according to your server requirements

      For itsm server

      ITSM_DOMAIN=on-premise.itsm.local
      ITSM_TURN_SERVERS=ip of turn server
      ITSM_XMPP_HOST=xmpp.itsm.local
      ITSM_XMPP_IP=ip of xmpp server
      ITSM_WEB_HOST=same as ITSM_DOMAIN on-premise.itsm.local
      ITSM_WEB_IP=ip of this host

      Where:
      ITSM_DOMAIN - domain name which must be the same as the certificate domain used in setup
      ITSM_TURN_SERVERS - list of ips where turn server is running separated by comma or space ( if turn servers has been setup )
      ITSM_XMPP_HOST - domain for xmpp server ( if xmpp has been setup )
      ITSM_XMPP_IP - ip for host specified in ITSM_XMPP_HOST if dns record cannot be resolved ( if xmpp has been setup without dns )
      ITSM_WEB_HOST - domain name which used by rmm microservices and points to itsm-server.
      ITSM_WEB_IP - ip for host specified in ITSM_WEB_HOST if dns record cannot be resolved

      For XMPP-server

      XMPP_DOMAIN=itsm.local
      ITSM_WEB_HOST=on-premise.itsm.local
      ITSM_WEB_IP=ip of itsm server

  5. Prepare and save SSL certificates

    1. Prepare ASCII encoded certificate files:

      • cert.crt - wildcard certificate for domain which specified as ITSM_DOMAIN
      • cert.key - private key without password for cert.crt
      • chain.crt - chaining certificate for cert.crt
      • root.crt - CA root certificate for chain
    2. Compose and save certificate files

      For itsm server

      Save cert.crt and cert.key

      sudo mkdir -p /opt/itsm/web/certs
      cat cert.crt chain.crt | sudo tee /opt/itsm/web/certs/cert.crt
      sudo cp cert.key /opt/itsm/web/certs/cert.key
      For XMPP-server

      Save single certificate bundle under the name specified in XMPP_DOMAIN and .pem extension

      sudo mkdir -p /opt/ejabberd/certs
      cat cert.key cert.crt chain.crt root.crt | sudo tee /opt/ejabberd/certs/itsm.local.pem
  6. Starting up and basic management

    • Starting system
      cd ~/itsm
      docker-compose up -d

    • Stopping system
      docker-compose down

    • View logs / debugging
      docker-compose logs

Manual upgrade

To perform upgrade it need to go on next steps:

  1. Go to itsm directory which contains docker-compose.yml and .env files
    cd ~/itsm

  2. Stop the system
    docker-compose down

  3. Get latest docker-compose.yml file for regarding to your server type
    for itsm server
    wget https://get-compose.on-premise.itarian.com/version/release/registry-docker-compose-with-turn.yml -O docker-compose.yml
    for XMPP-server
    wget https://get-compose.on-premise.itarian.com/version/release/registry-ejabberd-docker-compose.yml -O docker-compose.yml

  4. Run the system again
    docker-compose up -d

SMTP settings

After complete installation need to setup SMTP to have ability to receive email from ITSM server

  1. Login to portal as admin
  2. Open link https://itsm.yourcompany.com/email/settings where itsm.yourcompany.com is your ITSM server domain specified during installation
  3. Click edit button Email settings view
  4. Fill all fields according to your corporate smtp settings and save it Email settings edit

Troubleshooting

  • Problem: ITSM server not responding

    Suggestion: Check docker logs for service "web"

    Command
    docker-compose logs -f web --tail 20
    Output should not have constantly appended rows like

    nginx: [emerg] host not found in upstream "service_name" in /etc/nginx/conf.d/website.conf:277

    In case you find those rows, it mean that your service_name is not started
    Check container logs with that service_name
    docker-compose logs -f --tail 20 service_name
    and check all containers that have not status Running or Exit 0

  • Problem: any docker command (example: docker ps) not works from your current user (Error like permission denied).

    Suggestion: Check your current user belongs to group docker

    Command
    groups
    Should contains docker

    If user not belongs to group docker run this command
    sudo usermod -a -G docker $USER

    After that need to logout and login again with current user

  • Problem: any docker command (example: docker ps) not works. (Error like docker daemon is not running).

    Suggestion: Check docker daemon status

    Command
    sudo service docker status
    Output should have Active: active (running)

    If output is Active: inactive (dead) Start docker daemon with command
    sudo service docker start

  • Problem: at least one container from list docker-compose ps has state Exit 1

    Suggestion: Try to start this container.

    Command
    docker start <container name from column Name>
    If problem persist - need to view container logs Command
    docker logs <container name from column Name> --tail 100
    Output need to send to ITSM team for analyzing.

  • Problem: server data consumes lot of disk space (/var/lib/docker/overlay2)

    Suggestion: Try to cleanup old docker images.
    Every on-premise update pull several new images and keep both old and new ones

    Command
    docker image prune -af

  • Problem: server data consumes lot of disk space (/opt/itsm/elk)

    Suggestion: Try to cleanup old elasticsearch indexes

    On-premise provided with built-in elasticsearch management system kibana
    Follow kibana setup guide to setup it
    Remove indexes with kibana GUI

  • Problem: EM CCS clients updates consume lot of internet traffic

    Suggestion: Setup local antivirus database updates
    On-premise provided with av-db offline package
    Follow antivirus local updates guide to setup CCS update preferences