Jumphost for Private Networks

By using the AMP Jumphost, one can provision apps into a private network where there is no direct access to the VMs.

Overview

Architecture

Jumphost Architecture

Each private network is (normally) dedicated to a single tenant. Within that private network, there is a jumphost that can access the other VMs within the private network. No network access is required to the jumphost from outside. This jumphost run an agent (the “AMP Jumphost” product).

A clustered message broker (e.g. RabbitMQ) is used send requests to the jumphost, and to receive responses from it. Through this mechanism, commands are executed on VMs within the private network.

The sequence for command execution (e.g. SSH or WinRM on a VM in the private network) is:

Manual pre-configuration:
1. The message broker cluster is pre-installed.
2. For a new private network, the jumphost is manually set up within private network. On startup it automatically subscribes to the message broker to receive the relevant requests.
AMP subscribes to a response queue, ready to receive the result.
AMP publishes a request to the appropriate queue on the message broker; this request describes the command to be executed and the response queue to use.
The jumphost picks up the request, validates it, and executes it.
The jumphost publishes the result to a response queue (e.g. exit status, stdout and stderr).
AMP receives the response via the message broker.

A RabbitMQ cluster is recommended, configured to ensure durability and high availability. This could be a centralised service, or could (if desired) be a separate RabbitMQ service per jumphost. It is assumed that the RabbitMQ cluster pre-exist and is pre-configured according to the enterprise’s security standards.

The AMP instances and the jumphost access the message broker via AMQP.

Network Connectivity Requirements

The jumphost must be able to reach the RabbitMQ broker’s AMQP port. The jumphost must be able to reach each VM on which commands are to be executed (normally on port 22 or 5985 for SSH and WinRM respectively).

Each AMP instance must be able to reach the RabbitMQ broker’s AMQP port.

The AMP Jumphost

The AMP jumphost is a stand-alone product.

Given the assumption that AMP does not have direct access to any VMs within the private network, it is assumed that the Jumphost is installed manually.

Server Setup

Server Specification

The size of server required by AMP Jumphost depends on the amount of activity: the number of WinRM and SSH commands to be executed concurrently, and the size of these commands (e.g. large file uploads are more expensive than simple commands).

For dev/test or when there are only a handful of VMs being managed, a small VM is sufficient. For example, an AWS m3.medium with one vCPU, 3.75GiB RAM and 4GB disk.

For larger production uses, a more appropriate machine spec would be two or more cores, at least 8GB RAM and 100GB disk. The disk is primarily for just for logs, and for some temporary files while uploading/downloading streams of data to/from VMs.

Supported Operating Systems

The recommended operating system is CentOS 6.x or RedHat 6.x.

Software Requirements

The AMP Jumphost requires Java (JRE or JDK) minimum version 1.7. OpenJDK is recommended.

Network Connectivity

No inbound connections are required to the AMP Jumphost. It will make outbound connections to the RabbitMQ broker to subscribe and publish messages.

The AMP Jumphost will also make SSH and WinRM connections directly to the other machines within the private network.

Locale

AMP Jumphost expects a sensible set of locale information and time zones to be available; without this, some time-and-date reporting may be surprising.

User Setup

It is recommended that the AMP Jumphost run as a non-root user.

No sudo permissions are required.

Linux Kernel Entropy

Check that the Linux kernel entropy is sufficient. See Increase Entropy.

Limits for Open Files and Processes

We recommend ensuring nproc and nofile are reasonably high (e.g. higher than 1024, which is often the default). We recommend setting it limits to a value above 16000.

If you want to check the current limits run ulimit -a.

Here are instructions for how to increase the limits for RHEL like distributions. Run sudo vi /etc/security/limits.conf and add (if it is “brooklyn” user running Cloudsoft AMP):

    amp           soft    nproc           16384
    amp           hard    nproc           16384
    amp           soft    nofile          16384
    amp           hard    nofile          16384

Generally you do not have to reboot to apply ulimit values. They are set per session. So after you have the correct values, quit the SSH session and log back in.

For more details, see one of the many posts such as http://tuxgen.blogspot.co.uk/2014/01/centosrhel-ulimit-and-maximum-number-of.html

Installation

Install the AMP Jumphost by unpacking the .tar.gz or .zip installer.

jumphost.properties

Configure the jumphost using either the default properties file at ~/.brooklyn/jumphost.properties, or by supplying the path to a properties file via the command line argument --globalProperties or --localProperties.

The properties file should include configuration like that below (but with the values changed):

jumphost.id=JUMPHOST_123
tenant.id=TENANT_123

messageManager.rabbitmq.host=1.2.3.4
messageManager.rabbitmq.port=5672
messageManager.rabbitmq.username=myname
messageManager.rabbitmq.password=pa55w0rd

messageManager.crypto.secretKey=UmFuZG9tRW5jcnlwdEtleQ==
messageManager.crypto.initVector=UmFuZG9tSW5pdFZlY3Rvcg==

A full description of the configuration options is shown below:

jumphost.id: a unique id for this jumphost; recommended to be used as part of the queue name, and also in logging.
tenant.id: a unique id for this tenant; recommended to be used as part of the queue name, and also in logging.
messageManager.rabbitmq.host: the IP or hostname of the RabbitMQ broker
messageManager.rabbitmq.port: the AMQP port of the RabbitMQ broker
messageManager.rabbitmq.username: the username for authenticating with the RabbitMQ broker, to publish and to subscribe to messages.
messageManager.rabbitmq.password: the password for autneticating with the RabbitMQ broker.
messageManager.crypto.secretKey: the base-64 encoded secret key, used to encrypt and decrypt messages (see the separate message encryption section).
messageManager.crypto.initVector: the base-64 encoded initialization vector, used when encrypting and decrypting messages.
messageManager.crypto.enabled: if false, disables all message encryption.
messageManager.crypto.secretKeyAlgorithm: the algorithm for encrypting/decrypting messages; defaults to “AES”, which is 128 bit.
messageManager.crypto.transformation: the Cipher transformation to use for padding ; defaults to “AES/CBC/PKCS5Padding”.

Launching

To launch the jumphost, run:

nohup ./bin/jumphost launch > /dev/null &

Note the redirect to /dev/null is optional - it reduces the disk usage, versus it writing to nohup.out.

A full list of the CLI options can be obtained by running ./bin/jumphost help, or help on a particular option such as .bin/jumphost help launch.

The pid of the jumphost process will be written to the file pid_java.

Logging

Logging is configured in ./conf/logback*. See Logback Documentation for further details.

By default, logs are written to ./jumphost.debug.log and ./jumphost.info.log.

AMP Configuration

Within AMP, location(s) can be configured to use the jumphost for all SSH-based and WinRM-based access to the VMs.

Named Location Configuration Example

Example configuration for a location is shown below (focusing on the configuration relating to the jumphost; other cloud-specific configuration is omitted for brevity):

brooklyn.location.named.MyPrivateLocation=jclouds:vcloud-director:https://acme.com/api

brooklyn.location.named.MyPrivateLocation.useJcloudsSshInit=false
brooklyn.location.named.MyPrivateLocation.pollForFirstReachableAddress=false

brooklyn.location.named.MyPrivateLocation.sshToolClass=io.cloudsoft.amp.jumphost.ssh.client.SshProxiedTool
brooklyn.location.named.MyPrivateLocation.sshToolClass.jumphost.id=JUMPHOST_123
brooklyn.location.named.MyPrivateLocation.sshToolClass.tenant.id=TENANT_123
brooklyn.location.named.MyPrivateLocation.sshToolClass.messageManager.rabbitmq.host=1.2.3.4
brooklyn.location.named.MyPrivateLocation.sshToolClass.messageManager.rabbitmq.port=5672
brooklyn.location.named.MyPrivateLocation.sshToolClass.messageManager.rabbitmq.username=myname
brooklyn.location.named.MyPrivateLocation.sshToolClass.messageManager.rabbitmq.password=pa55w0rd
brooklyn.location.named.MyPrivateLocation.sshToolClass.messageManager.crypto.secretKey=UmFuZG9tRW5jcnlwdEtleQ==
brooklyn.location.named.MyPrivateLocation.sshToolClass.messageManager.crypto.initVector=UmFuZG9tSW5pdFZlY3Rvcg==

brooklyn.location.named.MyPrivateLocation.winrmToolClass=io.cloudsoft.amp.jumphost.winrm.client.WinRmProxiedTool
brooklyn.location.named.MyPrivateLocation.winrmToolClass.jumphost.id=JUMPHOST_123
brooklyn.location.named.MyPrivateLocation.winrmToolClass.tenant.id=TENANT_123
brooklyn.location.named.MyPrivateLocation.winrmToolClass.messageManager.rabbitmq.host=1.2.3.4
brooklyn.location.named.MyPrivateLocation.winrmToolClass.messageManager.rabbitmq.port=5672
brooklyn.location.named.MyPrivateLocation.winrmToolClass.messageManager.rabbitmq.username=myname
brooklyn.location.named.MyPrivateLocation.winrmToolClass.messageManager.rabbitmq.password=pa55w0rd
brooklyn.location.named.MyPrivateLocation.winrmToolClass.messageManager.crypto.secretKey=UmFuZG9tRW5jcnlwdEtleQ==
brooklyn.location.named.MyPrivateLocation.winrmToolClass.messageManager.crypto.initVector=UmFuZG9tSW5pdFZlY3Rvcg==

Note that the configuration values must match those of the jumphost (i.e. the same jumphost.id, tenant.id, RabbitMQ broker, and cryptography configuration).

The previous amp.id is now deprecated. If not supplied, the unique id of the AMP instance will be used.

The useJcloudsSshInit=false will disable any attempts by jclouds to execute SSH commands (which would thus not have gone over the jumphost).

The pollForFirstReachableAddress=false ensures that AMP will not try to reach the ip:port of the VM, to determine which of the multiple possible addresses is reachable from the AMP server. Instead, the first address returned by the Cloud provider is taken, and that is subsequently passed to the AMP Jumphost for executing commands. Note this may cause strange behaviour or fail in clouds that give the VM multiple IP addresses, where only one of those IPs is accessible from the AMP jumphost.

With this configuration, VM provisioning will be done in the normal way. However, when attempting to execute SSH or WinRM commands on the VMs, the execution will be done via the AMP Jumphost.

BYON Location Configuration Example

It is also possible to use the AMP Jumphost for a BYON location. For example, in YAML:

location:
  byon:
    hosts:
    - ssh: 10.0.0.1:22
      privateKeyFile: ~/.ssh/brooklyn.pem
      user: myuser
    - winrm: 10.0.0.2:5985
      password: mypassword
      user: myuser
      osFamily: windows
    
    sshToolClass: io.cloudsoft.amp.jumphost.ssh.client.SshProxiedTool
    sshToolClass.jumphost.id: JUMPHOST_123
    sshToolClass.tenant.id: TENANT_123
    sshToolClass.messageManager.rabbitmq.host: 1.2.3.4
    sshToolClass.messageManager.rabbitmq.port: 5672
    sshToolClass.messageManager.rabbitmq.username: myname
    sshToolClass.messageManager.rabbitmq.password: pa55w0rd
    sshToolClass.messageManager.crypto.secretKey: UmFuZG9tRW5jcnlwdEtleQ==
    sshToolClass.messageManager.crypto.initVector: UmFuZG9tSW5pdFZlY3Rvcg==
    
    winrmToolClass: io.cloudsoft.amp.jumphost.winrm.client.WinRmProxiedTool
    winrmToolClass.jumphost.id: JUMPHOST_123
    winrmToolClass.tenant.id: TENANT_123
    winrmToolClass.messageManager.rabbitmq.host: 1.2.3.4
    winrmToolClass.messageManager.rabbitmq.port: 5672
    winrmToolClass.messageManager.rabbitmq.username: myname
    winrmToolClass.messageManager.rabbitmq.password: pa55w0rd
    winrmToolClass.messageManager.crypto.secretKey: UmFuZG9tRW5jcnlwdEtleQ==
    winrmToolClass.messageManager.crypto.initVector: UmFuZG9tSW5pdFZlY3Rvcg==

Entity Configuration

The Jumphost only accepts SSH and WinRM commands. This means that no other types of interaction with the VMs inside the private network is possible from AMP (given that the primary use-case for the jumphost is to provision and manage applications within an isolated private network).

The entities within AMP must be configured to turn off other connection mechanisms (e.g. management over JMX, HTTP, etc).

For example:

For JBoss AS 7, use httpMonitoring.enabled: false.
For JBoss AS 6, use jmx.enabled: false.
For Tomcat, use jmx.enabled: false.
For MongoDB, use clientMonitoring.enabled: false (though this will not work for clustered MongoDB).
For Riak, use httpMonitoring.enabled: false.
For Cassandra, use thriftMonitoring.enabled: false and jmx.enabled: false.

Message Encryption

All messages sent and received via the message broker are encrypted/decrypted by AMP and by the AMP Jumphost (unless crypto.enabled is set to false).

By default, this uses the “AES” algorithm, which uses 128 bit keys.

For AES-256, it may require the Java Cryptography Extension (JCE) Unlimited Strength
Jurisdiction Policy, depending on the JVM’s provider and version. This must be installed on the AMP Jumphost and on each AMP instance. Appropriate length private keys and initialization vectors must be used.

The mechanism used to generate the secret key and the initialization vector is up to the enterprise.

Out of scope

It is assumed that the RabbitMQ cluster already exists, and has been configured according to the enterprise’s security standards.

The processes for operations staff (e.g. end-users or support staff SSH’ing to the VMs) is out-of-scope. Such considerations are dependent on the security model within the enterprise.

Future Work and Known Limitations

There are a number of enhancements

Longevity and stress testing is still on-going. Testing of failure scenarios is also on-going (e.g. temporary loss of connectivity to the RabbitMQ broker).
WinRM execution from the AMP Jumphost is intermittently unreliable. The underlying cause is a problem in the WinRM clinet (pywinrm, using jython).

The long-term fix is to switch to a pure Java WinRM client implementation. Work is underway for this, but is unfortunately not ready for this release.
On AMP Jumphost restart, it will automatically re-subscribe to its queues and execute messages on those queues. However, the tasks that were being executed with the Jumphost stopped will have been lost. Worst case is that no response message is sent for the request, causing the AMP server to continually wait. The AMP server will (by default) timeout after waiting three hours for the response.
The behaviour of pollForFirstReachableAddress=false may be changed in future versions, to better handle clouds that give multiple addresses to the VM. An improved design would be for AMP to query the Jumphost, for which ip:port is reachable.
If there is a single location that can deploy both SSH and Windows machines, the need to duplicate the jumphost configuration for both the sshToolClass and winrmToolClass is sub-optimal. This may be changed in a future version.
The auto-populated amp.id will change when the AMP server is restarted. This means the response queue name will change. See https://issues.apache.org/jira/browse/BROOKLYN-202.
The SSH connections and WinRM connections are not re-used. Each new execution will create a new connection on the AMP Jumphost. This is CPU-intensive.
The only monitoring performed by AMP on entities within the private network is over SSH or WinRM (via the Jumphost, because that is all the Jumphost supports).

One approach to monitoring is to use a third party monitoring service such as NewRelic: have each VM push to that, and then have AMP query the monitoring (central) service to get metrics about each entity.
A future feature could be to support VM-provisioning via the jumphost. This would be useful for clouds where the cloud endpoint is private - i.e. where it can be accessed by the jumphost, but not directly by AMP.