cloudsoft.io

Riak CS Persistence

AMP can persist its state either to the file system, or to any Object Store API that jclouds supports. This section describes how to set up AMP to persist to Riak using Riak CS Sorage API, which is compatible with the Amazon S3 REST API.

Setting up Riak CS

You need to install Riak, Stanchion and Riak CS.

Download links, version compatibility information and instructions are available in the RiakCS site.

Server Configuration

Riak CS performs best with a large server, and with plenty of RAM. For more information, see Planning for a Riak System.

Running a Riak cluster is important for high availability.

There is also important Riak configuration advice in the “Tuning” section of the “Riak for Operators” docs, as well as the sections on “Running a Cluster” and “Security”.

For example, ensure that the open file limit is sufficiently high (e.g. 65536).

Install Riak

Follow the Riak installation instructions.

Stop Riak, and configure riak.conf:

listener.http.internal = <internal_ip>:8098
listener.protobuf.internal = <internal_ip>:8087
buckets.default.allow_mult = true

See the Basho Docs for configuring Riak for more information.

Install Stanchion

Install Stanchion.

Set up stanchion.conf

listener = <internal_ip>:8085
riak_host = <internal_ip>:8087
nodename = stanchion@<erlang_nodename>

The <erlang_nodename> must have the same RDNS. When using IP it shouldn’t matter.

Before continuing with the set up check whether riak and stanchion run properly.
Start Riak: ulimit -n 65536 && service riak start
Start Stanchion: ulimit -n 65536 && service stanchion start
Don’t forget to stop them, as you’ll need to make changes to their configurations.

See Basho Docs for configuring Stanchion for more information.

Install Riak CS

Follow the Riak CS instructions.

Set up riak-cs.conf:

listener = <x_ip>:8080
riak_host = <internal_ip>:8087
stanchion_host = <internal_ip>:8085
anonymous_user_creation = on
nodename = riak-cs@<internal_ip>
root_host = <external_ip> - for vCloud Director
root_host = <endpoint> - for AWS
pool.request.overflow = 0
pool.list.overflow = 0

The listener <x_ip> indicates the network interface(s) and port to listen on. For clouds where the private or internal IP address is used by VMs within the network (e.g. AWS or vCloud), the internal IP should be used. For clouds where the external IP address is used (and in particular where there is a separate NIC for the private network - e.g. SoftLayer), this should be the external IP. You could use 0.0.0.0 to listen on all network interfaces.

The root_host <endpoint> is the address that will subsequently be used in the S3 API’s URL. For example, on AWS this would be something like “ec2-54-193-206-235.us-west-1.compute.amazonaws.com”; on SoftLayer it would be the public IP of the VM.

See Basho Docs for configuring Riak CS for more information.

Also create /etc/riak/advanced.config, and add configuration like the following:

[
{riak_kv, [
    %% Other configs
    {add_paths, ["/usr/lib64/riak-cs/lib/riak_cs-2.0.1/ebin"]},
    {storage_backend, riak_cs_kv_multi_backend},
    {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
    {multi_backend_default, be_default},
    {multi_backend, [
        {be_default, riak_kv_eleveldb_backend, [
            {total_leveldb_mem_percent, 30},
            {data_root, "/var/lib/riak/leveldb"}
        ]},
        {be_blocks, riak_kv_bitcask_backend, [
            {data_root, "/var/lib/riak/bitcask"}
        ]}
    ]}
    %% Other configs
]},
{riak_core, [
    %% Other configs
    {default_bucket_props, [{allow_mult, true}]}
    %% Other configs
]}
].

You may need to change the path added in add_paths based on the version of Riak CS used.

Launch Riak CS

Run Riak, Stanchion and Riak CS in that order:

service riak start && service stanchion start && service riak-cs start

Create user credentials by running the command below (where <x_ip> is the address that Riak CS is listening on):

curl  -H 'Content-Type: application/json' \
  -XPOST http://<x_ip*>:8080/riak-cs/user \
  --data '{"email":"email1@example.com", "name":"example"}'

Apply this admin key in the Riak CS configuration riak-cs.conf:

  • Stop Riak CS by running service riak-cs stop
  • Modify riak-cs.conf by adding:
admin.key =  <"key_id" from the returned credentials>  
admin.secret =  <"key_secret" from the returned credentials>  
anonymous_user_creation = off

Apply this admin key in the Stanchion configuration stanchion.conf:

  • Stop Stanchion by running service stanchion stop
  • Modify stanchion.conf by adding:
admin.key =  <"key_id" from the returned credentials>  
admin.secret =  <"key_secret" from the returned credentials>

Start again Stanchion and Riak CS:

service stanchion start && service riak-cs start

Setting up AMP

Configuring brooklyn.properties

Create a named location for Riak CS, using its S3 compliant API:

brooklyn.location.named.<location-name>=jclouds:s3:http://<s3_hostname>:8080
brooklyn.location.named.<location-name>.identity=<key_id>
brooklyn.location.named.<location-name>.credential=<key_secret>

If using an isoloated network, ensure there is a NAT rule to access port 8080 of the Riak CS node from the AMP server. The s3 URL would then instead be of the form:

http://<gateway_ip>:<natted_port>

Launch AMP

  • Run AMP with named:<location-name> in the persistenceLocation of the org.apache.brooklyn.osgilauncher.cfg file.

The <location-name> corresponds to the named location defined in brooklyn.properties.

The named bucket <bucket.name> will be created automatically, if it does not already exist. Bucket names should follow the Riak CS bucket naming convention.

For documentation of the AMP persistence options, see the persistence docs.

AMP High Availability

To run AMP in high-availability mode, run two AMP instances pointing at the same Riak CS store and the same bucket.

For documentation of the AMP high availability options, see the high availiability docs.

Riak CS Hints

Always start services in order: Riak, Stanchion, Riak CS; and stop them in reverse order.

If service riak/stanchion/riak-cs stop fails or service riak/stanchion/riak-cs status returns “riak/stanchion/riak-cs dead but subsys locked”, then check configurations and reboot.

If org.jclouds.rest.AuthorizationException: Access Denied is thrown after running AMP, check if anonymous_user_creation = off.

Make sure that port 8080 (or the NAT’ed port) is reachable from your AMP instance.

If any problems occur after running AMP:

  • Check whether Riak, Stanchion and Riak CS are still running properly. If not - restart them and/or reboot the server(s).
  • If there is a problem with memory after trying to start: (1) try to remove some or all of access.log.<date> or (2) rm -rf /var/lib/riak/ring/*

See the Riak documentation from Basho for the definitive guide to setting up Riak CS.

Riak performs very well when properly configured but care must be taken when configuring Riak, giving it ample resources.

Snapshots of Persisted State

A snapshot copy of the persisted state can be taken at any time by using the REST API /v1/server/ha/persist/export (for example to export the data from Riak CS, and then to write it to a given file system location or to a backup bucket).

It is strongly recommended to perform this backup prior to upgrading AMP, and also to do periodic backups.

See the docs Persisted State Backup.

AMP and Maeztro are developed and supported by Cloudsoft Corporation.