cloudsoft.io

AMP upgrade

This guide provides all necessary information to upgrade AMP for both the RPM/DEB and Tarball packages.

Upgrade from AMP 5.0 onward

  1. Stop AMP:

    # CentOS 7 / RHEL
    sudo systemctl stop amp
    # CentOS6 and older
    sudo initctl stop amp
    

    If this does not stop it within a few seconds (as checked with sudo ps aux | grep karaf), then use sudo kill <JAVA_PID>

  2. Important! Backup persisted state and custom configuration, in case you need to rollback to a previous version.

    1. By default, persisted state is located at /var/lib/amp.

      The persistence dir is configured with the environment variable export AMP_PERSISTENCE_DIR=... Alternatively, persistenceDir can be configured in the file /etc/amp/org.apache.brooklyn.osgilauncher.cfg. The persistenceLocation can also be configured in that file. The persistence details will be logged in /var/log/amp/amp.info.log at startup time.

    2. Configuration files are in /etc/amp.

  3. Upgrade AMP:

    1. Download the new RPM/DEB package

    2. Upgrade AMP:

      # CentOS / RHEL 7 
      sudo yum upgrade cloudsoft-amp-karaf-5.x.x.rpm
      

      Note this creates a new directory at /opt/amp-5.x.x, and updates the symbolic link at /opt/amp to point at this. The previous AMP directory is left as-is.

      If there are conflicts in configuration files (located in /etc/amp), the upgrade will behave differently based on the package you are using:

      • RPM: the upgrade will keep the previously installed one and save the new version, with the suffix .rpmnew. You will then need to check and manually resolve those. In particular, check for /etc/amp/config.properties.rpmnew

      • DEB: the upgrade will ask you what to do.

  4. (This step is only required if your persisted state references Java OSGi bundles in the catalog items’ libraries section, which need to be updated/recompiled for the new AMP release. The steps below will update the persisted state to use the new versions of the OSGi bundles.)

    1. Prepare the new releases of your OSGi bundles, uploaded to stable URLs (e.g. using Artifactory).

    2. Prepare a configuration file, which lists the old and new bundle URLs. For example:

      https://example.org/foo-1.0.0.jar=https://example.org/foo-1.1.0.jar
      https://example.org/bar-1.0.0.jar=https://example.org/bar-1.1.0.jar
      
    3. Use the amp-upgrade-bundles.sh script to update the persisted state to reference the new bundle versions:

      chmod u+x amp-upgrade-bundles.sh
      
      export AMP_PERSISTENCE_DIR=/path/to/persisted-state-v2
      export BUNDLE_MAPPINGS_FILE=/path/to/bundle-list.conf
      
      ./amp-upgrade-bundles.sh
      
    4. Check the stdout and stderr of the script to ensure it did the transformations expected (e.g. no typos in the bundle list configuration file).

  5. Validate that the new release works, by starting in “HOT_BACKUP” mode.

    1. Before starting AMP, reconfigure /etc/amp/org.apache.brooklyn.osgilauncher.cfg and set highAvailabilityMode=HOT_BACKUP. This way when AMP is started, it will only read and validate the persisted state and will not write into it.

    2. Start AMP:

      # CentOS / RHEL 7
      sudo systemctl start amp
      
    3. Check whether you have rebind ERROR messages in /var/log/amp/amp.info.log, e.g. sudo grep -E "WARN|ERROR" /var/log/amp/amp.debug.log If you do not have such errors you can proceed.

    4. Stop AMP:

      # CentOS / RHEL 7
      sudo systemctl stop amp
      
    5. Change the highAvailabilityMode back to the default (AUTO) by commenting it out in /etc/amp/org.apache.brooklyn.osgilauncher.cfg

  6. Start AMP:

    # # CentOS / RHEL 7
    sudo systemctl start amp
    

    Wait for AMP to be running (i.e. its web-console is responsive)

    Confirm that there are no errors reported in the log (/var/log/amp/amp.info.log).

  7. Add the new AMP catalog items (as per the release notes “Upgrade Instructions”):

    1. Download the br tool (i.e. from the “CLI Download” link in the web-console)

    2. Login with br: br login http://localhost:8081 <user> <password>

    3. Update the catalog: br catalog add /opt/amp/catalog/catalog.bom

Troubleshooting:

  • AMP fails to start, writing to /var/log/amp/karaf.log the message “SEVERE: Could not launch framework” … “Could not resolve mvn:org.apache.felix/org.apache.felix.framework/5.4.0”.

    This means there is an invalid configuration file. Mostly likely, the upgrade encountered conflicts in the configuration file /etc/amp/config.properties.rpmnew, and created a file with suffix “.rpmnew”, leaving the old file in place.

    If so, consider replacing /etc/amp/config.properties with /etc/amp/config.properties.rpmnew, but ensure that any manual customization of the previous config.properties is also applied to the new version of the file.

  • AMP fails to read the persisted state (having performed the state tranformation step above).

    Ensure the persisted state (e.g. at /var/lib/amp) is owned by amp:amp.

  • AMP’s yum upgrade fails with an error like:

    Transaction check error:
    file /opt/amp from install of cloudsoft-amp-karaf-4.5.1_1.noarch conflicts with file from package cloudsoft-amp-karaf-4.2.0-1.noarch
    

    These upgrade instructions only apply when upgrading from AMP 4.3 onwards. If upgrading from earlier AMP releases, see the instructions in the sections below.

  • AMP upgrade fails on CentOS 6.

    Use of CentOS 7 is strongly recommended, instead of CentOS 6. However, if using CentOS 6, there are the following differences:

    • To start and stop AMP, instead use sudo initctl start amp and sudo initctl stop amp.

    • Instead of using the yum upgrade ... command, use sudo rpm -Uvh --nopreun cloudsoft-amp-karaf-4.x.x.rpm.

  1. Stop AMP:

    ./bin/stop amp
    

    If this does not stop it within a few seconds (as checked with sudo ps aux | grep karaf), then use sudo kill <JAVA_PID>

  2. Important! Backup persisted state and custom configuration.

    1. By default, persisted state is located at ~/.brooklyn/brooklyn-persisted-state. The persistenceDir and persistenceLocation are configured in the file ./etc/org.apache.brooklyn.osgilauncher.cfg. The persistence details will be logged in ./log/amp.info.log at startup time.

    2. Configuration files are in ./etc/. Any changes to these configuration files will need to be re-applied after reinstalling AMP.

  3. Install new version of AMP:

    1. Download the new tarball zip package

    2. Install AMP:

      tar -zxf cloudsoft-amp-karaf-4.x.x-xxxxxxxx.xxxx.tar.gz
      cd cloudsoft-amp-karaf-4.x.x-xxxxxxxx.xxxx
      
    3. Restore any changes to the configuration files (see step 2).

  4. (This step is only required if your persisted state references Java OSGi bundles in the catalog items’ libraries section, which need to be updated/recompiled for the new AMP release. The steps below will update the persisted state to use the new versions of the OSGi bundles.)

    1. Prepare the new releases of your OSGi bundles, uploaded to stable URLs (e.g. using Artifactory).

    2. Prepare a configuration file, which lists the old and new bundle URLs. For example:

      https://example.org/foo-1.0.0.jar=https://example.org/foo-1.1.0.jar
      https://example.org/bar-1.0.0.jar=https://example.org/bar-1.1.0.jar
      
    3. Use the amp-upgrade-bundles.sh script to update the persisted state to reference the new bundle versions:

      chmod u+x amp-upgrade-bundles.sh
      
      export AMP_PERSISTENCE_DIR=/path/to/persisted-state-v2
      export BUNDLE_MAPPINGS_FILE=/path/to/bundle-list.conf
      
      ./amp-upgrade-bundles.sh
      
    4. Check the stdout and stderr of the script to ensure it did the transformations expected (e.g. no typos in the bundle list configuration file).

  5. Validate that the new release works, by starting in “HOT_BACKUP” mode.

    1. Before starting AMP, reconfigure ./etc/org.apache.brooklyn.osgilauncher.cfg and set highAvailabilityMode=HOT_BACKUP. This way when AMP is started, it will only read and validate the persisted state and will not write into it.

    2. Start AMP:

      ./bin/start amp
      
    3. Check whether you have rebind ERROR messages in ./log/amp.info.log, e.g. sudo grep -E "WARN|ERROR" /opt/amp/log/amp.debug.log If you do not have such errors you can proceed.

    4. Stop AMP:

      ./bin/stop amp
      
    5. Change the highAvailabilityMode back to the default (AUTO) by commenting it out in ./etc/org.apache.brooklyn.osgilauncher.cfg

  6. Start AMP:

    ./bin/start amp
    

    Wait for AMP to be running (i.e. its web-console is responsive)

  7. Update the catalog, using the br command:

    1. Download the br tool (i.e. from the “CLI Download” link in the web-console)

    2. Login with br: br login http://localhost:8081 <user> <password>

    3. Update the catalog: br catalog add /opt/amp/catalog/catalog.bom

Rollback

This section applies only with you are using the RPM/DEB packages. To perform a rollback to a previous AMP version, please follow these instructions.

Note there are no “forwards compatibility” guarantees, that persisted state generated with a new version of AMP will work with older versions of AMP. Therefore persisted state written by a newer version of AMP will not necessarily work when rolling back to an older version of AMP.

  1. Stop AMP:

    # e.g. on CentOS 7 / RHEL
    sudo systemctl stop amp
    

    If this does not stop it within a few seconds (as checked with sudo ps aux | grep karaf), then use sudo kill <JAVA_PID>, or sudo kill -9 <JAVA_PID>.

  2. Restore the persisted state and custom configuration from the backup you made during the previous upgrade.

    1. By default, persisted state is located at /var/lib/amp.

      The persistence dir is configured with the environment variable export AMP_PERSISTENCE_DIR=... Alternatively, persistenceDir can be configured in the file /etc/amp/org.apache.brooklyn.osgilauncher.cfg. The persistenceLocation can also be configured in that file. The persistence details will be logged in /var/log/amp/amp.info.log at startup time.

    2. Configuration files are in /etc/amp.

  3. Downgrade AMP (to 4.3.x or later):

    # e.g. for CentOS / RHEL
    sudo yum downgrade cloudsoft-amp-karaf-4.x.x.rpm
    

    Note this creates a new directory at /opt/amp-4.x.x, if necessary, and updates the symbolic link at /opt/amp to point at this. The previous AMP directory is left as-is.

    Check if there are any conflicts in configuration files (located in /etc/amp), for example if there are any files with the suffix .rpmnew, and resolve those conflicts.

  4. Validate that the new release works, by starting in “HOT_BACKUP” mode.

    1. Before starting AMP, reconfigure /etc/amp/org.apache.brooklyn.osgilauncher.cfg and set highAvailabilityMode=HOT_BACKUP. This way when AMP is started, it will only read and validate the persisted state and will not write into it.

    2. Start AMP:

      # e.g. on CentOS 7 / RHEL sudo systemctl start amp

    3. Check whether you have rebind ERROR messages in /var/log/amp/amp.info.log, e.g. sudo grep -E "WARN|ERROR" /var/log/amp/amp.debug.log If you do not have such errors you can proceed.

    4. Stop AMP:

      # e.g. on CentOS 7 / RHEL
      sudo systemctl stop amp
      
    5. Change the highAvailabilityMode back to the default (AUTO) by commenting it out in /etc/amp/org.apache.brooklyn.osgilauncher.cfg

  5. Start AMP:

    # e.g. on CentOS 7 / RHEL
    sudo systemctl start amp
    

    Wait for AMP to be running (i.e. its web-console is responsive)

    Confirm that there are no errors reported in the log (/var/log/amp/amp.info.log).

Migrating AMP to a Different Server

Moving AMP to a different server has much in common with upgrade. The steps are described below, which assume that you are running AMP 4.3 or later. If upgrading AMP at the same time, see the additional steps described in the appropriate upgrade instructions.

  1. On the old server:

    1. Stop AMP:

      # CentOS 7 / RHEL
      sudo systemctl stop amp
      # CentOS6 and older
      sudo initctl stop amp
      

      If this does not stop it within a few seconds (as checked with sudo ps aux | grep karaf), then use sudo kill <JAVA_PID>

    2. Make a copy of the persisted state and custom configuration.

      1. By default, persisted state is located at /var/lib/amp.

        The persistence dir is configured with the environment variable export AMP_PERSISTENCE_DIR=... Alternatively, persistenceDir can be configured in the file /etc/amp/org.apache.brooklyn.osgilauncher.cfg. The persistenceLocation can also be configured in that file. The persistence details will be logged in /var/log/amp/amp.info.log at startup time.

        If using an external object store (e.g. S3), you do not have to copy the persisted state. The new AMP server can instead be configured to point at the same object store.

      2. Configuration files are in /etc/amp.

  2. On the new server:

    1. Install AMP, but do not start the service.

    2. Apply any configuration file customizations from the old server, on the new server.

    3. If using file-based persistence, unpack the persisted state to the appropriate location.

    4. Validate that the new release works, by starting in “HOT_BACKUP” mode.

      1. Before starting AMP, reconfigure /etc/amp/org.apache.brooklyn.osgilauncher.cfg and set highAvailabilityMode=HOT_BACKUP. This way when AMP is started, it will only read and validate the persisted state and will not write into it.

      2. Start AMP:

        # e.g. on CentOS 7 / RHEL
        sudo systemctl start amp
        
      3. Check whether you have rebind ERROR messages in /var/log/amp/amp.info.log, e.g. sudo grep -E "WARN|ERROR" /var/log/amp/amp.debug.log If you do not have such errors you can proceed.

      4. Stop AMP:

        # e.g. on CentOS 7 / RHEL
        sudo systemctl stop amp
        
      5. Change the highAvailabilityMode back to the default (AUTO) by commenting it out in /etc/amp/org.apache.brooklyn.osgilauncher.cfg

    5. Start AMP:

      # CentOS 7 / RHEL
      sudo systemctl start amp
      

      Wait for AMP to be running (i.e. its web-console is responsive)

      Confirm that there are no errors reported in the log (/var/log/amp/amp.info.log).

How to stop your service

On systemd:

systemctl stop amp 

On upstart:

stop amp

Web login credentials

  • User credentials should now be recorded in ${karaf.home}/etc/brooklyn.cfg

  • AMP will still read them from both ${karaf.home}/etc/brooklyn.cfg and ~/.brooklyn/brooklyn.properties

  • Configure a username/password by modifying brooklyn.cfg. An example entry is:

brooklyn.webconsole.security.users=admin
brooklyn.webconsole.security.user.admin.password=password2

Persistence

If you have persisted state you wish to rebind to, persistence is now configured in the following files:

For example, to use S3 for the persisted state, add the following to brooklyn.cfg:

brooklyn.location.named.aws-s3-eu-west-1:aws-s3:eu-west-1
brooklyn.location.named.aws-s3-eu-west-1.identity=<ADD CREDS>
brooklyn.location.named.aws-s3-eu-west-1.credential=<ADD CREDS>

To continue the S3 example, for the persisted state, add the following to org.apache.brooklyn.osgilauncher.cfg:

persistenceLocation=aws-s3-eu-west-1
persistenceDir=<ADD HERE>

AMP should be stopped before this file is modified, and then restarted with the new configuration.

Note that you can not store the credentials (for e.g. aws-s3-eu-west-1) in the catalog because that catalog is stored in the persisted state. AMP needs to know it in order to read the persisted state at startup time.

If binding to existing persisted state, an additional command is required to update the existing catalog with the AMP 4.0 versions. Assuming AMP has been installed to /opt/amp (as is done by the RPM and DEB):

    br add-catalog /opt/amp/catalog/catalog.bom
  

All existing custom jars previously added to lib/plugins (e.g. for Java-based entities) need to be converted to OSGi bundles, and installed in Karaf. The use of the “brooklyn.libraries” section in catalog.bom files will continue to work.

High Availability

High Availability mode can not be specified via a CLI parameter (as was the case in v3.x) but should be configured in org.apache.brooklyn.osgilauncher.cfg.

For details on how to configure HA, see the HA tutorial.

Upgrading Blueprints and Bundles

You can install and deploy new versions of blueprints at any time. AMP tracks multiple versions of the blueprints you install, as can be seen in the catalog.

Defining and Forcing Upgrade Paths

Bundles can declare what bundles and types they can upgrade, and they can also force the removal of installed bundles and types on startup/rebind. Forcing can be useful when upgrading AMP to replace any installed bundle not compatible with the newer version of AMP.

To add these definitions, use the following headers in the bundle’s OSGi META-INF/MANIFEST.MF:

  • Brooklyn-Catalog-Force-Remove-Bundles
  • Brooklyn-Catalog-Force-Remove-Legacy-Items
  • Brooklyn-Catalog-Upgrade-For-Bundles
  • Brooklyn-Catalog-Upgrade-For-Types

The most common patterns are to indicate that a bundle can replace all previous versions of itself and all types therein with types in the current bundle of the same name, using:

Brooklyn-Catalog-Upgrade-For-Bundles: *

And you can indicate that previous bundles should be uninstalled, forcing the above upgrades, with:

Brooklyn-Catalog-Force-Remove-Bundles: *

The above items can also take a range syntax, e.g. "*:[1,2)" when releasing a 2.0.0 to restrict to versions equal to or greater than 1.0.0 but less than 2.0.0. (Note that ranges must be quoted.) Entries can also take comma-separated lists, and in the case of replacements, they can define explicit renamed targets using sourceNameAndVersionRanges=targetNameAndVersion entries.
These fields are defined in full in the BundleUpgradeParser’s javadoc.

Upgrading the Version of Deployed Blueprints

New versions of blueprints are not automatically applied to existing deployments from older versions. This requires a rebind using the above techniques, or programmatic intervention: please ask on the mailing list for more information (and to help us identify the most common wishes in this area!).

Upgrading Systems Under Management

Blueprints can encode update processes for the systems they describe. The mechanisms for updating systems vary, depending whether it is stateless or stateful, whether following an immutable pattern (replacing components) or doing it on box (traditional, possibly taking systems out of action while upgrading), and whether applying an upgrade to many resources on a rolling fashion (repaving, blue-green). For this reason there is not a one-size-fits-all upgrade pattern to use in blueprints, but there are some common patterns that may be applicable:

  • Defining an upgrade effector on nodes, and on a cluster to apply to all nodes
  • Using a config key such as version which can be updated and reapplied
  • Exposing a deploy effector to pass files that should be run, such as WAR files, and invoking this effector with newer versions of WAR files to install

There are many more, and if you’ve written some good pieces to share, please consider contributing them so others can take advantage of them!

~~~~