How to Upgrade OpenNebula Enterprise Edition

Upgrading an OpenNebula instance is a relatively easy step, but now we have made this process even easier for corporate users. We have developed a tool called onecfg, a new command that is able to read all the changes and upgrade the configuration files of your OpenNebula cloud all at once. This is one of the exclusive Enterprise Tools available for customers as part of OneScape, the new extension that comes with the OpenNebula Subscription. In this article we will give you a brief introduction to this tool.

OneScape Benefits

One of the most difficult steps when upgrading OpenNebula is to migrate all the configuration files to the new version. Packages overwrite your configuration files with new ones, so you need to check the differences between each other. Or, sometimes, some files are moved from one location to another, which makes it a bit more difficult.

To migrate configuration files to new version, that's what we have created onecfg for. This tool is designed for cloud administrators and only available via CLI. The tool is able to read all your configuration files and apply your changes to the new ones. It also checks if new directories have been created and is able to create them and move files to them automatically.

How to Install OneScape

Installation depends on the distribution, but mainly the only thing you need to do is configure the repository using your support token and install the package. For example, if we are using Centos7:

$ cat << EOT > /etc/yum.repos.d/onescape.repo
[onescape]
name=onescape
baseurl=https://<support_token>@enterprise.opennebula.io/repo/onescape/5.12/CentOS/7/\$basearch
enabled=1
gpgkey=https://downloads.opennebula.io/repo/repo.key
gpgcheck=1
repo_gpgcheck=1
EOT

$ sudo yum install onescape

Upgrade Steps

The upgrade process contains a few steps:

1. Preparation for the upgrade, basically you need to configure the new repository and stop all OpenNebula services.
2. Packages upgrade, you upgrade all OpenNebula packages.
3. Configuration files upgrade, this is the upgrade using the onecfg tool.
4. Final checks, mainly you upgrade the OpenNebula database and check the consistency of it. Finally you start again all the services.

Example

Please, find below a step-by-step procedure on how to upgrade OpenNebula 5.12 EE running on Ubuntu 20.04 LTS to OpenNebula 6.8 EE.

Install OpenNebula key:

sudo wget -q -O- https://downloads.opennebula.io/repo/repo2.key | apt-key add -

Install files for onescape repository:

sudo apt-get install onescape

Before proceeding, make sure you don’t have any VMs in a transient state (prolog, migrate, epilog, save). Wait until these VMs get to a final state (running, suspended, stopped, done). Check the Managing Virtual Machines guide for more information on the VM life-cycle.

Set all hosts into offline mode:

onehost offline <hostids>

Stop OpenNebula and any other related services you may have running:

sudo systemctl stop opennebula*.service

Make sure that every OpenNebula process is stopped. The output of the following command should be empty:

sudo systemctl list-units | grep opennebula

Backup configuration files and database:

sudo cp -ra /etc/one /etc/one.$(date +'%F_%T')

sudo cp -ra /var/lib/one/remotes/etc /var/lib/one/remotes/etc.$(date +'%F_%T')

sudo onedb backup

Change OpenNebula version in repository configuration files:

sudo sed -e 's/5.12/6.8/g' -i /etc/apt/sources.list.d/opennebula.list

Upgrade opennebula packages:

sudo apt-get update

sudo apt-get install --only-upgrade opennebula opennebula-sunstone opennebula-gate opennebula-flow opennebula-provision opennebula-fireedge python3-pyone

Keep your currently-installed version of configuration files.

Before upgrading OpenNebula, you need to ensure that the configuration state is clean without any pending migrations from past or outdated configurations. To check the configuration state run

sudo onecfg status

sudo onecfg upgrade

If you faced with some issues during upgrade, please, check OpenNebula release known issues you are trying to upgrade from.

Thus during OpenNebula 5.12.13 upgrade to 6.8.1 the following issue may appear:

sudo onecfg upgrade
ANY   : Found backed up configuration to process!
ANY   : Snapshot to update from '/var/lib/one/backups/config/2024-01-02_12:57:26-v5.12.13'
ANY   : Backup stored in '/var/lib/one/backups/config/2024-01-02_13:14:35_24591'
ERROR : Error updating file '/etc/one/ec2_driver.default'
FATAL : ** TERMINATING UPDATE WITHOUT CHANGING SYSTEM ** Can't process files due to a fatal error (OneCfg::Config::Exception::PatchException, desc. 'Patch command failed'). Use --verbose/--debug modes to get more information.
FATAL : FAILED - Patch command failed

It’s  known issues for 5.12.13. To workaround it, please, execute

sudo sed -i 's/Copyright 2002-2023/Copyright 2002-2020/' /var/lib/one/backups/config/*v5.12.13/etc/one/az_driver.default /var/lib/one/backups/config/*v5.12.13/etc/one/ec2_driver.default

Upgrade database and check it for errors:

sudo onedb upgrade

sudo onedb fsck

Start OpenNebula services:

sudo systemctl start opennebula.service opennebula-sunstone.service opennebula-flow.service opennebula-gate.service

Check if upgrade was successful. E.g. one can list running VMs as oneadmin:

onevm list

To upgrade CNs one needs to execute on all CNs the following commands:

sudo sed -e 's/5.12/6.8/g' -i /etc/apt/sources.list.d/opennebula.list

sudo apt-get update

sudo apt-get upgrade opennebula-node-kvm

Enable upgraded CNs in OpenNebula and sync remote scripts:

onehost enable <hostid>

onehost sync --force

Check the status of upgraded CNs:

onehost list