Linux Notes: leapp (migration tool)

1. The information presented here is intended for educational use by qualified computer technologists.
2. The information presented here is provided free of charge, as-is, with no warranty of any kind.

Migrating Linux

Overview

Linux terminology: update vs. upgrade vs. migrate

• any minor point change is known as an update (upgrade is the same as update with the --obsoletes flag set)
• any major point change is known as migrate

leapp

• leapp is an open-source migration tool created by Red Hat for their own use. The source code is published on github
  • RHEL-leapp recipes are only available to RHEL customers via their private RHEL repos enabled by their subscription manager ($$$)
    • not sure if they offer any magic scripts
• Oracle adapted leapp for use with Oracle Linux
  • Oracle-leapp recipes are only available to systems already using Oracle Linux
    • so they offer a magic script to convert from CentOS-7 to Oracle Linux-7; then you use Oracle-leapp to migrate from 7 to 8, then 8 to 9
    • this script failed badly on a scratch system yesterday in my lab
• AlamaLinux adapted leapp for community use
  • Alma-leapp recipes are available via their ELevate project: https://wiki.almalinux.org/elevate/ELevate-quickstart-guide.html
  • ELevate-leapp is the safest way to migrate from CentOS-7:
    • to CentOS Stream-8, then to CentOS Stream-9
    • to Oracle Linux-8. hen use Oracle-leapp to jump from 8 to 9
    • to any other Enterprise Linux (EL) distro

speculation

• SUSE publishes SLES (comparable to RHEL), openSUSE leap (comparable to CentOS-8.5), and openSUSE tumbleweed (comparable to CentOS Stream)
• Last year they also published SUSE Liberty Linux: https://www.suse.com/products/suse-liberty-linux/ with tools to migrate from any version of RHEL to Liberty
• SUSE and Oracle are now behind https://openela.org/ (some have speculated that SUSE Liberty will appear here in a new form)

Linux terminology

Manual Methods (less safe)

• We all have seen plenty of migration procedures published by various self-help sites.
• The dozen I tested all worked with systems connected to the public internet.
• I did not find a single procedure which worked from behind a proxy server.
• I suspect that most will not stand the test of time as various repos move, and certificates expire (I had to tweak a few URLs in order to get some of them to work)
• Anyway, these procedures appear fragile (the upgrade might fail whether you made a mistake or not). Once started, there is no clean back out procedure.

Notable Magic Scripts (less safe)

Caveats: 

• None of these scripts are 100% safe.
• None offer a clean back out feature so could leave your system in an intermediate state if they hit an error.
• If you must use them, then first perform a trial run on a scratch system.
• Make backup copies of everything before you begin.

Orchestrated Method (safer)

• leaap is an open-source migration tool created by a Red Hat, and intended for use by paying customers wishing to migrate from anything to RHEL.
• If you are considering a jump to RHEL-8, then you had better talk to Red Hat who will provide a detailed procedures for use leapp. Many of the so-called recipes are only available from a Red Hat repo which sits behind a paywall (enabled via the Red Hat subscription manager)
• AlmaLinux created the ELevate project (which is based upon leapp) for moving to almost every other rpm-based Linux. AlmaLinux recipes are provided free-of-charge.
• Some rpm-based Linux distros also provide a  leapp-based procedure to help you migrate to their Linux offerings. Here is one for moving to Oracle Linux
  • https://docs.oracle.com/en/learn/ol-linux-leapp/ (documentation is the best I've ever seen)
    • https://blogs.oracle.com/linux/post/upgrade-oracle-linux-8-to-oracle-linux-9-using-leapp
    • If you are on CentOS-7 and wish to migrate to Oracle Linux-8, then I recommend using ELevate-leapp (keep reading). 

ELevate - leapp (intro)

• Moving from Enterprise Linux version 7 to 8 is a very big deal. Why? Here are a few major points:
  1. some v7 system tools, like yum and firewall-cmd, depend upon python2.7 which need to be updated to python3.6
  2. part way through, yum will be replaced with dnf
  3. caveat: if you own any third-party programs (or cgi scripts) which depend upon python2 then I suggest you rewrite them, or purchase python3 replacements before you migrate.
• The neat thing about leapp is that it begins the upgrade in a private namespace. If anything fails in the namespace, then your system continues to work properly
  
  • the leapp preupgrade command will allow you to do no-commitment trials
    • do as many as you like
    • all the work is loaded into a namespace which is where the replacement software is built
    • uninstalling leapp will to free up most space allocated by leapp
      

      # uninstall leapp
      yum erase \*leapp\*
      # -----------------------------------------
      # OPTIONAL: very dangerous commands so type carefully
      rm -fr	/var/lib/leapp/*
      rm -fr	/var/log/leapp/*

  • the leapp upgrade command
    • may repeat some preupgrade steps, may goes further.
      • If it fails, then then "no permanent changes have been made to your system".
      • If it doesn't fail, then you are, more-or-less, committed to doing a reboot
        • the first GRUB entry on the boot menu contains hooks to continue the migration (this can fail so always pretest on a scratch server)
        • from the main console, you could still select the second entry which would take you to the your existing OS 
  • caveats:
    • leapp preupgrade will run at the level of the system you are migrating from (say, CentOS7)
    • leapp upgrade will run at the level of the the system you are migrating to (say, AlmaLinux8)
    • if you sit behind a proxy server then you will need to get that information to both layers (more on this in a moment)
• General docs
  • https://leapp.readthedocs.io/en/latest/
  • https://leapp-to.github.io/gettingstarted

CentOS-7 to AlmaLinux-8 (or something else)

• The leapp procedures documented here work with systems connected to the public internet
• I wanted to migrate from CentOS-7 to AlmaLinux-8 so decided to only play with Elevate-leapp
  • caveat: later on I did a trial conversion from CentOS-7 to Rocky-8 (which worked) followed by a conversion to Rocky-9 (which did not work)
• The Elevate procedure from Alma did not work (as-is) from behind a proxy server on 2023-08-28, but I was able to develop a few workarounds that did work.

Playing it safe with trial converts

1. steps:
   • grab a spare computer then do a fresh install of the OS you want to migrate from (I used CentOS-7.9)
   • connect it to the public internet
   • follow the Elevate procedure described here: https://wiki.almalinux.org/elevate/ELevate-quickstart-guide.html
   • take notes about what you see AND how many times the system reboots (for this experiment it is important that you are physically located in front of the server's console device)
   • Note: I did a dozen trial conversions before I used leapp on a working server.
2. repeat step-1 but on a system connected it to a network behind a proxy server
   caveats:
   • anyone who has ever used CentOS-7 already knows you need add 1-2 lines of additional information to file /etc/yum.conf
   • anyone who has ever used CentOS-8 already knows you need add 1-2 lines of additional information to file /etc/dnf/dnf.conf
   • so it should be no surprise that you may need to update various copies of dnf.conf in the leapp namespace
   • but one of those files will be overwritten several times when you run leapp
     • hack-1: requires you to update this file: /etc/leapp/files/leapp_upgrade_repositories.repo
       • insert this line into every labelled stanza: proxy=http://proxy.neilrieck.net:8083
       • optionally, also add this line in every labelled stanza: sslverify=false (this could be dangerous so only use thus if you see CURL (60) error messages)
     • hack-2: requires you to add proxy information to file /var/lib/leapp/el8userspace/etc/dnf/dnf.conf then also run a little script on a second session which will monitor the file then update it whenever it is overwritten (usually 4 times)
     • hack-3: requires you to copy your organization's certificates into the namespace
     caveats:
     • hack-1 works 99% of the time
     • hack-2 (with hack-1) was twice at a remote location. "I think" this might be due to a misconfiguration of my company's proxy server.
       
       • hack-1 is necessary for leapp preupgrade
       • hack-2 is necessary for leapp upgrade
     • I haven't yet tried hack-3
3. repeat step-1 but use a backup copy of the server you wish to convert
4. repeat step-2 but use a backup copy of the server you wish to convert and connect it to a network behind a proxy server

Details (for use behind a proxy sever)

1. sudo yum update -y
   sudo reboot
   sudo yum install -y http://repo.almalinux.org/elevate/elevate-release-latest-el$(rpm --eval %rhel).noarch.rpm
   sudo yum install -y leapp-upgrade leapp-data-almalinux

2. tweak-1
   

   # we we will do everything from root (so do not use sudo)
   # become root 
   su -

   tweak-2a ( you might wish to skip this step )
   

   # changes to root's  shell
   # ------------------------
   #    changes for leapp
   export LEAPP_PROXY_HOST=http://proxy.neilrieck.net:8083/
   #    changes for curl
   export http_proxy=http://proxy.neilrieck.net:8083/
   export https_proxy=http://proxy.neilrieck.net:8083/

   tweak-2b
   

   # changes to your terminal emulator
   # ---------------------------------
   # set term to autowrap
   # set an ssh keepalive of 60 seconds
   

   tweak-3

   # add proxy information to the ELevate repo
   vim /etc/leapp/files/leapp_upgrade_repositories.repo

3. leapp preupgrade
   • this should work with little difficulty
   • if you see errors then you will need to correct them
   • if you see inhibited then you may need to edit the answer file
4. leapp upgrade (caveat: do not do this unless to intend to upgrade right now)
   • this works almost entirely in the target namespace
   • I have never had this phase go as-is:
     • in one case I was warned about multiple kernels so needed to remove them from CentOS then start over
     • in a second case I was warned about no available update for OpenSSL-devel which was version locked to OpenSSL which was going to be replaced
       • the solution was to yum erase openssl-devel then restart leapp upgrade
     • in a third case I was warned about no available update for Make-devel which was version locked to Make which was going to be replaced
       • the solution was to yum erase make-devel then restart leapp upgrade
   • If you see any "Curl (60)" errors then you will need to do the following ugly hack:
     • log into a second session; become root; execute my workaround script (see below)
     • repeat leapp upgrade on the primary session
5. reboot
• be patient here; you may see console messages which indicate that the whole thing could fail, but then the upgrade continues
• EL7-8: you will see some activity for 20 seconds followed by a 4 minute pause
• EL8-9: you will see some activity for 10 seconds followed by a 7 minute pause
6. wait 45-60 minutes then log in
7. vi /etc/dnf/dnf.conf
   1. comment out the exclude line
   2. add a proxy line
8. yum erase \*leapp\*
9. yum erase ELevate

1. Missing a file

   file /usr/share/redhat-logos from install of rocky-logos-90.14-1.el9.x86_64 conflicts with
   file from package rocky-logos-86.3-1.el8.x86_64
   # note: you cannot yum erase rocky-logos-86.3-1.el8.x86_64
   

2. Rather than rebooting, the system comes up in Recovery Mode but no commands appear to work. The system boots properly after cycling the power.
   Note: checking the logs, I see a python error associated with main.cli
3. No drama here. This worked like a charm

My workaround script

This hack was quickly written during a moment of frustration and panic
Only use this if you see failures associated with curl (60) errors.

# ================================================================
# title  : watch_elx_file_dnf-dot-conf.sh 
# author : Neil Rieck
# created: 2023-08-28
# notes  :
# 1) to be used during the leapp upgrade:
#	el="78" for CentOS-7 to AlmaLinux-8
#	el="89: for AlmaLinux-8 to AlmaLinux-9
# 2) run this on a second session while you upgrade from the first
# ================================================================
el="89"					# choices: "78" or "89"
if [[ ${el} == "89" ]]
then
    echo "for use with leapp 8-to-9"
    fs="/var/lib/leapp/el9userspace/etc/dnf/dnf.conf"
else
    echo "for use with leapp 7-to-8"
    fs="/var/lib/leapp/el8userspace/etc/dnf/dnf.conf"
fi
echo "sslverify=0"                            >> ${fs}
echo "proxy=http://proxy.neilrieck.net:8083/" >> ${fs}
n=1
c=0
prevModificationSeconds=$(date -r ${fs} +%s);           # snapshot 1
for (( ; ; ));
do
    echo "test: "${n}" changes: "${c}
    ((n=n+1));
    watchModificationSeconds=$(date -r ${fs} +%s);      # test
    if [[ ${prevModificationSeconds} != ${watchModificationSeconds} ]]
    then
        echo "-i-oops, the file changed"
        echo "sslverify=0"                            >> ${fs} 
        echo "proxy=http://fastweb.int.bell.ca:8083/" >> ${fs} 
        prevModificationSeconds=$(date -r ${fs} +%s);   # snapshot 2
        echo "-i-update complete"
	((c=c+1));
    fi
    sleep 1
done
echo "-i-exited"
#