Linux Notes: ELevate - leapp (migration tool)

  1. The information presented here is intended for educational use by qualified computer technologists.
  2. The information presented here is provided free of charge, as-is, with no warranty of any kind.
Edit: 2024-11-18
back to: my Linux Notes (index)

Migrating Linux (EL6-EL7-EL8-EL9)

Executive Summary: ELevate-leapp is the coolest tool I've seen in the past decade. If your computer is connected directly to the internet, then you probably do not need any of the following information. If your computer is behind a firewall and connects to the internet via a proxy-server, then the following information may help.
Clarification: This page started off describing the many ways to migrate off CentOS-7. Since then I've come to realize that the ELevate-leeap tool from the good folks at AlmaLinux is the only reasonably safe method, so if you are in a hurry then click here
Caveat: Linux computer systems of any commercial value should (must?) be covered by a support contract. The following information is very dangerous so should only be attempted by Linux professionals working in colleges and universities, or students and hobbyists working or personal systems. And as always, make sure you have backup copies of everything before you begin.

Overview

Linux terminology

leapp

speculation

Linux terminology

verb scope notes 1 notes 2
update moving from CentOS-7.8 to CentOS-7.9
minor version number changes
upgrade moving from CentOS-7.8 to CentOS-7.9 same as "update --obsoletes" minor version number changes
migrate
 moving from CentOS-7.x to CentOS-8.x (or anything else)
major version number changes

Manual Methods (less safe than leapp)

Lateral scripts (less safe than leapp)

Caveats: 

lateral transfer
 from to tested?
https://wiki.almalinux.org/documentation/migration-guide.html
almalinux-deploy.sh		 CentOS-8
Rocky Linux-9		 AlmaLinux-8
AlmaLinux-9		 yes (note-1)
yes (note-1)
https://github.com/rocky-linux/rocky-tools/tree/main/migrate2rocky CentOS-8
CentOS-9 		Rocky Linux-8
Rocky Linux-9		 no
no
https://linux.oracle.com/switch/centos/
( https://linux.oracle.com/switch/ )
centos2ol.sh		 CentOS-6
CentOS-7
CentOS-8
Rocky Linux-8
Rocky Linux-9 		Oracle Linux-6
Oracle Linux-7
Oracle Linux-8
Oracle Linux-8
Oracle Linux-9		 no
no
no
no
no
  1. In both these tests, I moved the server from the data center to a location where a public internet connection was available, then executed the script. Both migrations worked perfectly but required more than a hour to finish the job.
  2. An alternative to note-1 would be to create alternative versions of curl and wget under /usr/local/bin which would include proxy information before calling the original versions of curl and wget under /usr/bin (caveat: this is an idea rolling around my head. I have never tested this)

Orchestrated Method (safer)

ELevate - leapp (intro)

Never run the following stuff via sudo. If at all possible, log into you system as root over a non-graphical port. I have had updates occasionally fail over a graphical connection but have never seen a fail over a plain-text connection.

CentOS-7 to AlmaLinux-8 (or something else)

Playing it safe with trial converts
  1. steps:
    • grab a spare computer then do a fresh install of the OS you want to migrate from (I used CentOS-7.9)
    • connect it to the public internet
    • follow the Elevate procedure described here: https://wiki.almalinux.org/elevate/ELevate-quickstart-guide.html
    • take notes about what you see AND how many times the system reboots (for this experiment it is important that you are physically located in front of the server's console device)
    • Note: I did a dozen trial conversions before I used leapp on a working server.
  2. repeat step-1 but on a system connected it to a network behind a proxy server
    caveats:
    • anyone who has ever used CentOS-7 already knows you need add 1-2 lines of additional information to file /etc/yum.conf
    • anyone who has ever used CentOS-8 already knows you need add 1-2 lines of additional information to file /etc/dnf/dnf.conf
    • so it should be no surprise that you may need to update various copies of dnf.conf in the leapp namespace
    • but one of those files will be overwritten several times when you run leapp
      • hack-1: requires you to update this file: /etc/leapp/files/leapp_upgrade_repositories.repo
        • insert this line into every labelled stanza: proxy=http://proxy.neilrieck.net:8083
        • optionally, also add this line in every labelled stanza: sslverify=false (this could be dangerous so only use thus if you see CURL (60) error messages)
      • hack-2: requires you to add proxy information to file /var/lib/leapp/el8userspace/etc/dnf/dnf.conf then also run a little script on a second session which will monitor the file then update it whenever it is overwritten (usually 4 times)
      • hack-3: requires you to copy your organization's certificates into the namespace
      caveats:
      • hack-1 works 99% of the time
      • hack-2 (with hack-1) was twice at a remote location. "I think" this might be due to a misconfiguration of my company's proxy server.
        • hack-1 is necessary for leapp preupgrade
        • hack-2 is necessary for leapp upgrade
      • I haven't yet tried hack-3
  3. repeat step-1 but use a backup copy of the server you wish to convert
  4. repeat step-2 but use a backup copy of the server you wish to convert and connect it to a network behind a proxy server
Details (for use behind a proxy sever)
reference: https://wiki.almalinux.org/elevate/ELevate-quickstart-guide.html 
  1. sudo yum update -y
sudo reboot
    sudo yum install -y http://repo.almalinux.org/elevate/elevate-release-latest-el$(rpm --eval %rhel).noarch.rpm
    sudo yum install -y leapp-upgrade leapp-data-almalinux
  2. tweak-1
    # we we will do everything from root (so do not use sudo)
    # become root like so:
    su -
    tweak-2a ( you should skip this step )
    # changes to root's  shell
# ------------------------
    #    changes for leapp
    export LEAPP_PROXY_HOST=http://proxy.domain.com:8083/
    #    changes for curl
export http_proxy=http://proxy.domain.com:8083/
export https_proxy=http://proxy.domain.com:8083/
    tweak-2b ( recommended )
    # changes to your terminal emulator
# ---------------------------------
    # set width to 132
# enable autowrap
# set an ssh keepalive of 60 seconds
    tweak-3 
    # add proxy information to the ELevate repo
    vim /etc/leapp/files/leapp_upgrade_repositories.repo
     
  3. leapp preupgrade
    • this should work with little difficulty
    • if you see errors then you will need to correct them
    • if you see inhibited then you may need to edit the answer file
  4. leapp upgrade     # caveat: do not do this unless to intend to upgrade right now
                      # you might wish to run my helper script (below) before starting this
    • this works almost entirely in the target namespace
    • I have never had this phase go as-is:
      • in one case I was warned about multiple kernels so needed to remove them from CentOS then start over
      • in a second case I was warned about no available update for OpenSSL-devel which was version locked to OpenSSL which was going to be replaced
        • the solution was to yum erase openssl-devel then restart leapp upgrade
      • in a third case I was warned about no available update for Make-devel which was version locked to Make which was going to be replaced
        • the solution was to yum erase make-devel then restart leapp upgrade
    • If you see any "Curl (60)" errors then you will need to do the following ugly hack:
      • log into a second session; become root; execute my workaround script (see below)
      • repeat leapp upgrade on the primary session
  5. reboot
    • be patient here; you may see console messages which indicate that the whole thing could fail, but then the upgrade continues
    • EL7-to-8: you will see some activity for 20 seconds followed by a 4 minute pause
    • EL8-to-9: you will see some activity for 10 seconds followed by a 7 minute pause
  6. wait 45-60 minutes then log in
    • the system will upgrade (30-45 minutes) then reboot
    • the system will begin an SELinux relabeling operation (5-10 minutes) then reboot
  7. vi /etc/dnf/dnf.conf
    1. comment out the exclude line
    2. add a proxy line
  8. yum erase \*leapp\*
  9. yum erase ELevate
My Trial Conversions (EL7 to EL8 to EL9)

EL7 to EL8 EL8 to EL9
Date From To Method score Then To Method score notes
2023-08-31 CentOS-7.9 Rocky Linux-8.8 ELevate pass Rocky Linux 9.2 Elevate fails 1
2023-09-07 CentOS-7.9 AlmaLinux-8.8 ELevate pass AlmaLinux 9.2 Elevate dirty pass 2
2023-09-14 CentOS-7.9 Oracle Linux-8.8 ELevate pass Oracle Linux 9.2 Oracle Leapp pass 3
Notes:
  1. Missing a file 
    file /usr/share/redhat-logos from install of rocky-logos-90.14-1.el9.x86_64 
    conflicts with file from package rocky-logos-86.3-1.el8.x86_64
    # note: you cannot yum erase rocky-logos-86.3-1.el8.x86_64 without causing damage
  2. Rather than rebooting properly, the system comes up in Recovery Mode but no commands appear to work.
    • However, the system boots properly after cycling the power (huh?)
    • Checking the logs, I see a python error associated with main.cli
    • On a later attempt one month later, migrating from to  AlmaLinux 9.3 worked flawlessly
  3. No drama here. This worked like a charm
  4. YOU MIGHT NOT want to migrate all the way to EL9 at this time. Many of my employer's other systems (mostly Windows; some HP-UX) could send files via sftp to EL9, but EL9 could not send files send back. Shifting the EL9 into LEGACY mode did not help.
My workaround script (a leapp helper)

This hack was quickly written during a moment of frustration and panic
Only use this if you see failures associated with curl errors.

# ================================================================
# title  : watch_elx_file_dnf-dot-conf.sh 
# author : Neil Rieck
# created: 2023-08-28
# notes  :
# 1) to be used during the leapp upgrade:
#       el="78" for CentOS-7 to AlmaLinux-8
#       el="89: for AlmaLinux-8 to AlmaLinux-9
# 2) run this on a second session while you upgrade from the first
#    start this program before you type "leapp preupgrade" then
#    let is run until after "leapp upgrade"
# ================================================================
el="78"                           	      # choices: "78" or "89"
LINE1="sslverify=0"
LINE2="proxy=http://your-proxy-server.com:8083/"
# ================================================================
if [[ ${el} == "89" ]]
then
    echo "for use with leapp 8-to-9"
    fs="/var/lib/leapp/el9userspace/etc/dnf/dnf.conf"
else
    echo "for use with leapp 7-to-8"
    fs="/var/lib/leapp/el8userspace/etc/dnf/dnf.conf"
fi
echo "-i-config line1: "$LINE1
echo "-i-config lines: "$LINE2
echo $LINE1    >> ${fs}
echo $LINE2    >> ${fs}
n=1
c=0
prevModificationSeconds=$(date -r ${fs} +%s);           # snapshot 1
for (( ; ; ));
do
    echo "test: "${n}" changes: "${c}
    ((n=n+1));
    watchModificationSeconds=$(date -r ${fs} +%s);      # test
    if [[ ${prevModificationSeconds} != ${watchModificationSeconds} ]]
    then
        echo "-i-oops, the file changed"
        echo $LINE1        >> ${fs} 
        echo $LINE2        >> ${fs} 
        prevModificationSeconds=$(date -r ${fs} +%s);   # snapshot 2
        echo "-i-update complete"
        ((c=c+1));
    fi
    sleep 1
done
echo "-i-exited"
#

Gotchas

I mentioned previously, I performed approximately 30 trial conversions before I did a real-word conversion. I have now done seven real world conversions (six large systems employing 24 CPUs, 128 GB of RAM and 1TB of direct-connect storage) so here are a few caveats.

  1. Check what is running properly BEFORE you do your migration
    systemctl status 
    or
    systemctl status >> snapshot1.txt
  2. Do your migration
  3. Now double check if something that was running is no longer running
    systemctl status 
    or
    systemctl status >> snapshot2.txt
    vim -d snapshot1.txt snapshot2.txt # inspect the differences

It should be no surprise that a some post-tweaking was necessary

 Back to Home
Neil Rieck
Waterloo, Ontario, Canada.