Linux Notes: Real-world problems

1. The information presented here is intended for educational use by qualified computer technologists.
2. The information presented here is provided free of charge, as-is, with no warranty of any kind.

Real-world Linux Problems

Note: all these problems are associated with CentOS-7.2 or higher

1) We cannot install or update software via YUM on one of our CentOS-7 platforms

We have two Linux platforms; one for development and second for production (comment: two platforms may not be enough when using Linux; see the IBM-Managed warning after this section). The recommended approach is to first install (or update) software on the development box. If testing for the next few days (to few weeks) proves that everything is working properly then we would repeat the procedure on the production box. This also keeps both platforms more-or-less in sync.

I wanted to install the tree utility so I logged onto the development box where I entered this command:

	sudo yum install tree

... which worked properly.

Then I repeated this command on the production platform which failed with numerous errors associated with file /usr/libexec/urlgrabber-ext-down which is a python script. What was worse was this: you could not execute "firewall-cmd" or most yum commands including "yum check-update". Investigating further, I noticed that someone had installed python3 then updated the symbolic link so that the python command pulls up python3 rather than python2 (most Linux utilities in 2018 require Python 2.7)

There are only two ways out of this problem (remember that this is an active business system).

1. modify the first line (the shebang or sharp-bang line) of broken system scripts from this "#! /usr/bin/python" to this "#! /usr/bin/python2"
2. modify the symbolic link for "python" and point it at the symbolic link "python2" which probably points to a directory like "python2.7". This will restore the system to its previous functionality but could break something if newer customer scripts required python3 to be the default. So before modifying the symbolic link, modify the shebang line of customer scripts like this:
   "#! /usr/bin/python" to this "#! /usr/bin/python3"

Tip: if this is an emergency, just make minimal changes. For now, just modify the scripts for whatever is broken (eg. yum or firewall-cmd). But you will eventually need to put everything back to a pristine state. If your stuff needs python3 then you need to rely upon shebang. I have no idea why the Linux maintainers didn't do this for their scripts requiring Python2.

#
# 1) this is an example of a good working setup with two versions of python installed
# 2) notice (on the first result line) that python is pointing to python2
#    and python2 is pointing to python2.7
# 3) if yum and firewall-cmd require python2 then you would have thought the
#    developers would have modified the shebang lines of those utilites
#
$ cd /usr/bin
$ ls pytho* -la
lrwxrwxrwx. 1 root root     7 Jan 12 15:25 python -> python2
lrwxrwxrwx. 1 root root     9 Dec 20  2016 python2 -> python2.7
-rwxr-xr-x. 1 root root  7136 Nov  5  2016 python2.7
lrwxrwxrwx. 1 root root     9 Apr 12  2017 python3 -> python3.4
-rwxr-xr-x. 2 root root 11312 Jan 17  2017 python3.4
lrwxrwxrwx. 1 root root    17 Apr 12  2017 python3.4-config -> python3.4m-config
-rwxr-xr-x. 2 root root 11312 Jan 17  2017 python3.4m
-rwxr-xr-x. 1 root root   173 Jan 17  2017 python3.4m-config
-rwxr-xr-x. 1 root root  3366 Jan 17  2017 python3.4m-x86_64-config
lrwxrwxrwx. 1 root root    16 Apr 12  2017 python3-config -> python3.4-config

2) Never use a graphical console to update Linux software

I have experienced several instances where updating software though the graphical interface fails for some reason then breaks the graphical interface (or the whole system). It should not surprise anyone that updating gnome-session, or any of its dependencies, might disturb the very session that is running yum or rpm

So if you are on the system console (which is almost always a VGA monitor) and want to move to non-graphical session, try one of the following keystrokes:

The only other way to safely disable graphics is to lower the runlevel of your system to 3. (but only do this if you are certain that you won't kill some process currently needed by your customers). Alternatively, use ssh to log into your system via the network then execute yum on that session.

3) Using Windows to access a Linux remote

The self-help blogs really fall down on this one because the only secure way to do this is to tunnel x-sessions over SSH. But whenever anyone on a self-help blog asks how to do this only using SSH, some idiot will chime in with a procedure on how to do it using VNC, RealVNC, TigerVNC or Vino which are all insecure.

To make matters worse, setting up a remote graphical session is almost impossible (at least under certain circumstances like Windows -> RedHat/CentOS) because GNOME3 contains 3-d extensions not found in most Windows clients. The best way out of this is to setup RedHat/CentOS on a machine at the client end then use it to connect to the desired Linux platform.
 
comment: some conspiracy-minded people think this change was deliberately done to stop support professionals from from using Windows as their default platform to support all others. 

Xming

• xming is a simple tool which is used in conjunction with a terminal emulator like: Tera Term or PuTTY

CygWin and CygWin/X

• documentation: https://x.cygwin.com/docs/
  • https://x.cygwin.com/docs/ug/setup.html (jump to chapter 2.15)
  • x11 documentation tends to only speak of servers at each end (not true client server as one would normally think).
  • On the client side you must do this:
    • start an x11-server
    • connect through to the far-end
    • start a client over there which will open one, or more, x11 sessions back to your local server
• Caveat: Caveat: do not install cygwin/x without first reading the build instructions (a full build will not produce what you want and will waste your time and bandwidth)
• connecting:

  action	: start your local x-server (on the start menu)
  	: task will appear on your horizontal task bar
  
  action	: start xterm
  	: it is one of the items associated with the x-server icon on the task bar
  
  type	: ssh -X name@fully-qualified-domain-name 
  	  (replacing "-X" with "-Y" is even better)
  action	: wait for the password prompt then enter it
  
  type	: xterm &
  action	: a new window should open
  
  type	: /gnome-weather &
  action	: a graphical weather app will pop up	
  
  type	: /gnome-session --disable-acceleration-check &
  action	: a new window manager should open (but does not)
  	: 1) the switch shown is only available in gnome starting with version 3.16
  	: 2) without it you may see something like this: 
  		"Oh no! Something has gone wrong.
  		A problem has occurred and the system can't recover.
  		Please log out and try again."
  	: 3) gnome3 requires advanced graphics so may never work this way;
  		try a desktop other than gnome 

4) Recovering a failed YUM update (2018-01-xx)

• I recently had a yum update fail on the graphical interface (see note #2 above regarding why you should never do this)
• YUM was in the middle of updating ~ 1200 items when the GUI collapsed. I waited 8-hours then rebooted. The system came up then attempted to start a graphics console which failed (could see an arrow cursor and nothing else)
• I typed CTL ALT F2 and was able to login as root
• more details to come ...

5) a recent (2018-01-xx) YUM update broke our development box (ouch!)

• we use sqlcmd from msodbcsql (ms odbc sql) to access an old database (SQL Server 2005) running on an old OS (Windows Server 2003). The platform is located in another city and province; they never installed mandatory patches; I have no faith in them upgrading any time soon; we have been able to connect for 14-months.
• a recent yum update on our development platform broke msodbcsql (the production platform continues to work)
• checking logs on our production platform shows that we can only connect to the Windows box only using the sslv2 protocol ???
• testing on PROD

  # notes:
  # 1) well-known port 1433 is reserved for Microsoft SQL Server
  # 2) SQL Server 2005 appears to support ssl2 but nothing higher
  # 3) contrary to popular belief, as soon as you specify a username and password in your
  #    ODBC connect string (via sqlcmd) then the initial handshake will be encrypted  
  #
  #	this passes 
  #
  openssl s_client -debug -state  -connect ip-address:1433 -ssl2
  #
  #	this fails
  #
  openssl s_client -debug -state  -connect ip-address:1433 -ssl3

• Platform Differences

  Platform	CentOS Notes	OpenSSL version	OpenSSL Notes
  Production	CentOS 7.3 (built 14-months ago)	OpenSSL-1.0.1e	ssl2 is supported
  Development	CentOS 7.4 (yum updated 2018-01-29)	OpenSSL-1.0.2k	ssl2 has been disabled prior to build

• The simplest way out (at this time) is to build a fully-functional new version of OpenSSL-1.0.2k (in a local folder) then then copy the binary to "/usr/bin" after renaming the old version (just being paranoid here). A good friend provided me with this link on the Ubuntu site (a Debian flavor) which seems to work properly on CentOS-7
• https://wiki.openssl.org/index.php/Compilation_and_Installation
• https://askubuntu.com/questions/893155/simple-way-of-enabling-sslv2-and-sslv3-in-openssl
• Building a new version of OpenSSL (only in your own folder for now)

  wget https://openssl.org/source/openssl-1.0.2k.tar.gz
  tar -xvf openssl-1.0.2k.tar.gz
  cd openssl-1.0.2k/
  # --prefix will make sure that make install copies the files locally instead of system-wide
  # --openssldir will make sure that the binary will look in the regular system location for openssl.cnf
  # no-shared builds a mostly static binary
  ./config --prefix=`pwd`/local --openssldir=/usr/lib/ssl enable-ssl2 enable-ssl3 no-shared
  make depend
  make
  #
  # these next two steps are not required if openssl-1.0.2k already exists on your system.
  #
  make -i install
  sudo cp local/bin/openssl /usr/local/bin/
  #
  # test the newly created binary like so:
  	./apps/openssl s_client -debug -state -connect ip-address:1433 -ssl2
  # ...remembering that many other destinations will no longer accept ssl2
  # then rename the old binary (paranoid):
  	mv /usr/bin/openssl	/usr/bin/openssl-old
  # then copy the new binary:
  	cp ./apps/openssl	/usr/bin/openssl
  #	

• Optional steps:
  1. for proper sslv23 handshaking (especially true when you only have an older ODBC connect string with no way to specify ssl parameters) you need to also include the switch no-tls-1-2-client
  2. building with the no-shared switch is necessary for testing your binary in a non-standard location but will result in the program being ~ 6 times larger (around 3.4 MB). Changing to switch shared will result in a binary size of ~ 600 KB)

6) a recent (2018-06-xx) update broke our production box (a major pain!)

One of our developers was experiencing problems developing a new LDAP-based application. So he invoked YUM to update LDAP on our production box. The big problem here is that the update was done in a careless way (without reading all the release notes). So the LDAP update also updated OpenSSL for the whole system so now we can no longer connect to that older Microsoft platform in Montreal. (see: this previous note)

It now appears that we will need to install a third CentOS platform whose only purpose would be to reach through to the older Microsoft platform. This platform would need to be modified so that it could never been updated.

7) Something is overwriting file "/etc/resolv.conf"

This problem is so weird that I'll stick to bullet points

• Last month I set up a new CentOS-7 platform for use in a project we will turn up in Feb of 2019
• I logged onto the console then used a GUI session to setup several network connections which included three corporate DNS references.
• From this point on, logging into that platform was slow (10 second delays). This included using the "su" command once you were logged in.
• One of my peers in a city 100km away discovered a typo in file "/etc/resolv.conf" which he fixed using a non-GUI login. The delays disappeared.
• However, just logging into the console with my GUI session, or rebooting the box, caused the manual repair to be overwritten (10 second delays were back)
• One needs to remember that Linux began its life as a personal computing platform and many programmers who work in this ecosystem still see it that way. This means there are all kinds of special hooks put in to support the GUI user.
• If you drop this text "centos networkmanager overwrites resolve.conf" into a Google search then you will get a bunch of hits like this:
  https://ma.ttias.be/centos-7-networkmanager-keeps-overwriting-etcresolv-conf/
  where we learn that this has been going on since CentOS-6.
• Apparently you are not supposed to enter DNS addresses into the GUI dialog for each NIC. If you do, the Network Manager will continue to copy this information from NIC config files then overwrite "/etc/resolf.conf"
• There are two ways you can fix this problem:
  1. use an editor to modify the appropriate network settings file(s) which is not recommended in case you make a typo
     • just "/etc/sysconfig/network-scripts/ifcfg-eno1" in my case
  2. use the GUI to remove all DNS references from all your active network configs then use an editor to edit "/etc/resolv.cong". My file now looks like this:
     •  

       # NOT Generated by NetworkManager
       # /etc/resolv.conf
       options timeout:1
       options attempts:1
       options rotate
       options no-check-names
       search on.bell.ca
       nameserver 142.182.48.71
       nameserver 142.182.48.105
       #nameserver 142.113.87.152

     • you might consider making a copy of this file like so:

       cd /etc
       cp resolv.conf resolv.conf-copy

8) Our console device is totally dead

We are running CentOS-7.2 on two HP-DL385-g6 servers (one PROD, one DVLP) and both have been running for 24 months without a reboot. These are older hardware platforms so I have been preparing to cut the whole thing over to to two new servers (HP-DL385p-gen8) next month. I just noticed I can't access one of the older via the console.

I've tried everything (short of rebooting) including a restart of various services (eg. "systemctl restart gdm.service") but it seems that the graphics card itself is locked up somehow.

I need to point out that we can do anything else we want via a remote ssh terminal session over the network. In fact, the customers are unaware of fact that anything is wrong. So let me advise everyone to ensure that at least one network port is configured -AND- that the firewall has been configured to allow ssh connections so you be able to manage your platform if your VGA console is fubared.