Linux Notes: SELinux (introduction for hackers-developers)

  1. The information presented here is intended for educational use by qualified computer technologists.
  2. The information presented here is provided free of charge, as-is, with no warranty of any kind.
Edit: 2022-08-27

Project 1: I need one directory write-enabled for use by FTP

Introduction (how a simple project went sideways)

project 1: overview

Our data-center backups (2016-2018):

+-------------------------+   +--------------------------+
| server : HP rx2800-i2   |   | server : HP DL385p-gen8  |
| OS     : OpenVMS-8.4    |   | OS     : CentOS-7        |
| client : MariaDB-5.5-60 |   | server : MariaDB-10.3.11 |
| net-1  : TCP/IP         +<--+ net-1  : TCP/IP          |
| net-2  : TCP/IP         |   +--------------------------+
| net-3  : DECnet         +<--- 4 other OpenVMS systems
| net-4  : TCP/IP         +---> Windows-7 PC (backup host)
+-------------------------+
  • Early every morning:
    • four smaller OpenVMS nodes (HP rx2660) employ the DECnet protocol on a private 1 Gb/s Ethernet (net-3) to drop their backup save-sets onto a large OpenVMS node (HP rx2800-i2)
    • we also backup our MariaDB database over a 1 GB/s Ethernet (net-1)
  • A little later, we employ FTP to copy the save-sets over a private 1 Gb/s Ethernet (net-4) to a Windows-7 PC which has a 1-TB drive connected via a USB cable
    • these drives are NTFS formatted
    • We have 30 of these disks and a technician is responsible for replacing the drive every day then making sure it is transported to an off-site location

Changes in 2019:

  • On January of 2019, corporate security asked us to remove the Windows-7 PC and replace it with something else.
    comment: "I think" they were worried that the OS was no longer supported by Microsoft but did not want to pay to upgrade to Windows-10
  • Since we have access to a lot of older HP-DL385 servers so it was no big deal to grab one then install CentOS-7.5 along with an FTP server.
  • The whole thing would have worked in no time at all on UNIX but did not because of SELinux

Project 1: details

initial steps (CentOS-7)
Command(s) Comments
yum install vsftpd
vi /etc/vsftpd.conf
install an ftp server configure the settings file
firewall-cmd --permanent --zone public --add-service ftp firewall-cmd --reload prep the firewall
systemctl stop vsftpd.service systemctl start vsftpd.service systemctl status vsftpd.service
systemctl enable vsftpd.service auto-start this service during reboot
yum install epel-release yum install ntfs-3g -y install ntfs software

caveat: on 2022-03-31 I realized that proftpd might be a better choice (at least for some applications) than vsftpd.

  • First off, there is much more SELinux information (some of it can be found in the man pages while more can be found in the config file).
  • Secondly, proftpd can be configured to execute scripts after a file transfer via mod_exec (provided you are not using chroot).
  • Running scripts is only possible with vsftpd by employing secondary products like incrond or inotify-tools (which add an additional layer of complexity to application support)
  • Final  points:
    • almost everyone prefers the chroot technique for security reasons
    • proftpd is the preferred solution according to Red Hat
    • the authors of proftpd warn you not to use mod_exec
commands to see connected disks
Commands Comments
fdisk -l  
fdisk -l /dev/sd*  
ls -la /dev/disk/by-label/  
ls -la /dev/disk/by-label/BKUP* see all disks with a label beginning with BKUP
overview of SELinux

SELinux was developed by America's  NSA (National Security Agency) and combines two approaches to security. (read on)

  • DAC (discretionary access control) is implemented in most systems by file protection bits as well as assigning owner and group information. All three can be seen as red in this display:
    #----------------------------------------------------------------------------
    #	inspect a file
    #	note: DAC information is shown in red
    #----------------------------------------------------------------------------
    [root@localhost ~]# ls -la /icsis
    total 8
    drwxr-xr-x.  3 root root   16 Jan 25 11:11 .
    dr-xr-xr-x. 18 root root 4096 Jan 25 11:30 ..
    drwxrwxrwx.  1 root root 4096 Sep 20 17:02 win
    [root@localhost ~]#
    #----------------------------------------------------------------------------
    #	inspect a process
    #	note: owner in red
    #----------------------------------------------------------------------------
    [root@localhost ~]# ps -ef | grep vsftp
    root 6051 1  0 Jan25 ? 00:00:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
  • MAC (mandatory access control) is additionally implemented in Linux systems via a product named SELinux (Security Enhanced Linux) shown in red:
    #----------------------------------------------------------------------------
    #	inspect a file (append "Z" to also see SELinux data)
    #	notes:
    #	1) MAC information is shown in red
    #	2) breakout:
    #		system_u is a user context
    #		object_r is a role context
    #		root_t   is a type context
    #		s0       is a security level
    #	3) notice that the second line has a context type of "fusefs_t"
    #----------------------------------------------------------------------------
    [root@localhost ~]# ls -laZ /icsis
    drwxr-xr-x. root root unconfined_u:object_r:default_t:s0 .
    dr-xr-xr-x. root root system_u:object_r:root_t:s0      ..
    drwxrwxrwx. root root system_u:object_r:fusefs_t:s0    win
    #----------------------------------------------------------------------------
    #	inspect a process (append "Z" to also see SELinux data)
    #	note: MAC information is shown in red
    #----------------------------------------------------------------------------
    [root@localhost ~]# ps -efZ | grep vsftp
    system_u:system_r:ftpd_t:s0-s0:c0.c1023 root 6051  1  0 Jan25 ...
    ... 00:00:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
  • With SELinux enabled and in Enforcing mode, the Linux system first checks the DAC rules then also checks the MAC policy rules so you are now getting two levels of protection.
  • Why is MAC important? Consider one hypothetical example where the webserver executable /usr/sbin/httpd was started with root privileges. A misconfigured server (either by accident or hack) now can access any file on the system as root. SELinux policies prevent this by default.
    • Note that the webserver could still be hacked. But any damage will be contained to the webserver portion of the server. Likewise, a virus or worm will be contained to this section of the server. Your system folders and files will remain unaffected.
  • Some Linux-based software products (Drupal is one example) will instruct the user to either "disable SELinux" or "permanently place SELinux into Permissive mode". Never do this. Since SELinux was developed by the NSA (National Security Agency) for their own servers then put into the public domain, you would be disabling a very powerful second level security safety net developed by government spooks. I suggest you take the time to educate yourself about this product if you support Linux servers as part of your professional life.
  • To learn more:

Now for a little hacking

part 1: adding optional man pages

You will not learn SELinux in one day. In fact, there are large tomes available on Amazon dedicated to this single topic; but you might be able to learn just enough about this to get yourself over the hump provided you are willing to do a little hacking. So try these two commands:

Commands Comments Additional Info
man ftpd_selinux view SELinux info specific to ftpd All FTPd programs are required to follow these rules
man httpd_selinux view SELinux info specific to httpd All HTTPd programs are required to follow these rules

If neither one of these commands worked but you would like them to then follow these steps:

Commands Comments
yum install selinux-policy-devel  
sepolicy manpage -a -p /usr/local/man/man8 generate new manpages
mandb integrate the new manpages into your index

At this point commands like "man ftpd_selinux" should work properly. Be sure to read the whole thing taking special note of any predefined sebooleans (these are topic-specific boolean variables stored in SELinux)

part 2: see what's already found in my SELinux implementation
Commands Comments
semanage boolean -l list all booleans
semanage boolean -l | grep ftp list booleans specific to ftp and sftp
part 3: let's make a few changes (this works but is not recommended)
Commands Comments
mkdir /icsis/win
chmod 777 /icsis/win
this will be my mount point (where my USB-DISK will be connected)
note: only need to do this once
setsebool ftpd_use_fusefs 1 since my USB-DISK is being attached by fusefs (see blue text below)
this change will allow all FTPd programs to access directories attached to path /icsis/win
#------------------------------------------------------------
#	inspect a file 
#	tack on "Z" to also see SELinux data
#	note: MAC stuff in red and blue
#------------------------------------------------------------
[root@localhost ~]# ls -laZ /icsis
drwxr-xr-x. root root unconfined_u:object_r:default_t:s0 .
dr-xr-xr-x. root root system_u:object_r:root_t:s0      ..
drwxrwxrwx. root root system_u:object_r:fusefs_t:s0    win
#------------------------------------------------------------
part 4: an alternate approach (recommended)
Commands Comments
ls -la /dev/disk/by-label/BKUP* take notice where CentOS auto-mounted my USB-DISK
probably will be /dev/sdb1 depending upon how many other drives are present
umount /dev/sdb1 dismount my USB-DISK from where ever it is right now
mount -t /dev/sdb1 /icsis/win mount it in a place where OpenVMS expects it
semanage fcontext -a -t public_content_rw_t "/icsis/win(/.*)?" tell SELinux that this location is sanctioned for read+write
restorecon -F -R -v /icsis/win necessary voodoo (copies info from SELinux back to the file system)
setsebool -P ftpd_anon_write 1 an optional "hall pass"

At this point the attached USB-DISK can be written to via FTP

caveat: on 2022-03-31 I realized that the account associated with this process should be added to "semanage login" with a label of "user_r". In fact, everyone logging into your system should be constrained in this way. See general-selinux-tips further down this page.

Project 2: I need one directory write-enabled for use by Apache/httpd

  • I was working a Python demo (named: save_file.py) which shows how to present "a file browser" then implements an "upload". Now I should mention that everything is set up properly as far as UNIX-Linux file protections are concerned but SELinux was helping to keep my system secure so did not allow it.
  • my python script is trying to write to this folder: /var/www/tmp
  • here's how things looked originally (note: to view SELinux info we append an uppercase "Z" onto the switches of the "ls" command)
    [prompt]# pwd
    /var/www
    [prompt]# ls -laZ
    drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 .
    drwxr-xr-x. root root system_u:object_r:var_t:s0       ..
    drwxrwxrwx. root root system_u:object_r:httpd_sys_script_exec_t:s0 cgi-bin
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_rw_content_t:s0 click_log
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 css
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 documents
    -rwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 favicon.ico
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 fonts
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 fragments
    drwxrwxrwx. root root system_u:object_r:httpd_sys_content_t:s0 html
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 images
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 js
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 mam
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 tmp
    [prompt]# 
  • here are my SELinux commands along with the changes
    [prompt]# pwd
    /var/www
    [prompt]# semanage fcontext -a -t httpd_sys_rw_content_t "/var/www/tmp(/.*)?"
    [prompt]# restorecon -R -v /var/www/tmp
    [prompt]# ls -laZ
    drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 .
    drwxr-xr-x. root root system_u:object_r:var_t:s0       ..
    drwxrwxrwx. root root system_u:object_r:httpd_sys_script_exec_t:s0 cgi-bin
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_rw_content_t:s0 click_log
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 css
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 documents
    -rwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 favicon.ico
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 fonts
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 fragments
    drwxrwxrwx. root root system_u:object_r:httpd_sys_content_t:s0 html
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 images
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 js
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_content_t:s0 mam
    drwxrwxrwx. root root unconfined_u:object_r:httpd_sys_rw_content_t:s0 tmp
    [prompt]# 
  • Now my python script works properly (yay!)

Project 3: which is really a FUBAR fix

  • If you are using rsync (or tar) to do live backups then you had better add switches to also copy meta-data which also includes SELinux information.
  • with rsync you need to use either "-X" or "--xattrs"
  • see this example: linux_notes: rsync

Project 4: procmail is broken

  • I am using procmail on several accounts to intercept email then start a python program to read and process it
  • With SELinux in permissive mode I am seeing this:
    • SELinux would have blocked execution of these scripts
    • SELinux would have blocked access to port 3306 (needed to open a connection to MySQL/MariaDB)
    • SELinux would have blocked access to port 25 (needed to email a reply)

Solution 4a (this first hack is NOT the correct solution)

Caveat: although the following works, it is not the correct way to solve the problem. Why? Relabeling the files for ownership by procmail_t means that they will not be accessible by user esppats (although this is not a real user). I am leaving this here because it is instructive.

type:	man procmail_selinux							#
#
# 1) the first three lines update SELinux config
# 2) "sequencer" is a small binary program which needs to run
# 3) "munpack" is a small binary program which needs to run
# 4) the fourth line is just a little hacking
# 4) the fifth line changes file labeling based upon SELinux config
#
type: semanage fcontext -a -t procmail_exec_t "/home/esppats/espp(.*\.sh)?" # scripts type: semanage fcontext -a -t procmail_exec_t "/home/esppats/sequencer" # this program type: semanage fcontext -a -t procmail_exec_t "/home/esppats/munpack" # this program type: semanage fcontext -l | grep esppats # hacking type: restorecon -FRv /home/esppats # now apply to files
#
# now allow two more ports
#
type: sealert -a /var/log/audit/audit.log # magic happens type: ausearch -c 'espp_step2.sh' --raw | audit2allow -o junk.txt # -+- just hacking type: cat junk.txt # -+ #============= procmail_t ============== #!!!! This avc is allowed in the current policy allow procmail_t mysqld_port_t:tcp_socket name_connect; #!!!! This avc is allowed in the current policy allow procmail_t smtp_port_t:tcp_socket name_connect; type: ausearch -c 'espp_step2.sh' --raw | audit2allow -M my-esppstep2sh # -+- not hacking
type: cat my-esppstep2sh.te # | type: semodule -i my-esppstep2sh.pp # -+
#
# problem solved
#

Solution 4b (this second hack is better)

#
# begin by removing everything related to the previous hack
#
type: semanage login -d esppats # might fail if nothing found
type: semanage fcontext -D "/home/esppats" # remove everything matching this path
type: restorecon -FRv /home/esppats # all files relabeled unconfined_u
type: semanage fcontext -l | grep "/home/esppats" # double check
#
# new solution starts here
#
type: semanage login -a -s user_u esppats # define this user type: restorecon -FRv /home/esppats # all files now labeled user_u event: generate some activity to cause SELinux to log AVC messages in permissive mode type: cat /var/log/audit/audit.log | audit2allow -m esppats > esppats.te (this is promiscuous; running through grep will NOT help) type: cat esppats.te (cuz not everything is usable; might want to remove http stuff) --------------------------------------------------------------------- contents start module esppats 1.0; require { type httpd_sys_script_exec_t; type user_home_t; type smtp_port_t; type mysqld_port_t; type httpd_sys_script_t; type procmail_t; class tcp_socket name_connect; class dir { add_name remove_name write }; class file { create execute execute_no_trans rename unlink write }; } #============= httpd_sys_script_t ============== allow httpd_sys_script_t httpd_sys_script_exec_t:dir { add_name remove_name write }; allow httpd_sys_script_t httpd_sys_script_exec_t:file { create rename unlink write }; #============= procmail_t ============== allow procmail_t mysqld_port_t:tcp_socket name_connect; #!!!! This avc can be allowed using the boolean 'nis_enabled' allow procmail_t smtp_port_t:tcp_socket name_connect; allow procmail_t user_home_t:file { execute execute_no_trans }; --------------------------------------------------------------------- contents end
type: vim esppats.te
(to remove the stuff in red) type: make -f /usr/share/selinux/devel/Makefile esppats.pp type: semodule -i esppats.pp
#
# problem solved
#
type: semodule -d esppats # disabling module causes problem to return
type: semodule -e esppats # enabling module solves problem

Project 5: Python3 needs access to __pycache__

  • normally when Apache executes a python script (name.py) for the first time under '/var/www/cgi-bin/' it calls a JIT compiler to create a compiled version (name.cpython-36.pyc) for execution.
    After all that, the JIT compiler will attempt to save the file under '/var/www/cgi-bin/__pycache__/' so that it doesn't need to be repeatedly compiled. SELinux is blocking this but here is a solution.
  • First you need to run your system for a few minutes with SELinux in permissive mode to build up log messages in the audit file
type:   ausearch -c 'python3' --raw | audit2allow -M my-python3	# magic happens
        ******************** IMPORTANT ***********************
        To make this policy package active, execute:
        semodule -i my-python3.pp
        (note: two files were just created: my-python3.te followed by my-python3.pp)
type:   cat my-python3.te                                       # is it what you want?
type:   semodule -i my-python3.pp                               # YES
---------------------------------------------------------------- otherwise
type:   vi my-python3.te                                        # remove want you don’t want
type:   rm my-python3.pp                                        # delete old .pp
type:   make -f /usr/share/selinux/devel/Makefile my-python3.pp # create new .pp
type:   semodule -i my-python3.pp                               # install it

Can't restart Apache with new certificate files

  • Every August, I am tasked to renew a multi-domain SSL certificate then install it on 8 servers running in our department
  • The servers are all identically configured so I cannot explain why the update procedure failed on only one of them (CentOS) but now Apache is down and will not start
  • Here is an extract from /var/log/messages
    Aug 26 14:55:32 kawc3v systemd: Stopping The Apache HTTP Server...
    Aug 26 14:55:33 kawc3v systemd: Stopped The Apache HTTP Server.
    Aug 26 14:55:33 kawc3v systemd: Starting The Apache HTTP Server...
    Aug 26 14:55:33 kawc3v httpd: AH00526: Syntax error on line 114 of /etc/httpd/conf.d/ssl.conf:
    Aug 26 14:55:33 kawc3v httpd: SSLCertificateKeyFile: file '/etc/pki/tls/private/kawc96_20220822.key' does not exist or is empty
    Aug 26 14:55:33 kawc3v systemd: httpd.service: main process exited, code=exited, status=1/FAILURE
    Aug 26 14:55:33 kawc3v systemd: Failed to start The Apache HTTP Server.
    Aug 26 14:55:33 kawc3v systemd: Unit httpd.service entered failed state.
    Aug 26 14:55:33 kawc3v systemd: httpd.service failed.
    
  • Now I need to point out that the key file exists, has the correct protection bits, and both group and owner are set to root
  • I wasted a little more time than I should have on this one until I remembered that corporate security had installed an alternate logger which was programmed to throw away SELinux messages (which means I was not seeing any)
  • So here is the two line solution (first su to root or prefix with sudo as required)
    restorecon -F -R -v /etc/pki/tls/certs
    restorecon -F -R -v /etc/pki/tls/private 
  • What does restorecon do? It uses the SELinux database to see what the files in those directories should be set to then sets them. 

General SELinux Security Tips

Overview

First type these two commands:

#
# legend: problems in red
#
[prompt]sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: permissive Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 31 # [prompt]sudo semanage login -l [sudo] password for neil: Login Name SELinux User MLS/MCS Range Service __default__ unconfined_u s0-s0:c0.c1023 *
ungar user_u s0-s0:c0.c0 *
admin sysadm_u s0-s0:c0.c1023 *
rootish system_u s0-s0:c0.c1023 * root unconfined_u s0-s0:c0.c1023 * system_u system_u s0-s0:c0.c1023 *

What is the security level of your Linux system?

SELinux
status
Current
Mode
Any
Unconfined
Users?
Result Security
Level
disabled

your system is no more secure than a typical UNIX system of the 1990s unix
enabled permissive
your system is no more secure than a typical UNIX system of the 1990s unix
enabled enforcing yes still better than UNIX because some processes (like httpd) are
confined (even though they are not found in semanage login)
u+1
enabled enforcing only root better because most users are confined in a way similar to vendor installed stuff like httpd u+2
enabled enforcing no best because now everything and everyone is confined u+3

The dirty details:

  1. Is your Linux system secure? Type: sestatus
    then take note of "SELinux Status" and "Current Mode" (see yellow text above)
    If SELinux is not running, or is running in permissive mode then your system is no more secure than a UNIX system of the 1990s.
  2. If SELinux is running in enforcing mode then type: semanage login -l
    then ask yourself "are all interactive (non-web) system users defined here? (if not, is the default account linked to a confined role?)"
    note: almost all software installed via yum (like httpd) runs in a confined domain when SELinux is enabled and running in enforcing mode
  3. DID YOU KNOW?
    1. that any accounts labelled as "unconfined" might be able to type "sudo" or "su"
      YOU WILL BE IN TROUBLE if anyone can learn any privileged passwords either by hacking, brute force cracking, typing or talking (loose lips sink ships)
    2. most confined accounts do not have access to "sudo" or "su"
      SO YOU HAVE JUST ELIMINATED A LARGE FRACTION OF MOST PASSWORD EXPLOITS
    3. anyone confined as sysadm_u will have access to "sudo" but not "su"
      note: but these accounts could be selectively further restricted to not allow any changes to SELinux (so set aside a special account which is the only one allowed to make changes to SELinux)
    4. anyone confined as system_u will have access to "sudo" and "su"
      note: this account could be further restricted to not allow access from anywhere except a directly connected console device (most likely behind a locked door)
  4. So consider some of these changes:
    1. every interactive account should have an entry in semanage login so do one of the following
      modify the default account named __default__ from unconfined_u to user_u relaxed
      modify the default account named __default__ from unconfined_u to guest_u stricter
      delete the default account named __default__ strict
    2. consider modifying the root account from unconfined_u to system_u
    3. on a fully locked down system you should use yum/dnf to remove any optional SELinux libraries
      • eg. removing package setools will remove commands like sesearch which may be useful to an inside hacker
  5. CAVEATS:
    • GENERAL ADVICE:
      1. whether using 'permissive domains' or running SELinux 100% in permissive mode, SELinux will (by default) be writing SELinux failure messages to these two locations but taking no action
        • '/var/log/messages'
        • '/var/log/audit/audit.log'
      2. So clean logs usually mean it is safe to switch SELinux from permissive to enforcing (but watching for a week is better than watching for a day). Use these logs to cleanup your system.
      3. prior to making initial SELinux changes, ensure that your system is set to reboot with SELinux in permissive mode. If you do not, and something goes horribly wrong, then all you need to do is a quick reboot by cycling the power (in words from the comedy program The IT Crowd you will have to try "turning it off then turning it back in again").
        Failure to follow this warning means that you will need to learn how to disable SELinux from the boot menu. Not impossible but also not fast -AND- may be behind a locked door in a remote location.
      4. After a short time you will want to set up SELinux so the system always comes up in enforcing mode.
    • Before step 4-1 above (SELinux changes to the __default__ account) ensure that your system admin accounts are properly configured as wheel accounts -AND- associated with SELinux role sysadm_u because they will not have traditional access to "su" or "sudo" after you switch SELinux into enforcing mode.
      Failure to follow this warning means that you will need to learn how to disable SELinux from the boot menu. Not impossible but also not fast -AND- may be behind a locked door in a remote location.
    • Before step 4-2 above (SELinux changes to the the root account) ensure that your system is set to reboot with SELinux in permissive mode.
      Failure to follow this warning means that you will need to learn how to disable SELinux from the boot menu. Not impossible but also not fast -AND- may be behind a locked door in a remote location.

External Links


 Back to Home
Neil Rieck
Waterloo, Ontario, Canada.