Linux Foundation Certified SysAdmin

I’ve been working with Unix/Linux for 25+ years but I found that this exam was extremely challenging. The breadth of material that could come up is such that studying for this exam was practically a full-time job.

I found two courses that proved invaluable during my preparation:

  • Linux Foundation Certified System Administrator (LFCS) by Andrew Mallett, on Pluralsight. This course was being updated as I was taking it, with new content being added on a daily basis. The course covered far more than what was expected for the exam but it was a great learning tool for Linux in general. When I started the course, I did not see the lab options and simply built my own Linux systems to use for practicing. However, when I neared the end of the course, there were pre-built lab courses available as well. I did not make use of these as they were unnecessary for me at this point.
  • Linux Foundation Certified Systems Administrator – LFCS by Aaron Lockhart, on Udemy. This course was far more focused on the exam itself than the Pluralsight course, which was great to go through right before sitting the exam. This course also came with pre-built labs that I did make use of and they worked quite well. The instruction was put together very well and the depth of each topic was great.

There was obviously some overlap between these two courses but I did feel that each had a few topics that were done better in that particular course. Overall, the Pluralsight course seemed more comprehensive but ended up having more material that was not relevant to the exam. I liked this from an overall perspective but if you’re just looking for an exam resource, it might lead you to studying more than is absolutely necessary.

As for the exam itself, I found some of the questions to be very difficult as you were expected to know what tool or function of the Linux operating system would be needed to complete a specific goal. There was very little that was laid out so clear that you couldn’t help but no where to go to get started…i.e. “Configure a file system with these parameters”. Questions were typically worded in a fashion that a non-technical person might relay a request…i.e. “My account needs to be able to write to xyz directory”, as a simple example. I liked this approach as it was a much more real-world example of the types of tasks a with which a Linux administrator will be dealing. I was worried going in to the exam that there would be multiple issues that would need to be addressed in a single question/scenario but this was unfounded. Each question/scenario was a single problem to solve or configuration to implement.

Aside from the two courses I mentioned, the best thing you can do to study for this exam is get as much hands-on time as possible and take lots of notes when you come across a topic you don’t understand so well. Hold on to whatever kind of lab environment you have…whether this is provided via a course or something you stand up on your own. Keep practicing right up to the very end. I thought I was a fairly-well-versed Unix/Linux administrator but quickly found that there was more that I didn’t know than I did. It also didn’t help that my last formal Linux training was close to 10 years ago and a lot has changed in that time. I needed to get up to speed on new ways of doing things that weren’t available years ago.

This might not prove to helpful, but it’s a dump of my “notes” that I took while studying. Some of what’s in here are things that I wasn’t too sure about or just wanted to jot down to keep it in my head better. As the exam was labeled as being platform agnostic, you’ll likely see similar commands for both Ubuntu and CentOS.


!! - last command
!<chr> - last command that starts with <chr>
Ctrl-a - go to beginning of line
Ctrl-e - go to end of line
Ctrl-l - clear screen
Ctrl-r - search command history, can continue ctrl-r to keep finding more of the same string
> <file> - overwrites file with nothing
&> <file> - send stdout and stderr to <file>
\<cmd> - run <cmd> unaliased
Esc-. - brings up last directory/file accessed

find <something> -exec cp {} <dir> \; - copy found objects to <dir>
find <something> -exec delete - delete found objects

locate - like find but uses an index (database) of all files for quicker searches, installed via mlocate package

grep -E - allows use of () and | like egrep
grep '^[^#].*yes$' /etc/-h/sshd_config - find lines in /etc/ssh/sshd_config that end in yes but do not start with # (^[])

ls -S - sorts by size, largest first

tee -a <file> - append to <file>

cp --backup=numbered <file1> <file2> - created backup of <file2> if it exists
cp -i - interactive, prompts before overwriting

vi +$ <file> - open <file> and go to last line
vi +/google/ <file> open <file> and go to first line with google

sed -I 'i1 <some text>' <file> - insert some text at line 1 of <file>
sed '$a <some text>' <file> - append <some text> to <file>
sed -i '/^#umask/s/^#//' .profile - find line staring with #umask in .profile, remove #
sed -i '/^umask/s/^/#/' .profile - find line starting with umask in .profile, comment it
sed -i '$d' .bash_logout - delete last line from .bash_logout

du -s - summary, gives total for directory and all objects under it

chmod +x <file> - makes <file> executable, honoring umask
chmod a+x <file> - makes <file> executable, regardless of umask
chmod -R a+X <dir> - recursively sets execute bit on objects where it's already set in at least one block, regardless of umask
chmod  o= <file> - removes all permissions for others from <file>
chmod -v <operand> <file> - displays old and new permissions after change

ssh-keygen -t rsa -b 4096
ssh-copy-id ubuntu@cli-vm-b
eval $(ssh-agent)
ssh-add

find .?* - finds anything starting with . characters, followed by any one character, problem is that it finds all .. files too, doubling the output
find .??* - finds anything starting with 1 . character, followed by any two characters

man:

       1   Executable programs or shell commands
       2   System calls (functions provided by the kernel)
       3   Library calls (functions within program libraries)
       4   Special files (usually found in /dev)
       5   File formats and conventions, e.g. /etc/passwd
       6   Games
       7   Miscellaneous (including macro packages and conventions), e.g. man(7), groff(7), man-pages(7)
       8   System administration commands (usually only for root)
       9   Kernel routines [Non standard]

If a command is under /usr/sbin, use man 8 to get man page for it

sudo visudo /etc/sudoers.d/bob - create a file just for bob (doesn't have to be per user, can be any sudo content)
sudo visudo - edit default sudoers file

bob ALL=(root) NOPASSWD: /usr/bin/passwd, !/usr/bin/passwd root - allow bob to run sudo passwd but not sudo passwd root

pkexec - like sudo, polkit, also run pkttyagent to intercept authentication 
/etc/polkit-1/localauthority.conf.d/51-ubuntu-admin.conf - default admin/sudo config file for polkit

echo $$ - get PID for current shell

vi:

	:x - same as :wq
	:r <file> - read in contents of <file>
	dG - deletes to end of file

systemd-analyze - see startup times for os components (blame and dump give more detail)

systemctl mask <service> - stop <service> and prevent it from starting, undo with unmask swtich
systemctl cat <service> - cat out unit file for <service>
systemctl edit --full --force <service>.service - create/edit a service file (new service file is under /etc/systmed, not /lib/systemd)
systemctl list-timers - show timer type services (new timer won't show up until it's been run once)
systemctl enable --now <service> - enable <service> and start it now
systemctl list-units --type target - show available boot targets
systemctl get-default - show default boot target (change it with set-default)
systemctl isolate <target> - change to <target> run level

ss  - like netstat -anp but much more concise, only sockets
python3 -m http.server - create a basic web server in the current dir on port 8000

journalctl -g <string> - like piping to grep for <string>

cockpit  - web based ubuntu mgmt

/etc/nologin - prevent non-root users from logging in, removed on reboot, can put something meaningful in it, 
/run/nologin - gets put in place automatically via shutdown command
shutdown -c - cancel a scheduled shutdown



Can put custom grub menu items in /boot/grub.d/40_custom
	Add /boot/grub/grub.cfg and menu entry section only
	
GRUB_TIMEOUT_STYLE=menu

update-grub - same as sudo grub-mkconfig -o /boot/grub/grub.cfg






Add rw init=/bin/bash to end of linux line at grub boot menu item for single user mode
exec /sbin/init - keep booting out of single user mode

kernel is under /boot

sudo passwd -dl root - clear root password







apt local package cache is stored at /var/cache/apt/pkgcache.bin
apt list - lists all packages, can add switches to filter on installed, upgradable, available or by pattern
apt search - find packages with text in their description or name
apt show - see details about a package
apt remove will remove the software but leave configuration in place, apt purge removes everything

apt repositories are at /etc/apt/sources.list



dpkg -S /usr/share/ufw/after.rules - show what package is responsible for the /usr/share/ufw/after.rules file
dpkg -L postfix - show all files added by postfix package
dpkg-reconfiugre postfix - go through setup of postfix package again

lsblk  - list block devices
fallocate -l 1G /root/disk1 - create 1GB /root/disk1 file
losetup /dev/loop30 /root/disk1 - create loop block device loop30 pointed to /root/disk1
parted /dev/loop30 mklabel msdos mkpart primary 0% 100% - create partition from /dev/loop30

/etc/systemd/journald.conf - set Storage=persistent to keep journaled log files (systemctl restart systemd-journald)

/etc/rsyslog.conf and /etc/rsyslog.d/* - logging configuration 

journalctl --since <date1> --until <date2>

for i in {1..25};do;<something>;done

while [ "$PWD_OK" != "true" ];do;<something>;until

if [ "PWD_LEN" -gt 6 ]; then; <something>;fi
&& - and

$# - number of arguments sent to script







Create system cron files under /etc/cron.d
/etc/crontab has diagram showing crontab structure

nproc - number of processors
lscpu - detailed cpu info

pgrep  - search for process by name
pkill - kill process by name

aa-status - apparmor status, add --enforced to see what is preventing, add --complaining to see what is just being logged and not denied
aa-notify - see alerts from app armor

aa-easyprof <file> - write out apparmor profile to the screen based on <file>
aa-easyprof /usr/bin/file_test.py | sudo tee /etc/apparmor.d/usr.bin.file_test.py
apparmor_parser -r /etc/apparmor.d/usr.bin.file_test.py
aa-complain /etc/apparmor.d/usr.bin.file_test.py
file_test.py
aa-logprof

getent passwd - show all users, from /etc/passwd and other sources

UID > 1000 - normal user, < 1000 - system users

awk -F: '{ if ($3 > 999) print $1 }' /etc/passwd - print user names were UID > 1000

/etc/default/useradd - defaults for creating new users
useradd -D list defaults
/etc/login.defs - user login default values (affects useradd)
/etc/skel/ - templates for creating new users, can create symbolic links under here that will then be created in home dir for new users

sudo find /home /var -nouser - find files in /home and /var not owned by a user

chfn or adduser can populate finger data for a user

openssl passwd -6 VMware1\! - create hashed passwd that can be used in /etc/shadow

date -d '1 Jan 1970 19013 days' - find date passwd was changed based on field in /etc/shadow

Start a command with a space and it won't get saved to history

chpasswd - set passwords in plain text, can do multiple (echo "user1:VMware1!" | sudo chpasswd)

sg - switch group (have to have group password set)
gpasswd  - set a group password
gpasswd -a <user> <group> - add <user> to <group>
gpasswd -A <user> <group - set <user> as admin for <group>, can then add users to group



/etc/profile and /etc/bash.bashrc have centralized login settings
Can put extra centralized config scripts under /etc/profile.d/










selinux-activate
getenforce
setenforce Permissive
sestatus
semanage boolean --list
sudo setsebool secure_mode_policyload on - disallow users from changing enforcing mode (temporary, doesn't survive reboot)

ls -Z, ps -Z - show context
chcon - change context
ausearch - audit search
restorecon - restore context

sudo mkdir /staff
sudo semanage fcontext -a -e /home /staff - copies contest from /home to /staff
sudo restorecon -v /staff - restores context to /staff

Netplan configuration file sample (under /etc/netplan, like 99_config.yaml):

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      addresses:
        - 10.10.10.2/24
      routes:
        - to: default
          via: 10.10.10.1
      nameservers:
          search: [mydomain, otherdomain]
          addresses: [10.10.10.1, 1.1.1.1]


netplan try, netplan apply

/etc/iNetworkmanager/system-connections/'Wired connection 1.nmconnection' - created via GUI





getent hosts www.vmware.com - uses services as defined in /etc/nsswitch.conf to resolve www.vmware.com
Edit /etc/nsswitch.conf to set hosts to dns first, edit /etc/systemd/resolved.conf to set ReadEtcHosts to no to prevent local dns from reading /etc/hosts (sudo systemctl restart systemd-resolved)


This can be enabled in /etc/systemd/resolved.conf

sudo systemctl restart systemd-resolved.service
^restart^status^ - run previous command but replace restart with status

timedatectl set-timezone America/Denver

/etc/systemd/timesyncd.conf

[Time]
NTP=192.168.100.1 time.vmware.com

timedatectl timesync-status
timedatectl show-timesync
timedatectl set-ntp false|true

ip route add 192.168.2.0/24 via 192.168.110.101

sudo !!:p - preview last command run with sudo in front of it



sysctl -ar ip_forward - check if forwarding is enabled, change it in /etc/sysctl.conf, additional files under /etc/sysctl.d/
sysctl -p - process sysctl changes
/proc/sys/net/ipv4/ip_forward should reflect the change


ss - replacement for netstat
ss -ltn - (listening, tcp, numeric…don't resolve service names)
ss -ltp - (listening, tcp, PID)



sudo iptables -A INPUT -p tcp -s 192.168.110.10 --dport 22 -j ACCEPT
sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A INPUT -i lo -j ACCEPT
sudo iptables -A INPUT -j DROP
sudo iptables -Z && sudo iptables-save -f /etc/iptables.conf
(add this to your /etc/rc.local or similar to have it run automatically at boot)
sudo iptables-restore /etc/iptables.conf

sudo iptables -nvL - see packet counters

sudo iptables -L --line-numbers
sudo iptables -I INPUT 3 -p tcp -s 192.168.110.0/24 --dport 80 -j ACCEPT
sudo iptables -D INPUT 3 - delete rule at line 3

/etc/default/ufw - defaults, drop is default so will lose access if you enable ufw without allowing ssh first

ufw allow ssh
ufw enable
/etc/ufw/user*.rules

sudo ufw allow from 192.168.110.0/24 to any port 22

 sudo ufw status numbered

sudo ufw --dry-run delete 3

sudo ufw allow proto tcp from 192.168.110.0/24 to any port http

ls -la /etc/hosts
-rw-r--r--. 1 root root 247 Nov  1 14:21 /etc/hosts
(. at the end of permissions indicates an sshd x context is in use)

 ls -Z /etc/hosts
system_u:object_r:net_conf_t:s0 /etc/hosts

stat /etc/hosts
  File: /etc/hosts
  Size: 247             Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 2361569     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:net_conf_t:s0
Access: 2023-11-06 09:22:06.379115116 -0700
Modify: 2023-11-01 14:21:49.220530690 -0600
Change: 2023-11-05 09:21:27.143996232 -0700
 Birth: 2023-11-01 14:21:49.220530690 -0600

grep -i acl /boot/config-$(uname -r)
apt list acl
dpkg -S /bin/getfacl
mount -t ext4
sudo tune2fs -l /dev/mapper/vgubuntu-root |grep -i acl

apt install -y attr
sudo getfattr -n system.posix_acl_access file1

setfacl -m u:root:rw file1
getfacl file1
# file: file1
# owner: ubuntu
# group: ubuntu
user::rw-
user:root:rw-
group::rw-
mask::rw-
other::r--

setfacl -m d:u:apache:r,d:o:- /var/www/html - give apache read, deny all to others (default)

setfacl -m u:ubuntu:rw /private - sets acl but also changes group permissions to 777 (from 000), creates a mask of rwx in the acl

setfacl -x u:user9 /private/ - remove entry for user9

setfacl -b /private - remove entire acl for /private

getfacl file1 | setfacl --set-file=- file2 - copy acl from file1 to file2

getfacl /private > /tmp/acl.txt - backup acl
setfacl --restore=/tmp/acl.txt - restore acl, have to be in dir where file/dir to be restore resides


sudo apt install -y sssd-ldap ldap-utils
/etc/sssd/sssd.conf

cat .dsrc
[localhost]
uri = ldapi://%%2fvar%%2frun%%2fslapd-localhost.socket
basedn = dc=example,dc=com
binddn = cn=Directory Manager

dsidm localhost account reset_password "uid=alice,ou=people,dc=example,dc=com"
dsidm localhost user get alice
dsidm localhost user modify alice add:loginShell:/bin/bash
systemctl restart sssd

sudo sshd -T - list sshd config
sudo sshd -t - syntax check against /etc/ssh/sshd_config


ssh  -o <option> <value> - set ssh options and values

ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no tux@cli-vm-b.corp.vmw

.ssh/config - local ssh config file




Local dns server on ubuntu listens on 127.0.0.53:53


/etc/netplan config files can configure dns client

# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager
  renderer: networkd
  ethernets:
    ens160:
      dhcp4: no
      addresses: [192.168.110.100/24]
      gateway4: 192.168.110.1
      nameservers:
        addresses: [192.168.110.10]
        search: [corp.vmw]


Can link /etc/resolv.conf to /run/systemd/resolve/resolv.conf (saw mine linked to stub-resolv.conf

apt install bind9 bind9-utils

/etc/default/named - set OPTIONS="-u bind -4" to only do ipv4
/etc/bind/named.conf.options - bind options (can configure security here)
/etc/bind/named.conf.local:

zone "local" {
  type primary;
  file "/etc/bind/zones/db.local";
};


named-checkconf /etc/bind/named.conf.local - check syntax

Sample zone files under /etc/bind

$TTL    8h
@       IN      SOA     local. root.local. (
                        1               ; Serial
                        8h              ; Refresh
                        4h              ; Retry
                        1w              ; Expire
                        1h )            ; Negative Cache TTL
;
@       IN      NS      cli-vm-b.local.
cli-vm-b.local. IN      A       192.168.110.101
cli-vm.local.   IN      A       192.168.110.100

sudo named-checkzone local /etc/bind/zones/db.local - check zone

rndc reload - reload zone files

Reverse lookup zone:
zone "110.168.192.in-addr.arpa" {
  type primary;
  file "/etc/bind/zones/db.192.168.110";
};


apt install apache2 apache2-doc w3m

w3m localhost

/etc/apache2/apache2.conf


a2* commands can be used to configure apache symbolic links under /etc/apache2

Apache2ctl -t - check apache config

Set ServerName parameter in /etc/apache2/apache2.conf, also in site specific file

Can run a2dissite <tab><tab> to see enabled sites, should be under /etc/apache2/sites-enabled

Set site specific logging parameters in /etc/apache2/apache2.conf (LogFormat)
Call log formatting tag in /etc/apache2/sites-enabled/<site file> (CustomLog)

<ip>/server-status - apache status
/etc/apache2/mods-enabled/status.conf - status page configuration
 <Location /server-status>
                SetHandler server-status
                Require ip 192.168.110.0/24 - default is local (127.0.0.0/8)
                #Require ip 192.0.2.0/24
        </Location>


sudo a2enmod proxy proxy_http - enable apache proxy
/etc/apache2/mods-enabled/proxy.conf
export http_proxy=http://192.168.110.101

192.168.110.100 - "GET http://www.pluralsight.com/ HTTP/1.0" 301 557 "-" "w3m/0.5.3+git20210102" - log entry of client that went through proxy, note the 301 status code (permanently redirected)
 

sudo apt install mariadb-server
sudo mysql -u root
show databases;
use mysql;
exit;

sudo mysql u root mysql
describe user;
select User, plugin form user;

Mariadb only listens on loopback by default (port 3306)

mMysql --help --verbose - get configuration/health

/etc/mysql/my.cnf has includedir statements that note what folders will hold more configuration files
/etc/mysql/mariadb.conf.d/50-server.cnf (can comment out bind-address and add skip-bind address, can also uncomment skip-name-resolve) 
systemctl restart mariadb

sudo systemctl restart mariadb
!?systemctl?:s/restart/status/ - run last command that had systemctl in it, replace restart with status (sudo systemctl status mariadb)


create database sales;
create user 'bob'@'192.168.110.101/255.255.255.255' identified by 'Password1';

mysql -u bob -h cli-vm-b.corp.vmw -p

create user 'bob'@'% identified by 'Password1'; - % means anywhere


sudo apt install postfix mailutils

/etc/aliases - maps user aliases to actual users, won't be written to during postfix installing/configuration if it already exists
Run newaliases command to get postfix to know about chagnes
dpkg-reconfigure postfix to set up more options, can set aliases, can set local networks to other sites that can relay

mail root -s "subject" 
echo "Hello" | mail webmaster -s message

sudo apt install dovecot-imapd - imap mail server (143 and 994 for secure)

sudo useradd -m jack -s /usr/bin/passwd - user can ssh in but only change their password

sudo apt install mutt - imap mail client

/etc/dovecot/conf.d/10-ssl.conf

/var/spool/mail

.muttrc
set folder="imap://jack@192.168.110.101"
set spoolfile="imap://jack@192.168.110.101/INBOX"


corp.vmw. IN MX 10 cli-vm-b.corp.vmw. - mx record in db.local, have to change serial number in zone file when adding new entries
rndc reload local. IN

ip link ls - show nics

docker version
docker info

usermod -aG docker <user>

docker image inspect hello-world:latest -f '{{ .RepoTags }}'

Images are under /var/lib/docker

docker search ubuntu -f is-official=true
docker pull ubuntu
docker container run -it --name u1 ubuntu

docker container ls - same as docker ps


<ctrl> pq exits container but leaves it running

docker container prune - deletes all non-running containers

docker container start -i <name> - gets into running container

docker container run -d - detached mode, doesn't need you to be in it to keep it up

docker container rm -f web

docker container run -d --name web -v /home/ubuntu/web:/usr/share/nginx/html -p 8000:80 nginx - run nginx, map web folder to html folder in container, expose port 80 on port 8000

cat Dockerfile
FROM ubuntu
RUN apt update && apt install -y nginx
EXPOSE 80/tcp
ADD index.html /var/www/html
CMD ["nginx", "-g", "daemon off;"]

docker image build -t customweb .

docker container run -d --name customweb -p 8000:80 customweb

sudo apt -y install bridge-utils libvirt-clients libvirt-daemon qemu qemu-kvm

sudo adduser $USER libvirt - add your account to the libvirt group (do to kvm group too)

virsh list - see vms (add --all to see stopped vms)

virsh start - start vm
virsh destroy - stop vm
virsh undefine - remove vm

sudo virt-install --name=debian-vm \
--os-type=Linux \
--os-variant=debian9 \
--vcpu=2 \
--ram=2048 \
--disk path=/var/lib/libvirt/images/Debian.img,size=15 \
--graphics spice \
--location=/home/james/Downloads/debian-11.1.0-amd64-DVD-1.iso \
--network bridge:virbr0


lsblk - list block defvices (add -f to see file system info)
sudo fallocate -l500M /root/disk1
sudo losetup /dev/loop90 /root/disk1

find /dev/ -type b

ls -l /dev/sda
brw-rw----. 1 root disk 8, 0 Nov  5 09:29 /dev/sda - 8 = -8 scsi device, 0 = 0 device number

sudo parted /dev/loop90 mklabel msdos mkpart primary 0% 50% mkpart primary 50% 100% print - create two partitions

fdisk (use m to see options)
partprobe <device> - load fdisk updates

losetup -d <device> - remove loop device


sudo mkfs.ext4 /dev/loop90p1 (mkfs -t ext4)

sudo apt install xfsprogs (supply ability to create xfs file systems)

sudo mkfs.xfs /dev/loop90p2

sudo mkdir -m 0000 /shared_ext4 /shared_xfs - don't let anyone in unless these dirs are mounted

sudo mount /dev/loop90p1 /shared_ext4/
sudo mount /dev/loop90p2 /shared_xfs/

sudo blkid /dev/loop90* (can also get this from lsblk -f)
/dev/loop90: PTUUID="3fe872a7" PTTYPE="dos"
/dev/loop90p1: UUID="8b061574-31af-4556-8a20-42317f8e29a3" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="3fe872a7-01"
/dev/loop90p2: UUID="22fd33e9-d710-4323-b70f-6faec5f001a2" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="3fe872a7-02"

fstab:

UUID=8b061574-31af-4556-8a20-42317f8e29a3 /shared_ext4 ext4 defaults 0 0
UUID=22fd33e9-d710-4323-b70f-6faec5f001a2 /shared_xfs xfs defaults 0 0

man 5 fstab


mount -t xfs -show xfs file systems

change defaults to uquota in /etc/fstab to enable quota, umount/mount file system

xfs_quota -x - run in expert mode

xfs_quota -x -c 'report -h' /shared_xfs
User quota on /shared_xfs (/dev/loop90p2)
                        Blocks
User ID      Used   Soft   Hard Warn/Grace
---------- ---------------------------------
root            0      0      0  00 [------]
ubuntu        20M      0      0  00 [------]


 xfs_quota -x -c 'limit bsoft=25M bhard=30M ubuntu' /shared_xfs

Create 8 mb file

xfs_quota -x -c 'report -h' /shared_xfs
User quota on /shared_xfs (/dev/loop90p2)
                        Blocks
User ID      Used   Soft   Hard Warn/Grace
---------- ---------------------------------
root            0      0      0  00 [------]
ubuntu        28M    25M    30M  01 [7 days]

fallocate -l10M /shared_xfs/file4
fallocate: fallocate failed: Disk quota exceeded

xfs_quota -x -c 'report -i' /shared_xfs
User quota on /shared_xfs (/dev/loop90p2)
                               Inodes
User ID          Used       Soft       Hard    Warn/ Grace
---------- --------------------------------------------------
root                3          0          0     00 [--------]
ubuntu              3          0          0     00 [--------]



Compare uname -r with latest kernel under /boot, need to reboot if the latest is not running

find /lib/modules/ -type f -name '*quota_v*.ko*'
/lib/modules/5.15.0-39-generic/kernel/fs/quota/quota_v1.ko
/lib/modules/5.15.0-39-generic/kernel/fs/quota/quota_v2.ko
(these weren't the latest)

sudo apt install linux-image-extra-virtual quota (have to reboot after this, or run modprobe -v quota_v1 and modprobe -v quota_v2)

mkfs.ext4 -O quota /dev/loop90p1 - create fs with quota support, have to update fstab and uuid will change
tune2fs -O quota /dev/loop90p1

quotaon  -vua (verbose, user, all ext4 filesystems with quota enabled)

edquota ubuntu

Disk quotas for user ubuntu (uid 1000):
  Filesystem                   blocks       soft       hard     inodes     soft     hard
  /dev/loop90p2                 28672      25600      30720          3        0        0
  /dev/loop90p1                     0      20000      25000          0        0        0


repquota -uv /shared_ext4/
*** Report for user quotas on device /dev/loop90p1
Block grace time: 7days; Inode grace time: 7days
                        Block limits                File limits
User            used    soft    hard  grace    used  soft  hard  grace
----------------------------------------------------------------------
root      --      20       0       0              2     0     0
ubuntu    --       0   20000   25000              0     0     0

Create 20MB file
repquota -uv /shared_ext4/
*** Report for user quotas on device /dev/loop90p1
Block grace time: 7days; Inode grace time: 7days
                        Block limits                File limits
User            used    soft    hard  grace    used  soft  hard  grace
----------------------------------------------------------------------
root      --      20       0       0              2     0     0
ubuntu    +-   20480   20000   25000  7days       1     0     0

fallocate -l8M /shared_ext4/file3
fallocate: fallocate failed: Disk quota exceeded


mkswap <device>; chmod 600 <device>; swapon <device>
Can pass -p <#> to swapon to set priority for swap device

/root/disk3 swap swap pri=5 0 0


losetup -f /root/disk2 --show - -f says to use next available


Default swapiness is 60, 0 is no swapping, 100 is aggressive (was set to 20 on my system)

/proc/sys/vm/swapiness
Set it in /etc/sysctl.d/99-swap.conf (vm.swappiness=10), could also add to the end of /etc/sysctl.d/99-sysctl.conf
systemctl -p

parted /dev/loop17 mklabel msdos mkpart primary 0% 50% mkpart primary 50% 100% set 1 lvm on set 2 lvm on print


apt instlal -y lvm2

pvscan
pvs
pvcreate -v --pvmetadatacopies=2 /dev/loop17p1 - this can be skipped if accepting defaults during vgcreate, default  metadata copies is 1
pvdisplay

vgscan
vgs
vgcreate -v -s8K vg1 /dev/loop17p1 - default extent size if 4K
vgdisplay

cat /etc/lvm/backup/vg1 - backup of lvm metadata

vgextend -v vg1 /dev/loop17p2 - don't have to pvcreate in advance

lvscan
lvs
lvcreate -v -n lv1 -L 300M vg1
lvdisplay

mkfs.xfs /dev/mapper/vg1-lv1
mount /dev/mapper/vg1-lv1 /shared_lvm

lvextend -l +100%FREE -r vg1/lv1 - -r = resize fs, -l = add extents


mdadm - utility to manage RAID

mdadm -D --scan - see raid devices

mdadm --create /dev/md0 --level=mirror --raid-devices=2 /dev/loop18 /dev/loop19

mdadm -D /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Thu Nov  9 08:49:24 2023
        Raid Level : raid1
        Array Size : 510976 (499.00 MiB 523.24 MB)
     Used Dev Size : 510976 (499.00 MiB 523.24 MB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Thu Nov  9 08:49:42 2023
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              Name : cli-vm:0  (local to host cli-vm)
              UUID : c2aaca01:f69952ea:6ac77fbb:1d562e9f
            Events : 17

    Number   Major   Minor   RaidDevice State
       0       7       18        0      active sync   /dev/loop18
       1       7       19        1      active sync   /dev/loop19

mdadm -Db /dev/md0 > /etc/mdadm/mdadm.conf
update-initramfs -u

Create service to configure loop disks at boot

Create FS on /dev/md0, mount

/proc/mdstat holds raid device info

df -x tmpfs - show all filesystems except for tmpfs

wipefs  -a <device>

Disk /dev/sdb: 1 GiB, 1073741824 bytes, 2097152 sectors
Disk /dev/sdc: 500 MiB, 524288000 bytes, 1024000 sectors

Create fs on sdc, mount to /key

dd if=/dev/urandom of=/key/keyfile bs=1024 count=4
chmod 400 /key/keyfile


Systemd mount: (use lsblk -f to get UUID)

[Unit]
Description=Key disk

[Mount]
What=/dev/disk/by-uuid/3a0e80d6-f557-49b2-8792-db6ab5b90773
Where=/key/
Type=xfs
Options=defaults

[Install]
WantedBy=multi-user.target

Put under /etc/systemd/systsem/
systemctl daemon-reload
systemctl enable key.mount

sudo apt install cryptsetup
cryptsetup luksFormat /dev/sdb1 - has to be run as root, not sudo, have to enter  YES
cryptsetup open /dev/sdb1 data
 ls -l /dev/mapper
total 0
crw-------. 1 root root 10, 236 Nov  9 09:47 control
lrwxrwxrwx. 1 root root       7 Nov  9 09:54 data -> ../dm-2
lrwxrwxrwx. 1 root root       7 Nov  9 09:47 vgubuntu-root -> ../dm-0
lrwxrwxrwx. 1 root root       7 Nov  9 09:47 vgubuntu-swap_1 -> ../dm-1

cryptsetup -v status data
cryptsetup close data

cryptsetup luksAddKey /dev/sdb1 /key/keyfile
lsblk -f
sdb
└─sdb1
     crypto 2           583f74b3-110f-4397-87cb-a19d3ebe7c62
sdc

Copy uuid
/etc/crypttab

data UUID=583f74b3-110f-4397-87cb-a19d3ebe7c62 /key/keyfile luks

cryptdisks_start data
mkfs.xfs /dev/mapper/data
mkdir /shared_crypt
mount /dev/mapper/data /shared_crypt
cp /etc/hosts /shared_crypt
umount /shared_crypt
cryptdisks_stop data

/dev/mapper/data /shared_crypt xfs noauto,user 0 0


nfs listens on 111 and 2049
apt install nfs-kernel-server

cat /proc/fs/nfsd/versions
-2 +3 +4 +4.1 +4.2


/etc/default/nfs-kernel-server
RPCMOUNTDOPTS="--manage-gids --no-nfs-version 3"
systemctl restart nfs-kernel-server
(this did not work in my lab, also said to restart nfs-config which did not exist)





/etc/exports
/home 192.168.110.100(rw,root_squash,no_subtree_check,sync)
exportfs -r

apt install -y nfs-common (client)

sudo mount 192.168.110.101:/home /mnt

sudo apt install autofs (automounter)
/etc/auto.master (has reference to /misc auto.misc)

/etc/auto.misc

linux           -rw,soft,intr           192.168.110.101:/home

/misc will be created automatically

cd /misc/linux and you'll see the home dirs from cli-vm-b



-----------------------------------------------------------------------------------------------------------------------------------------------------------------------


apropos  - search tool for man pages, apropos director will search man pages that contain "director"
sudo mandb - create apropos database


readlink <symbolic link> - show source of symbolic link

ls -l --time-style=

SUID - file is executed as user who owns the file
SGID - file is executed as the group who owns the file

ls -l suidfile (SUID set)
-rwSr--r-- 1 clittle clittle 0 Nov 13 09:41 suidfile
(lowercase s means execute bit is set as well)

ls -l sgidfile (SGID file)
-rw-r-Sr-- 1 clittle clittle 0 Nov 13 09:42 sgidfile
(lowercase s means execute bit is set as well)

find . -perm /4000 - find files with SGID set (/6000 will find SGID and SUID files)

The sticky bit prevents anyone without permission from deleting or renaming a file or directory, even if other users have write permission to the directory. 

chmod +t <dir> or chmod 1777 <dir>

ls -ld stickydir/
drwxr-xr-t 2 clittle clittle 4096 Nov 13 09:45 stickydir/ (will be capital T if execute bit is not set)


find -mmin 5 - find files modified exactly five minutes ago
find -mmin -5 - find files modified in the last five minutes
find -mmin +5 - find files modified more than five minutes ago
-mtime - days
-ctime - change time

Modfiied time does not equal change time - changing permissions would be a change not a modification

-o - combine find parameters
-not - does not match a criteria (can also use \!)

-perm -664 - permissions at least 664
-perm /664  - any of these permissions

find /var/log -perm -g=w -not -perm /o=rw - find files under /var/log that are group writable but not read/write for others

find /usr -type f -size +5M -size -10M - find files under /usr whose size is between 5 and 10 MB


cut -d <delimeter> -f <field #> <file>

diff -y - side by side diff

While in less and search mode, -i to ignore case
While in vi and search mode, \c at the end of search term to ignore case

grep -w - match only whole word
grep -o - return only matching part of line


. - exactly one character
* - match previous element 0 or more times
+ - match previous element 1 or more times (have to use -E or \)
{} - previous element can exist this many times (id two numbers, min and max times)
? - exists 0 or 1 times
[] - range or sets
() - subexpression
[^] - negated ranges or sets

sed 's/disabled/enabled/gi' - the i means case insensitive
sed '500,2000s/disabled/enabled/g' - only work between lines 500 and 2000


tar -tf <tar file> - list contents of a tar file
tar -rf <files> <tar file> - add to tar file
tar -xf <tar file> -C <dir> - extract to <dir>

gzip -k - keep source file while creating archive
gzip -l <gz file> - see contents of <gz file>


rsync -a <local dir> <user>@<IP>:<dir>/

bc <<<1+2+3+4 - bc is calculatr, <<< sends the equation to it

column -t - put output into aligned columns

man openssl-<section> - to see more specific man page for openssl sub-functions, can then search for EXAMPLES to see specific examples (for example, openssl-req, openssl-x509)

openssl req -newkey rsa:2048 -keyout key.pem -out req.pem - create a csr
openssl req -x509 -newkey rsa:2048 -keyout key.pem -out -req.pem - create a cert

openssl req -x509 -noenc -newkey rsa:4096 -days 365 -keyout myprivate.key -out mycertificate.crt - create self-signed cert

openssl x509 -in mycertificate.crt -text - see cert contents

req  is for creating certs and csrs, x509 is for viewing them


git config --global user.name "clittle" (remove global to make something project specific)
git config --global user.email "chris@little-stuff.com"
git init - set ups directory as a project directory, creats .git directory

git add <files> - add files to staging area (to be commited)
git status - see status of project
git reset <file> - remove file from staging area but leave original file
git rm <file> - removes file and updates git so it knows that a commit is needed

git commit -m "comment about commit" (can do git commit -a to skip git add)

git branch <name> (git branch --delete <name> to delete)
git branch --list
git checkout <branch name>

git log - see activity on repository

git show <snippet of commit> - see specifics of what was changed in a commit

git merge <branch> - update current branch with <branch>

git remote -v - list associated remote repositories
git remote add origin <connection string for remote repositories> - origin is an alias for remote repository
Use ssh-keygen to create public/private key pair, can use this at github to allow traffic
git push origin master - push from local master branch to remote repo (origin)
git pull origin master - pull from remote repo (origin) to local master branch
git clone <remote repo> - create local repo from <remote repo>


systemctl reboot or powerroff, --force if necessary, can also pass it twice on the same command (like a reset)

Can add a wall message to shutdown by putting it in quotes after the command

systemctl get-default - see default run level
systemctl set-default multi-user.target
systemctl isolate graphical.target - go to specific run level
systemctl isolate emergency.target - like single user mode
systemctl isolate rescue.target - slightly higher than single user mode

chroot /mnt/sysroot - move into normal linux installation while in rescue mode

grub2-mkconfig -o /boot/grub2/grub.cfg - for BIOS based system
grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg - for EFI based system

lsblk - list block devices

grub2-install /dev/sda - for BIOS system
dnf resinstall grub2-efi grub2-efi-moudles shim - for EFI system (centos specifically)

/etc/default/grub - config settings for grub

help if - help on if, this can be used on other bash builtins like test

test -f - check if a file exists

grep -q - quiet, used for checking if something exists and returning an exit code

can always check /etc/crron.hourly/0anacron


systemctl edit --full <service>
systemctl revert <service> - undo edits
systemctl reload <service> - reload service config
systemctl reload-or-restart <service>
systemctl is-enabled <service> - check if a service is enabled
systemctl mask <service> - prevent service from starting (and stop it immediately)
systemctl list-units --type service --all - list all services

man systemd.service
  (look for OPTIONS and EXAMPLES)
good starting point is the sshd.service
[UNIT] [SERVICE] [INSTALL] sections

new services go to /etc/systemd/system/ (builtin services go to /lib/systemd/system/)

in ps output, process commands wrapped in [] are kernel processes

ps -U <username>
ps  f - show tree like format (forest)

pgrep - process grep

nice goes from -20 to +19, lower is a higher priority
nice -n # <command> - run command with # nice value
only root can set negative nice values
renice - change nice value on an existing process (have to be root), can only renice once per session

kill -L - list signals
kill -1 (or -HUP) - restart or reload
default signal is 15 (TERM), graceful stop
pkill - kill process by name

bg - send job to background

lsof -p <PID> - list all open files/dirs by <pid>
lsof <file> - see what's  using a file

/var/log/secure - password changes and other security events

journalctl -e - start at end (can also press > while in journalctl to go to end, or < to go to beginning)
info warning err crit - levels in journalctl 
journalctl -p <level>
journalctl -g '^b' - filter only messages that start with b
journalctl -S 02:00 - filter only messages after 2AM
journalctl -S 01:00 -U 02:00 - only messages between 1 and 2
can also use a date/time stamp with -S

journalctl -b 0 - logs for current boot (-1 for previous boot)

mkdir /var/log/journal - persist journalctl logs


cat /etc/crontab to see default cron entries
, - multiple values
- - range of values
*/# - every # minutes,hours
0-8/4 - between 0-8, every 4 hours

crontab -e - edit current user's crontab
sudo crontab -u <user>
crontab -r - remove crontab for current user

/var/log/cron

can put scripts under /etc/cron.<period>/ and will run at that interval, no extension on script  name

/etc/anacrontab - anacron file
anacron -T - test file syntax
sudo anacron -n - run today's jobs now (can add -f to rerun things that have already run)

at the end of command in crontab can put: | systemd-cat --identifier="some text", makes finding it in journalctl easier

at 15:00 
at> <command> (run command at 3PM)
ctrl-d

at '2:30 August 20 2024'
at 'now +3 days'

atq - see scheduled at jobs
at -c # - see contents of job #
atrm # - remove at job #


dpkg --listfiles <package>
dpkg --search /usr/sbin/nginx - show which package provides /usr/sbin/nginx

apt show <pkg> - in Description will show what it is and includes
apt search --names-only <string>  - only search package names for string
apt search <1> <2> <3> - must match all three
apt autoremove <pkg> - remove package and dependencies

cat /etc/apt/sources.list - apt repositories

curl <gpg url> -o docker.key - save repository key info to docker.key
gpg --dearmor docker.key - create docker.key.gpg
mv docker.key.gpg //etc/apt/keyrings/
add docker to new sources (docker.list) file under /etc/apt/sources.list.d/
deb [signed-by=/etc/apt/keyrings/docker.key.gpg] https://download.docker.com/linux/ubuntu jammy stable
apt update

add-apt-repository ppa:graphics-drivers/ppa
add-apt-repository --list

./autogen.sh && ./configure && make - compile source code
make install - puts binary in /usr/local/bin/, also puts docs in correct places

lscpu - list cpu details
lspci - list other hardware details

xfs_repair -v /dev/vdb1 - fix issues on xfs file system
fsck.ext4 -v -f -p /dev/vdb2 - fix issues on ext4 file system (-f is force, -p is preen…automatically fix some easy issues)

systemctl list-dependencies - light colored bullets mean service is stopped

free --mega - in megabytes (-m is memabytes)

sudo xfs_repair /dev/vdb > /home/bob/fscheck 2>&1 - always a good idea to send all output to file


sysctl -a - show all kernel parameters in use

sudo sysctl -w <parameter>=<value> - does not persist
sysctl <parameter> - return value for parameter

/etc/sysctl.d/ - add *.conf files here
man sysctl.d
sudo sysctl -p <file> - process file immediately


selinux is enabled by default on centos

ls -Z to see selinux context info
user:role:type:level
each user is mapped to selinux user
selinux users have specific selinux roles assigned
roles are allowed under certain types (domains for processes)
level is almost never actively used

chcon -t <type> <file>

ps axZ - list processes and their selinux context info
user:role:domain:level
only a file marked as a certain type can start a process that can enter a certain domain
unconfined_t is unrestricted

id -Z - see your security context
semanage login -l - see users and their context (semanage user -l)
getenforce
setenforce enforcing|permissing|disabled

sudo chcon - change context info (-u user, -r role -t type)
seinfo -u - see all selinux users
seinfo -r - see all selinux roles
seinfo -t - see all selinux types
sudo chcon --reference=<good file> <file> - use good file as reference to set selinux context on file
sudo restorecon -R <dir> - recursively set selinux label on files to what they most likely should be (pass -f to restore user and role)
sudo semanage fcontext --add --type <type> <file> - add record to selinux db incase a relabel happens
sudo restorecon <file> - will now set to type value saved in previous command
sudo semanage fcontext --list - see all default type values
sudo semanage fcontext --add --type <type> "<regex>"  - set default type on specific types of files
sudo semanage booleans --list - 
sudo setsebool <boolean> <value>
getsebool
sudo semanage port --list - port restrictions
sudo semanage port --add --type ssh_port_t --proto tcp 22

sudo systemctl stop apparmor.service
sudo systemctl disable apparmor.service
sudo apt install selinux-basics auditd
sestatus
sudo selinux-activate
reboot
sestatus - should be enabled and permissive

sudo audit2why --all = see what has been learned while in permissive mode
ps -eZ |grep sshd_t - see processes running with sshd_t selinux type
ls -Z /usr/sbn/sshd - type should match sshd_exec_t
sudo audit2allow --all -M mymodule - generate a module/policy package to be used when enforcing (can view mymodule.te file)
sudo semodule -i mymodule.pp ( can ignore sddm user error if you get it)
sudo setenforce 1 (or Enforcing) - this is temporary
/etc/selinux/config - change SELINUX= value to persist mode



files under /etc/skel copied to user home dir upon creation

useradd --defaults - see default settings
cat /etc/login.defs - see user account defaults

useradd --shell <some shell> --home-dir <some dir> --uid <####> <username>

id will show you user and context info

useradd --systsem <system acct> - uid will be under 1000, no home dir

userdel -r <username> - remove home dir too

usermod --home <dir> --move-home <username>
usermod --loign <john> <jane>
usermod --lock <username> (and --unlock)
usermod --expiredate <date> <username> - can set "" to set it to remove it

chage --lastday 0 <username> - expires password immediately (set to -1 to unexpire it)
chage --maxdays 30 <username> - change every 30 days
chage --maxdays -1 - never expire

users in wheel group have root-like privileges
users in docker group can connect to docker daemon

gpasswd --add john developers - add john to developers group
gpasswd --delete john developers

usermod -g developers john -modify john's primary group (-G adds to secondary group

groupmod -n programers developers - change programmers group name to developers
gropudel programmers - will error out if users have primary group set to this

/etc/environment can hold systemwide env vars

/etc/profile.d - add scripts that run on user login


/etc/security/limits.conf - security configuration for system
<user/group/*> <type(hard/soft/-)> <item> <value>

user can manually raise limit above soft
- means both hard and soft

man limits.conf

sudo -iu <user> - like sudo su -i <user>

ulimit -a - see all limits
ulimit -u <value> - set maximum user processes to 5000

visudo - edit sudoers

<user/group> <host/ip>=<runas user> <commands>

sudo -u <user> <command>

NOPASSWD: <command> - don't ask for user password


sudo passwd root - set passwd and allow login
sudo passwd -u root - unlock

sudo passwd -l root - lock root account logins


/etc/pam.d - all authentication modules
man pam.conf
required - will move on if failed, requisite - will stop processing more modules
sufficient - exists when something succeeds

man pam_wheel - see what the pam_wheel.so module will do

! in front of passwd entry in /etc/shadow shows that account is locked
passwd -S <user> - same

restrict root ssh access via pam:
sudo vi /etc/pam.d/sshd
auth    required       pam_listfile.so onerr=succeed  item=user  sense=deny  file=/etc/ssh/deniedusers
sudo vi /etc/ssh/deniedusers
root


libnss-ldapd - ldap client package for linux

/etc/nsswitch.conf
passwd:   files systemd ldap
group:   files systemd ldap
shadow:   files ldap

/etc/nslcd.conf - ldap client config file

getent passwd --service ldap - show only ldap users

sudo pam-auth-update - choose to create home directory on login



ip link - show network devices
ip -c addr - color code output
sudo ip link set dev <int> up
sudo ip addr add 192.1685.55/24 dev ens160

netplan get - show current settings
netplan files are under /etc/netplan/, in yaml formt
  addresses:
    - 192.168.110.100/24
    - fe80::921b:eff:fe3d:abcd/64
  nameservers:
    addresses:
      - 8.8.8.8
  routes:
    - to: 10.0.0.0/24
      via: 192.168.110.100
    - to : default
      via: 192.168.110.1
sudo netplan try - change will revert if you don't press enter, can pass --timeout <value in seconds>
sudo netplan apply

resolvectl status
/etc/systemc/resolved.conf - system name resolution settings
sudo systemctl restart sytemd-rsolved.service

resolvectl dns - show dns servers

man netplan, look for address label, Default routes, 

/usr/share/doc/netplan/examples - sample netplan files


ss - like netstat
ss -tunlp- listening, TCP, UDP, numeric values, processes


7 different bonding modes
0 - round robin
1  - active backup
2 - XOR
3 - broadcast
4 - IEEE 802.3ad
5 - adaptive transmit load balancing
6 - adaptive load balancing

/usr/share/doc/netplan/examples - bridge.yaml and bonding.yaml are examples


for bond, have to add after renderer:
ethernets:
  ens160:
    dhcp4: no
  ens161:
    dhcp4: no

can't use netplan try with bonds and bridges

/proc/net/bonding/<int> - has more info on bond


sudo ufw status - see ufw status (firewall)
don't enable with default settings, will block ssh
sudo ufw allow 22 - allow ssh over tcp and udp
sudo ufw allow 22/tcp - just tcp ssh is allowed
sudo ufw enable
sudo ufw status verbose - see details

sudo ufw allow from 192.168.1.60 to any port 22

sudo ufw status numbered - see rules with line numbers

sudo ufw delete 1 - delete rule #1
sudo ufw delete allow 22

sudo ufw deny from 10.11.12.100 - block 10.11.12.100

sudo ufw insert 3 deny from 10.11.12.100 - block 10.11.12.100 at position 3 (moving other rules down one)

sudo ufw deny out on ens160 8.8.8.8 - block outbound traffic on ens160 to 8.8.8.8

sudo ufw allow in on ens160 from 192.168.1.60 to 192.168.110.100 port 80 proto tcp - allow http traffic from 1921.68.1.160 to 192.168.110.100 on ens160

sudo ufw allow out on ens160 to 1921.68.1.60 from 192.168.110.100 port 80 proto tcp - reverse of above

man ufw


/etc/sysctl.conf or /etc/sysctl.d/99-sysctl.conf - can edit these to enable ip forwarding, should try to edit the second one

net.ipv4.ip_forward=1
sudo sysctl --system - to process changes
sudo sysctl -a |grep forward

should use nft but it's complicated, can still use iptables

sudo iptables -t nat -A PREROUTING -i ens160 -s 10.0.0.0/24 -p tcp --dport 8080 -j DNAT --to-destination 192.168.110.100:80 - create NAT rule to forward traffic coming in on ens160 on the 10.0.0.0/24 network at port 8080 to 192.168.110.100 on port 80
sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o ens160 -j MASQUERADE - change source IP to local source IP before sending it to NAT destination

sudo nft list ruleset - see result

sudo apt install iptables-persistent

sudo netfilter-persistent save

sudo ufw allow route allow from 10.0.0.0/24 to 1921.68.110.100

man ufw-framework

sudo iptables --lsit-rules --table nat - list NAT rules

sudo iptables --flush --table nat - clear NAT rules


sudo apt install nginx

sudo vi /etc/nginx/sites-available/proxy.conf
server {
  listen 80;
  location / { #can put specific URL in here
    proxy_pass http://1.1.1.1;
    include proxy_params;  # /etc/nginx/proxy_params, puts more data in the header
  }
}

sudo ln -s /etc/nginx/sites-available/proxy.conf /etc/nginx/sites-enabled/proxy.conf - enable proxy site

sudo rm /etc/nginx/sites-enabled/default - remove default web page

sudo nginxt -t - test config files

sudo systemctl reload nginx.service


sudo vi /etc/nginx/sites-available/lb.conf
upstream mywebservers { 
  least_conn; # send to server with least connections
  server 1.2.3.4 weight=3; # higher weight will mean more traffic
  server 5.6.7.8 down; # down means don't send traffic here
  server 10.20.30.40 backup; # only used if one of the first two is not available
}

server {
  listen 80;
  location / {
    proxy_pass http://mywebservers;
  }
}

sudo ln -s /etc/nginx/sites-available/lb.conf /etc/nginx/sites-enabled/proxy.conf - enable lbsite

sudo nginxt -t - test config files

sudo systemctl reload nginx.service



firewalld - tool for packet filtering

firewall-cmd --get-default-zone
firewall-cmd --set-default-zone=<zone>
firewall-cmd --list-all

firewalll-cmd --info-service=cockpit - list firewalld info about cockpit service

sudo firewall-cmd --add-service=http or --add-port=80/tcp
sudo firewall-cmd --remove-service=http or --remove-port=80/tcp

all ports are denied for incoming in public zone
sudo firewall-cmd --add-source=10.11.12.0/24 --zone=trusted - allow traffic from 10.11.12.0/24
firewall-cmd --get-active-zones
sudo firewall-cmd --remove-source=10.11.12.0/24 --zone=trusted - remove rule allowing traffic from 10.11.12.0/24

sudo firewall-cmd --runtime-to-permanent (or run original rule command a second time with--permanent at the end)


sudo ip route add 192.168.0.0/24 via 10.0.0.100 dev ens160
sudo ip route delete 192.168.0.0/24
sudo ip route add default via 10.0.0.100 

nmcli connection show - see what network manager connection is active
nmcli connection modify ens160 +ipv4.routes "192.168.0.0/24 10.0.0.100"
nmcli device reapply ens160
nmcli connection modify ens160 -ipv4.routes "192.168.0.0/24 10.0.0.100"
nmcli device reapply ens160

nmtui - TUI interface for editing a connection

systemd-timesyncd 

timedatecetl list-timezones - see all timezones
timedatectl set-timezone America/New_York
timedatectl - to see current time settings

sudo apt install systemd-timesyncd
sudo timedatectl set-ntp true
/etc/systemd/timesyncd.conf - ntp settings
put multiple servers on NTP= line

timedatectl show-timesync
tiemdatectl timesync-status


sudo apt install bind9 bind9utils bind9-doc ( or dnf instead of apt)

/etc/named.conf
man named.conf
listen-on port 53 { 127.0.0.1' }; # change 127.0.0.1 to an externally accessible ip as well { 127.0.0.1; 192.168.110.100; }, could also just put any
allow-query { localhost; }; # this can also be changed to allow external addresses
recursion yes; # lets bind get dns data from upstream servers

systemctl start named.service
systemctl enable --now named.service
firewall-cmd --add-servcie=dns
firewall-cmd --add-servcie=dns --permanent

dig @127.0.0.1 google.com


/etc/named.conf
zone "example.com" IN {
          type master;
          file "example.com.zone";
};

/var/named/named.localhost can be used as a starting point for a new zone, make a copy of it in the same place as example.com.zone
edit zone file
$TTL 1H # set ttl
@    IN SOA @ administrator.example.com. (
								0    ; serial # increase this each time a zone change is made
								1D   ; refresh # how long to wait between checking for changes
								1H   ; retry # if no response after refresh, how long to wait before trying again
								1W   ; expire # how long to wait after no response before giving up
								3H ) ; minimum # how long to cache negative responses
@              NS        ns1.example.com. # example.com has one name server at ns1.example.com
@              NS        ns2.example.com. # example.com has one name server at ns2.example.com
ns1            A         10.11.12.9
ns2            A         10.11.12.10
@              A         203.0.113.15 # record for example.com
www            A         203.0.113.15 # allow www.example.com to resolve
www            CNAME     203.0.113.15 # CNAME record for www, same as above, use one or the other
example.com.   MX 10     mail.example.com.
               MX 20     mail2.example.com.
mail1          A         203.0.113.80
mail2          A         203.0.113.81
sever1         AAAA      2001:DB8:10::1
example.com.   TXT       "We can put text here"

systemctl restart named.service

dig -t any - get all record types



mail is usually saved to /var/spool/mail/<username>

sudo dnf install postfix
sudo systemctl enable --now postfix

sendmail aaron@localhost <<< "test"
cat /var/spool/mail/aaron

/etc/aliases - user/email aliases
advertising: aaron

sudo newaliases - update system to read /etc/aliases
can now send to advertising@localhost

/etc/aliases
contact: aaron,john,jane - all mail to contact will go to the three users
advertising: aaron@somesite.com - forward to external server


sudo dnf install dovecot
systemctl enable --now dovecot
sudo fiewall-cmd --add-service=imap
sudo fiewall-cmd --add-service=imaps
sudo firewall-cmd --runtime-to-permanent

/etc/dovecot/dovecot.conf
protocols = imap # enables imap and imaps
listen = *, :: # listen on all ipv4 and ipv6 interfaces

/etc/dovecot/dovecot.conf.d - folder with more config settings
10-master.conf - change listening ports
10-mail.conf - setup where mail is stored (mail_location = mbox:~/mail:INBOX=/var/mail/%u)
10-ssl.conf - setup certificates (ssl_cert and ssl_key fields, also set ssl = yes to allow non TLS sessions)

echo "text> | mailx -s <subject> <user> - send message with subject to user


man sshd_config

AddressFamily (any, inet, inet6)
Match User aaron
    PasswordAuthentication yes # allow aaron to login with password

man ssh_config

vi .ssh/config
Host centos
	HostName 10.11.12.9
	Port 22
	User aaron
chmod 600 .ssh/config
ssh centos

ssh-keygen
ssh-copy-id aaron@10.11.12.9 - copy public key to .ssh/authorized_keys on 10.11.12.9 for user aaron

ssh-keygen -R 10.11.12.9 - remove 10.11.12.9 from .ssh/known_hosts

/etc/ssh/ssh_config - default client settings, should not edit
/etc/ssh/ssh_config.d/99-our-setting.cnf - create your own here


sudo dnf install squid - proxy
sudo systemctl enable --now squid
sudo fiewall-cmd --add-service=squid
sudo fiewall-cmd --add-service=squid --permanent

/etc/squid/squid.conf
acl localnet src 10.11.12.0/8 - allow this cidr
acl external src 208.4.15.20/16
acl SSL_ports port 443
acl Safe_ports port 80
acl Safe_ports port 443
acl youtube dstdomain .youtube.com
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny to_localhost
http_access allow localnet !youtube
http_access allow localhost
http_access allow external
http_access deny all

sudo systemctl reload squid.service


sudo dnf config-manager --enable powertools - enable powertools repository on CentOS
dnf repolist --all - list all enabled/disabled repos
sudo dnf group install <group>
dnf repoquery --list nginx - show all files provided by the nginx package
dnf check-upgrade - look for packages that can be upgraded
sudo dnf repolist - check repositories
sudo dnf search <string> - look for a package
sudo dnf provides <file> - find the package that provides <file>


sudo dnf install httpd
sudo firewall-cmd --add-service=http
sudo firewall-cmd --add-service=https
sudo firewall-cmd --runtime-to-permanent
sudo dnf install mod_ssl
sudo systsemctl enable --now httpd

httpd -M |grep ssl - check if ssl is enabled

man httpd.conf

/etc/httpd/ - config files
/etc/httpd/conf.d/ 
/etc/httpd/conf/

/etc/httpd/conf/httpd.conf
Listen - change ip and/or port httpd listens on
ServerAdmin - email address
ServerName - fqdn or ip of web server and port
DocumentRoot - starting folder for html files


/etc/httpd/conf.d/two-websites.conf
<VirtualHost *: 80>
	ServerName store.example.com
	DocumentRoot /var/www/store/
</VirtualHost>
<VirtualHost *:80>
	ServerName blog.example.com
	DocumentRoot /var/www/blog/
</VirtualHost>

apachectl configtest - check config

sudo systemctl reload httpd.service

/etc/httpd/conf.d/ssl.conf - ssl configs
can create file similar to two-websites.conf file above but with 443
<VirtualHost *: 443>
	ServerName www.example.com
	SSLEngine on
	SSLCertificateFile "/path/to/file.cert"
	SSLCertificateKeyFile "/path/to/file.key"
</VirtualHost>

/etc/httpd/conf.modules.d/ - module conf files


/etc/httpd/conf/httpd.conf - log file configuration
ErrorLog - relative to ServerRoot value
LogLevel - only log this level or higher
LogFormat - how log entries are created

/etc/httpd/conf.d/two-websites.conf
<VirtualHost *: 80>
	ServerName store.example.com
	DocumentRoot /var/www/store/
	CustomLog /var/log/httpd/store.example.com_access.log combined
	ErrorLog /var/log/httpd/store.example.com_error.log
</VirtualHost>
<VirtualHost *:80>
	ServerName blog.example.com
	DocumentRoot /var/www/blog/
	CustomLog /var/log/httpd/blog.example.com_access.log combined
	ErrorLog /var/log/httpd/blog.example.com_error.log
</VirtualHost>

sudo systemctl reload httpd.service


/etc/httpd/conf/httpd.conf

<Directory "/vvar/www/html">
	Options Indexes FollowSymLinks - Indexes allows users to traverse directories under documentroot, FollowSymLinks allows users to follow symbolic links
	AllowOverride All - allow the use of .htaccess files
	Require all granted - who can access the docroot
	
	
<Directory "/var/www/html/admin">
	Require all denied - block users from accessing /var/www/html/admin
</Directory>

Require ip 192.168.1.79 203.0.1.113 - these IPs can get access (192.168.1 - the same as 192.168.1.0/24)

<Files> ".ht*">
	Require all denied - block access to .ht* files
</Files>

sudo htpasswd -c /etc/httpd/passwords aaron - create http password for aaon in /etc/httpd/passwords (omit -c if file already exists, -D to delete an entry)

<Directory "/var/www/html/admin/">
	AuthType Basic
	AuthBasicProvider file
	AuthName "Secret admin page"
	AuthUserFile /etc/httpd/passwords
	Require valid-user
</Directory>



sudo dnf install mariadb-server -y
sudo systemctl enable --now mariadb.service
sudo firewall-cmd --add-service=mysql
sudo firewall-cmd --add-service=mysql --permanent

mysql -u root - access mariadb as root

mysql_secure_installation

mysql -u root -p - prompt for password

/etc/my.cnf -main config file

/etc/my.cnf.d/mariadb-server.cnf
datadir - where db is housed on disk
socket - socket where other apps can connect
log-error - error log
pid-file - process id saved here
bind-address - 0.0.0.0 by default
port=<####> - not present by default

docker search nginx

docker pull nginx:1.22.1

docker rmi nginx:latest

docker run --detach --publish 8080:80 --name mywebserver --restart-policy always nginx:1.22.1 - run ngxinx in detached mode, map port 8080 to 80 in the container, restart if not running

docker start <id|name> - start a stopped container

docker stop <container name>

nc localhost 8080
type GET / at prompt to get root of webserver

docker rm <container> - remove container

docker kill <container> - stop container

cd ~/myimiage
echo "welcome" > index.html
vi Dockerfile
FROM nginx
COPY index.html /usr/share/nginx/html/index.html

cd ~
docker build --tag clittle/myimage:1.0 myimage

docker run --detach --publish 9000:80 --name mywebserver 3 clittle/myimage:1.0



sudo dnf install libvirt quemu-kvm -y

testmachine.xml
<domain type="qemu">
	<name>TestMachine</name>
	<memory> unit="GiB">1</memory>
	<vcpu>1</vcpu>
	<os>
		<type arch-'x86_64'>hvm</type
	</os>
</domain>

run all virsh commands with sudo

virsh define testmachine.xml

virsh help

virsh list - see VMs (active, use --all to see inactive too)

virsh start TestMachine
virsh reboot TestMachine
virsh reset TestMachine
virsh shutdown TestMachine
virsh destroy TestMachine - forced power off
virsh undefine TestMachine - delete vm, not virtual disks, use --remove-all-storage to delete everything
virsh autostart TestMachine - start when server boots, can pass --disable to not autostart
virsh dominfo TestMachine - see info about vm
virsh setvcpus 2 TestMachine --config --maximum - set num cpus to 2 on VM next time it boots (and increases maximum to 2)


sudo apt install virt-manager -y

download os .img file

qemu-img info <image name> - see data about the image
qemu-img resize <image> 10G - increase image size to 10GB

/var/lib/libvirt/images - where vm disk images should live

man virt-install - look for EXAMPLES
virt-install --help

sudo virt-install --os-info ubuntu22.04 --name ubuntu1--import -memory 1024 --vcpus 1 --graphics none --disk /var/lib/libvirt/images/<.img file> - --import will just use a disk image, not install a new OS

get kicked into login prompt for vm when install is done
ctrl-] to exit console
sudo virsh list --all

sudo virt-install --os-info ubuntu22.04 --name ubuntu1--import -memory 1024 --vcpus 1 --graphics none --disk /var/lib/libvirt/images/<.img file> --cloud-init root-password-generate=on - create random password for root, displayed for 10 seconds on screen before boot

network connectivity even though it was not specified

sudo virsh console ubuntu1 - reattach to VM console

virt-install --osinfo detect=on - figure out what the OS is if you don't know, could also use a more generic option like linux2022 if you know the image was made after 2022


lsblk - show block devices

cfdisk - easier fdisk
should use gpt over mbr


swapon --show - see swap configuration and usage

sudo mkswap <dev>
sudo swapon -v <dev>
sudo swapoff <dev>
can also use a file, dd if=/dev/zero of=/swap bs=1M count=128 status=progress; chmod 600 /swap


sudo mkfs.xfs /dev/sdb1
sudo mkfs.ext4 /dev/sdb1

man mkfs.<fstype>

-L - label for the fs
-i - inode size

xfs <tab><tab> - to see other xfs utilities
xfs_admin will let you change parameters

on ext4, can use -N # to set number of inodes when creating

tune2fs - make changes to ext4 fs


man fstab

/etc/fstab
<block device>  <mount point>  <fstype>  <options>  <dump backup enabled?> <scan for errors?, and order>
run sudo systemctl daemon-reload after making changes

can use UUID=#### in place of block device name in fstab

blkid <dev> to see block UUID
blkid - with no parameters will list all block devices, their UUID and the file system type (if there is one)



sudo dnf install autofs
sudo systemctl enable --now autofs.service

sudo dnf install nfs-utils
sudo systemctl enable --now nfs-server.service

/etc/exports
/etc 127.0.0.1(ro)

sudo systemctl reload nfs-server.service

/etc/auto.master
/shares /etc/auto.shares --timeout=400

/etc/auto.shares
mynetworkshare -fstype=auto 127.0.0.1:/etc  (can set fsytpe to nfs4 or auto,ro) (can use an fqdn instead of ip) (can use block device instead of mount point)
sudo systemctl reload autofs

/etc/auto.master
/- /etc/auto.shares --tiemout=400 - /- means no parent directory

/etc/auto.shares - no subfolder name now, need absolute path
/mynetworkshares -fsytpe=auto 127.0.0.1:/etc
/localfiles/myext4files -fstype=auto :/dev/vdb2



findmnt - show all mount points and details about them

findmnt -t xfs - show only xfs file systems (can pass multiple fs types separated by commas)

sudo mount -o ro /dev/vdb2 /mnt - mount read only

noexec makes it impossible to run a program
nosuid makes suid not work

sudo mount -o rw,remount - remount with rw 

man mount (look for MOUNT OPTIONS)
man xfs

/etc/fstab
/dev/vdb1 /mybackups xfs ro,noexec 0 2


sudo apt install nfs-kernel-server
/etc/exports -  has examples
can put multiple hostname entries, or specify a CIDR or * in FQDN
sync - synchornous writes of data, client writes are persisted before moving on to next transaction (slower)
async - client can issue multiple writes before any are persisted to disk (faster)
no_subtree_check - default, can cause problems if subtree check is enabled
no_root_squash - allows root user to have remote root privileges, defaults is root_squash

man exports

sudo exportfs -r - re-export after changes to /etc/exports
sudo exportfs -v - see verbose share info

sudo apt install nfs-common

sudo mount <IP>:/<Path> <local mount point>


/dev/nbd# - network block devices

sudo apt install nbd-server
/etc/nbd-server/config
change nbd to root, or comment out user/group entries
allowlist = true - allow clients to list what is available on this server
[partition1]
	exportname = /dev/vda1 - export /dev/vda1 block device as partition1
	
sudo sysemctl restart nbd-server

man 5 nbd-server

sudo apt install nbd-client

sudo modprobe nbd
/etc/modules-load.d/modules.conf
add nbd

sudo nbd-client 192.168.110.100 -N partition1
will end up with /dev/nbd0
sudo mount /dev/nbd0 /mnt

sudo umount /mnt
lsblk
sudo nbd-client -d /dev/nbd0

sudo nbd-client 192.168.110.100 -L - list nbd resources


sudo apt install sysstat

iostat - storage i/o performance, from boot time
tps - transfers per second, reads or writes

iostat -10 - see last 10 seconds, will refresh every 10 seconds
iostat -d - don't show cpu stats
iostat -h - human readable format

dd if=/dev/zero of=DELETEME bs=1 count=1000000 oflag=dsync &

pidstat -d see storage perf into by process

iostat -P all - see on a per-partition basis, instead of all, can pass in single disk, like cda


sudo lvmdiskscan - see disks that can be used for lvm

sudo vgextend <vg name> <pv name>

lvs, pvs, vgs

sudo vgreduce <vg name> <pv name>

sudo lvcreate --size 2G --name partition1 <vg name>

sudo lvresize --extents 100%VB <vg>/<lv>
sudo lvresize --size 2G <vg>/<lv>
sudo lvresize --resizefs --size 3G <vg>/<lv>

man lvm - has list of all lvm commands at the end



sudo cryptsetup --verify-passphrase open --type plain /dev/vde mysecuredisk - verify-passphrase makes you enter it twice, end up under /dev/mapper/mysecuredisk

sudo mkfs.xfs /dev/mapper/mysecuredisk
sudo mount /dev/mapper/mysecuredisk /mnt
sudo umount /mnt
sudo cryptsetup close mysecuredisk


sudo cryptsetup luksFormat /dev/vde
sudo cryptsetup luksChangeKey /dev/vde - change password
sudo cryptsetup open /dev/vde mysecuredisk
sudo mkfs.xfs /dev/mapper/mysecuredisk
sudo cryptsetup close mysecuredisk

sudo dnf install lvm2 


RAID 0 - stripe
RAID 1 - mirror
RAID 5 - stripe with parity
RAID 6 - RAID 5 but with two parity disks
RAID 10 - mirror then stripe

sudo mdadm --create /dev/md0 --level=0 --raid-devices=3 /dev/vdc /dev/vdd /dev/vde
sudo mkfs.ext4 /dev/md0
sudo mdadm --stop /dev/md0
sudo mdadm --zero-superblock /dev/vdc /dev/vdd /dev/vde

sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/vdc /dev/vdd --spare-devices=1 /dev/vde

sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/vdc /dev/vdd
sudo mdadm --manage /dev/md0 --add /dev/vde

/proc/mdstat - raid status

sudo mdadm --manage /dev/md0 --remove /dev/vde



sudo setfacl --modify user:aaron:rw examplefile - allow aaron to have rw perms on examplefile

ls -l - files with a + will have an ACL applied

getfacl <file>
mask defines maximum permissions that the file can have, is adjusted automatically as acls are created

sudo setfacl --modify mask:r examplefile - set maximum perm to r

sudo setfacl --modify group:wheel:rw examplefile - allow group to rw examplefile

sudo setfacl --modify user:aaron:--- examplefile - don't let aaron do anything with the file

sudo setfacl --remove user:aaron examplefile - remove aaron's acl

sudo setfacl --recursive --modify user:aaron:rwx <dir> (recursive can also be used with --remove)


sudo chattr +a newfile - set append only attribute, can only add new data to file
sudo chattr -a newfile - remove append only

sudo chattr -i newfile - immutable

lsattr - show special attributes


sudo dnf install quota

/etc/fstab
/dev/vdb1 /mybackups xfs defaults,usrquota,grpquota 0 2
sudo systemctl reboot

sudo quotacheck --create-files --user --group /dev/vdb2 - cfreates aquota.group and aquota.user (for ext4)
sudo quotaon /mnt - for ext4

fallocate --length 100M /mybackups/aaron/100Mfile

sudo edquota --user aaron - edit quota for aaron

sudo quota --user aaron - check quotas

soft limit gets a grace period, after which soft limit becomes a hard limit

sudo quota --edit-period - set grace period

use --group to manage groups with edquota and quota commands

sudo xfs_quota -x -c 'limit bsoft=100m bhard=500m john' /mnt/

2 thoughts on “Linux Foundation Certified SysAdmin”

Leave a Comment

Your email address will not be published. Required fields are marked *