Unix crash course

Introduction to Unix

Syllabus

1. Recommended material:
   • Unix and Linux System Administration 5th Edition - Evi Nemeth, et al.
   • CompTIA Linux+ Study Guide 4th Edition - Chritine Bresnahan and Richard Blum
2. Objectives
   1. Shell scripting, input redirection, and C++/C code in a GNU/Linux environment
   2. Understanding of Linux file system including permissions
   3. Understanding of command and how to intermingle them to achieve more complex goals
   4. Basic system architecture, installation process, and management of Linux/Unix
3. General notes
   1. Learning to use Bash scripting and vi/neovim essential to professional usage of Linux in real life
   2. Any commands discussed in class can be looked up via ‘man’
   3. More detailed information for any command available from the command shell with an internet connection: curl cheat.sh/command_na

History

Bell System

2. Dates back to late 1960s, Originally developed by Ken Thompson and Dennis Ritchie of Bell Labs
3. Ritchie described the initial goal as to create a good environment in which to:
   2. do programming and development
   3. foster communal computing through remote access and close communication
   4. Unix originally written in assembly, but was later re-written in C in 1972
   5. First public releases in 1973, at Symposium on Operating Systems Principles; early source code sent by standard mail
   6. 1975, Version 6 of Unix was licensed to companies for the first time. However, commercial users were rare. Licensees only received source code from AT&T.
   7. Also in 1975, Ken Thompson takes a sabbatical from Bell Labs as a professor at UC Berkeley.
   9. Creates BSD: The Berkeley System Distribution of Unix 4.2.
   10. Includes C-shell and vi editor for first time
   11. 1983, Bell broken into several companies in anti-trust suit against AT&T.
   12. AT&T new restrictive license almost kills commercial Unix.
   14. By 1991, BSD had grown to a complete rewrite of AT&T Unix and comes with BSD license.
   15. BSD survives today as: FreeBSD, OpenBSD, NetBSD, etc.
   16. Additionally, Darwin kernel of Mac OSX continues as a form of Unix.
   17. GNU Project into Linux
5. Around 1983, Richard Stallman creates GNU project
6. Stallman’s GNU system goal was to create a ‘free’ system of software that had many of other capabilities of the commercial tools that many Unix also had
7. By 1985, Stallman had founded the Free Software Foundation to raise funds to develop free versions of all the components of the Unix syste, under GNU GPL
8. In 1991, Linus Torvalds began the development of a new free OS based on the MINIX operation system and SunOS. The 21 year old puts together the kernel.
9. As the kernel for the GNU project was still very much in development, the two projects became a natural fit.
10. To this day, Stallman insists that Linux is GNU/Linux due to the large reliance on GNU toolchain

Introduction to Red Hat Enterprise Linux

1. Package Manager - YUM
   1. Stands for YellowDog Updater Modified
   2. Provides a set of tools for retrieving software from the internet using subscriptions to an update server
   3. Utilizes .rpm software files to install packages for management
   4. Creates an Enterprise Linux environment
      • cons: less software options overall
      • pro: all available software is tested and works without additional configuration
   5. 5. yum has an additional, alternative wrapper program dnf to improve output and ease of use
      • stands for DaNdified YUM - commands in the format sudo dnf [verb] [objects]
      • -y flag gives YES as passed option for less attended installation 6. command commands
      • sudo dnf -y install [package_name] install a particular package and dependencies
      • sudo dnf -y update update all packages

Introduction to Ubuntu

1. Package Manager - APT
   1. Stands for Advanced Package Manager
   2. Ubuntu’s set of tools to install packages and keep them up to date
   3. Utilizes .deb software packages for management
   4. Keep in mind this package management system is far from standard
      • Many commercial Unixes have their own package managers
      • When all else fails, a tarball (.tar) file can be used to build from source
   5. Commands
      • apt-get update synchronizes the index of packages to your system
      • apt-get upgrade downloads and installs new versions of all currently installed packages in place, based on the index
      • apt-get dist-upgrade downloads and installs new versions, and moves the system up to the next version, based on the index
      • sudo was originally “superuserdo”, now works to substitute - users and run programs as that user. /etc/sudoers controls this behavior.
      • sudo apt-get install foo : installs package named foo
      • sudo apt-get remove bar : removes a package named bar, but not the config files
      • sudo apt-get purge bar : removes a package named bar, and removes config files
      • sudo apt-get source * : gets the source code for a package
      • sudo apt-get clean : removes downloaded packages (.deb files) used to install

File System

1. Directories
   1. first and bottom of the directory of the file structure tree is called root /
   2. /root in addition, there is a file directory, within the root directory called /root (“slash root”), which acts as root’s home directory
   3. /home normal users’ directories
   4. /boot contains the boot directory, including kernel, initial RAMdisk, and bootloader
   5. /dev lists all devices of the computer
   6. /etc configuration files for nearly all programs that need a state
   7. /tmp temporary files for the system
   8. /var variable-length files used by programs
      • /var/spool old-school webmail location
      • /var/www location of http server files
   9. /usr files available to users, including binaries for daily user tasks in /usr/bin
   10. /sbin binaries used only by root
   11. /bin binaries for the system everyone needs to use
2. Commands
   1. cd change directories
   2. ~ alias used to shortcut the current user directory
   3. Ctrl+R search history of commands in bash; also available as output from history
   4. !! runs the previous command
   5. ls list in the contents of current directory
   6. .: the current directory
   7. .. : go up one directory#User and Group Configuration

useradd/usermod

1. useradd and usermod do much the same thing
   • -c allows to add comments
   • -d allows to change user home directory
   • -e expiration date - date an account will expire (in YYYYMMDD format), kept in /etc/shadow
   • -f specify how many days the account can be inactive before password expires, kept in /etc/shadow
   • -g specify default group of the user
   • -G append a list of additional groups for the user
   • -l(usermod only) changes a user login ex: usermod -l darren darseife
   • -u uid
   • -s specifies login shell
   • -p specify pre-hashed password
   • -L (usermod only) locks the user’s password so it cannot be entered directly (best used with -e, as well)
   • -U will unlock a user’s password

passwd

commands

1. sudo passwd darseife change the password of darseife
2. passwd can also be used to lock and unlock accounts
   • passwd -l will lock the account’s password
   • a locked password will always be input incorrectly
   • does not necessarily lock user out of the account
   • puts an ! at the beginning of the password field in /etc/shadow
   • crypt, the algorithm used to implement ciphers, cannot generate an !so the password hash is disabled
   • passwd -u would unlock the account’s password
3. passwd -d removes the password of the account
4. passwd -S displays account information:
   • last password change
   • password expires
   • password inactive
   • account expires
   • minimum days between password change
   • maximum days before password change
   • warn period before changes

chage

1. The chage command allows you to modify account settings related to password expiration and aging 2. chage can also be used to configure accounts to automatically expire if: - the password hasn’t been changed - or a date has been passed 3. chage -l print expiration settings of an account 4. chage -m DAYS username to set the minimum time between password changes 5. chage -M DAYS username to set the maximum time between password changes 6. chage -d YYYYMMDD username update the last day YYYY-MM-DD on which to change the password 7. chage -E YYYYMMDD username set the expiration date as YYYY-MM-DD on which username’s account expires - setting the number to -1 will make it never expire 8. chage -I DAYS username will lock the account password after number of days inactive 9. chage -W DAYS username number of days the user will be warned before their account is made inactive 10. All of these settings are stored in /etc/shadow

1. 1. 1. userdel 1. userdel username deletes users 2. userdel -r username deletes the user, their mail spool, their home directory, and everything in it 3. userdel -f username deletes the user right now even if they are logged in

1. 1. 1. groupadd and groupmod 1. groupaddadd a group to a system 2. groupadd -r used to create a system group, creating a GUID above 1000 3. groupmod change a group 4. groupmod -n change a group name

1. 1. 1. gpasswd - group password 1. gpasswd groupname sets password for a group; it is most common to not have a group password 2. gpasswd -R restricts access to users logging in with password via newgrp

1. 1. 1. getent - get entities 1. can be used to view individual records in /etc/passwd, and other files 2. useful for getting exact matches for items 3. getent password username - get passwd information for user 4. getent shadow username - get shadow information for user 5. getent group groupname - get group information for user#The *nix CLI ###The Shell explained 1. A shell is a command interpreter where the user interacts with the system via command line interface (CLI) 2. Different shells currently in-use A. Bourne-Again SHell - located at /bin/bash is the main shell and is used was a default on most Linux systems B. sh - The Bourne Shell, old school version does not require any libraries. Rarely seen as a user-interface shell; typically a symbolic link to another shell binary. Created by Stephen Bourne, first included in Unix V7 C. tcsh Tee Shell. Based on the earlier C-shell from Unix. Has some additional features built into it. D. csh C-shell, a BSD component not often seen on Linux. Created by Bill Joy at UC Berkeley. E. ksh Korn shell from Bell Labs in the 1980s. The idea was to incorporate features from the Bourne Shell along with C-programming syntax. F. zsh 1990s, incorporates additional ideas into ksh. 3. Built-in Commands A. cd changes directory - cd ~ or cd /home/user are equivalent B. ls - lists files in a directory - default behavior just lists files - -a lists all files including hidden ones - -l long format - -d just directories - -F uses a special character to rep filetypes - -R recursive listing C. Single characters are substituted by ?, wildcards by * D. Exiting the system - exit exits the current shell - logout only works for login shells E. Shutting down the system - shutdown -h now halts the system now - shutdown -r now restarts the system now - shutdown -h +5 " " broadcasts the message in quotes and tells people it will be down in 5minutes - halt on most systems, will drop the system into single user mode - poweroff “on fire” graceless exit. Sends HW signal to kill power - reboot essentially ‘shutdown -r’ - reboot -f graceless restart exit, Sends HW signal to reboot F. init relies on a series of programs or scripts to change run-state of machine and to start/stop linux services - init 0 - shuts the machine down - init 6 - reboots the machine - init S - initializes system boot

1. 1. 1. Man pages and getting help 1. displays the manual pages for any packages on the system 2. typing [/] will allow you to search through the page 3. allows you to move via [spacebar] or [pgup/pgdn] 4. man is broken up into seperate chapters: man 5 would show chapter 5 1. executable programs and shell 2. system calls 3. library calls 4. device calls 5. file formats 6. games 7. misc. (macro packages) 8. sys admin commands 9. kernel routines 5. man -k * : to find any pages matching * 6. most programs also allow for --help or -? to be passed for a quick rundown on commands and usage

1. 1. 1. Unix filesystem permissions 1. filesystem permissions saved as a series of 3 bit numbers 2. format: d,rwx,rwx,rwx for read-write-execute a. d - denotes a Directory or file b. first triad is for the User of the file c. second triad is for Group members of the file d. third triad is for Everyone else 3. Read-write-execute for files a. Read allows the file to be opened b. Write allows for the file to be edited c. Execute allows for the file to run as a program 4. Read-write-execute for directories a. Read allows for the directory to view filenames in the directory b. Write allows files to be added or deleted from a dir c. Execute allows the directory to be visited, “permission to traverse through”; must have this to execute programs within a directory 5. chmod changes file and directory permissions a. two modes for chmod 1. symbolic mode chmod {u,g,o}[+,-,=]{r,w,x} 2. absolute mode - chmod 777 /path/to/file` * 0 no permissions * 1 execute * 2 write * 4 read F. chown - change file owner and group 1. chown user /foo - changes foo’s owner to user 2. chown root:root /foo - changes foo’s owner to root, also changes foo’s group to root G. echo - displays any text you enter after the command 1. primarially used in shell scripts or ‘programs’ that utilize a combination of of built-in commands, variables, and executables to complete tasks 2. variables reached from echo by invoking the variable name as $NAME #Input Redirection and Utils

1. 1. 1. Redirection symbols 1. prog > - redirects std output to the file, truncating and rewriting 2. prog 2> - redirect std error to the file, truncating and rewriting 3. &> - redirects both std output and error to file 4. >> - non-destructively appends to end of file rather than rewriting 5. - "backtick" process commands in the middle of another command a. A tool similar to xargs b. text within backticks are treated as a separate command line who's results are substituted on this command line c. things inside of backticks are attempted to be used as a command d. example: rm find / - user darren ` 6. $() - is used like a backtick; more resistant to weird syntax

1. 1. 1. Piping data between programs 1. programs on a UNIX system frequently use pipes to redirect each other’s output. 2. for instance you might use a text-filtering command on output from a file reading command. 3. allows small programs to operate on a flow of information in a row 4. | - this is an unnamed pipe 5. mkfifo - allows for the creation of named pipes a. mkfifo fifo1 b. ls -l > fifo1 c. cat < fifo1 6. tee - read from standard input and write to standard output and files a. splits standard input so that is it is both displayed via standard output and also captured to one of more files b. allows for logging and redirection while the process is also shown on screen c. overwrites to a file as per ‘>’ by default rather than like ‘>>’ d. example - echo $PATH | tee file1 file2 - file1, file2, and stdoutput all have same information 5. find - searches for files in directory hierarchy 6. xargs - builds a command from standard input, using command name and arguments to that command and then executes the command on each line of standard input a. example: find / - user darren | xargs -d %rm 7. file - returns the type of file and metadata information regarding files 8. join - combines two text files by matching the contents of specified fields a. by default, join uses spaces as the limiters (unless -t CHAR is given) b. example: join -1 3 -2 2 - this joins the third field of the first file and the second field of the second file 9. paste A. Paste is like cat rotated 90 degrees. When operating on multiple files, it will merge the lines of the files. 10. expand - converts tabs to spaces A. by default assumes you want 8 spaces per tab; -t changes B. unexpand - converts spaces to tabs 11. od - dump files into octal and other formats A. default usage is dumping binary file into octal binary format B. -x for 2byte hex output 12. sort - sorts lines of text files A. writes sorted concatenation of all files as output B. by default, uses first column C. Important options 1. -f : ignore case 2. -m : sort by three digit month abbreviation; ex, jan 3. -n : numeric sort 4. -v : sort by versio 5. -h : compare human readable numbers 6. -r : reverses the sort 7. -R : sorts randomly but groups like files 8. -d : dictionary-order 9. -k : sort via key, sorts by a field number, can be comma-seperated listq D. the opposite of this is ‘shuf’ 13. split - splits a file into 2 or more files A. needs to have an output file prefix B. where do you want to split 1. -b : size in bytes 2. -l : lines C. example: split -l 2 listening1.txt numbers -> output is: ‘numbersaa’ and ‘numbersab’ D. by default: splits every 10,000 lines 14. tr - trnaslate or delete characters A. changes individual characters coming in through standard input and outputs via standard output B. options 1. [:lower:] - all lower case chars 2. [:upper:] - all upper case chars 3. [::] 15. cut - extracts portions of lines of input and prints out as output A. used to remove items within lines, delimited by spaces or tabs B. options 1. -b : cuts the specified list by byte 2. -c : cuts the specified characters 3. -f : cuts specified list of fields from input 4. -d : specifies the delimiter for fields C. EXAMPLE: cut -f 7 -d : /etc/passwd 16. uniq - take an input list and report or omit repeated lines A. only if they are next to each other, so you have to sort first B. EXAMPLE: cut -f 7 -d : /etc/passwd | sort | uniq

C++ programs and Makefiles

1. Basic C++ program with input and output

int main (int args, char * argv[]){
	for(int x = 0; x < argv[],x++){
		cout << arv[x] << endl;
	}
return 0;
}
// outputs every character given to it

2. Let’s write a less good version of grep called greb:

#include <iostream>
#include <fstream>
using namespace std;

int main(int argc, char * argv[]) {
	string curLine;
	if (argc == 2){
		while(getline(cin,curLine)) {
			if (curLine.find(argv[1]) != string::npos) {
				cout << curLine << endl;
			}
		}
	}

	else if (argc == 3){
		ifstream inFile(argv[2], ifstream::in);

		while(getline(inFile,curLine)) {
			if (curLine.find(argv[1]) != string::npos) {
				cout << curLine << endl;
			}
		}
	}
	else {
		cerr << "Error: unknown parameters." << endl;
		return 1;
	}
	return 0;
}

3. Make A. Allows for automatic configuration and creation of usable binararies from OOP files, and can be used to install and uninstall programs B. Allows for the creation of archives of source code that build as packages into useful code C. typically named ‘makefile’

# set a variables for the compiler
CC = g++
LC_MESSAGES=C
CFLAGS=-g -Wall

# target name variable
TARGET = greb

# file extension
FILEXT = cpp

# the build target executable => install path of the executable
PREFIX = /usr/local
DESTDIR = /home/bernd/staging/

# basic makefile format
# name of instruction: dependencies
# <tab> command to execute
# the FIRST one you list is the default
$(TARGET): $(TARGET).$(FILEXT)
	$(CC) $(CFLAGS) -o $(TARGET) $(TARGET).$(FILEXT)

.PHONY: all
all: $(TARGET).$(FILEXT)
	$(CC) $(MAKEOPTS) $(CFLAGS) -o $(TARGET) $(TARGET).$(FILEXT)

.PHONY: clean
clean:
	rm $(TARGET)

.PHONY: install
install: $(TARGET)
	mkdir -p $(DESTDIR)$(PREFIX)/bin
	cp $< $(DESTDIR)$(PREFIX)/bin/$<

Tar and Packaging

1. tar A. the name ‘tar’ comes from its original purpose: Tape ARchive B. you can bundle files as .tar, ‘tarballs’, which contain a series of file entries and terminating bytes between files C. each entry has a file descriptor and the binary contents of the file D. file descriptor, or header, contains the name of a file, a checksum of that file, and permissions data E. does not compress data, but instead keeps it in one file F. tar options: -c: Creates a tar archive -v: Verbose output -x: eXtract from a tar archive -f: specify a Filename -A: Appends two archives together -M: create or extract a multivolume archive -j: specify bzip2 compression -J: specify xz compression -z: specify gzip compression –lzma: specify lzma compression -t: Test an archive

More Unix Commands

1. fmt - format A. some files have crazy long lines of text B. reformats to a certain -width C. by default, truncates after 75 characters and cleans up paragraphs D. anything with 2+ blankline removed, and adds indention E. -s : prevents truncation of paragraphs
2. nl - line number A. nl does count whitespace by default B. identical to cat -b (but not cat -n)
3. pr - prepare a file for printing A. includes header, footer, and page breaks B. create colummns: pr -3 file.txt C. -l sets the length of lines D. -o choose the header text
4. regular expressions - sed and awk A. simliar to wildcard expansion: ex; ls .txt B. grep and sed are two examples of regEx programs C. unix system programs have two different forms of regular expressions: basic and extended D. types of expresssions: 1. exact text match: grep “blah” returns all lines with “blah” 2. bracketed expressions: b[aeiou]g - returns bag, beg, big, bog, bug 3. bracketed with range: a[2-4]z - returns a2z, a3z, a4z 4. matching single char: a.z (allows for anything between a and z) 5. matching begin/end of line: ^ beginning, $ end 6. any of these can be combined with the wildcard: ex: A.* Lincoln (with 0 or more occurances of a pattern.) ex: A[ae]* Lincoln (with 0 or more occurances of a-e after A. 7. matching at least one of: + symbol ex. A[ae]+ Lincoln - returns at least Aa or Ae Lincoln 8. multiple possibilities or: | needs to be set up with ’ ex. ‘Darren (F|Fredrick) Seifert’ 9. grep -E invokes regex functionality

Shell Scripting

1. The first line of a shell script tells which program to use to interpret the contents: #!/bin/bash or another shell A. the first two characters are (#!) are special code to tell the kernel this is a script - crunchbang, shebang, hashbang, poundbang B. the remaining lines beginning with # are considered comments # this is a comment
2. Classic Hello World

# hello.sh
#!/bin/bash
# Print out hello world
echo 'Hello World!'

3. When you want to run a shell script, it needs to be made executable before it is an executable file: ‘# chmod u+x ./file.sh’
4. Can be invoked with ./file.sh
5. Alternative routes: A. bash file.sh works whether or not #!/bin/bash or chmod +x has been given B. exec file.sh is used to run scripts from within other scripts and to stop the tty instance after it is done (used to run scripts within scripts)
6. Passing Parameters A. much like C++ programs, bash scripts can be passed parameters B. accessing these, the dollar sign is used, much like PHP or Perl, to call the number of the parameter 1. $0 - the name of the running script itself 2. $1 - the first parameter 3. and so on… 4. $? - returns the exit status of the last run command 5. $$ - returns the process ID (PID) of the current script 6. $USER - returns the username of the user running the script 7. $SECONDS - returns the runtime of the current script 8. $RANDOM - generates a random number (int 0 - 32000) 9. $LINENO - current line of script C. creation of new variables 1. variablename=value # no spaces ever 2. if you do want spaces, they must be escaped by quotes 3. double quotes recommended for variable substitution later 4. for instance varname=“value $1” allows for insertion of other variables into variables whereas varname=’value ${\textstyle 1'willnotworkD.savingoutputintovariables1.outputfromprogramscanbesavedintoavariable2.example:curdir=}$(pwd) will run pwd and return the standard output as a variable 3. if you want to capture the standard error for a command, it can be saved as error=$(</tmp/error) 4. alternative use: curdir=pwd note the backtics

# foo.sh
#!/bin/bash
# An example script
cp $1 $2
#verify it worked
echo "Details for $2"
ls -l $2

• handling user input and output can use ‘echo’ to create interactive scripts:

# plususer.sh
#!/bin/bash
echo -n 'Please enter user name for account:'
read username # this saves the stdin input as username
read -p 'Username' $username # -n no-newline
echo $username
read -sp 'Password:' password # -s stealth

• scripts can also handle files as standard input like other commands

# salesreport.sh
#!/bin/bash
#a sales report generated from standard input
echo Data Summary:
cat /dev/stdin | cut -d ' ' -F2,3 | sort

• variables can be made global via export

# script.sh
#!/bin/bash
var1=blah
var2=foo
echo $0 :: var1 : $var1 , var2 : $var2
export var1
exec ./script2
echo $0 :: var1 : $var1 , var2 : $var2

script2.sh
#!/bin/bash
echo $0 :: var1 : $var1 , var2 : $var2
var1=flop
var2=bleh
echo $0 :: var1 : $var1 , var2 : $var2

Sed - Stream EDitor

1. the sed command uses regular expressions to modify stdin and sends the changes to stdout
2. usage : sed [options] script.txt [inputfile]
3. some of the commands that can be used in the script portion
   • = - display the current line number
   • a\text - append text to file
   • i\text - insert text to file
   • r file - append text from filename into the file
   • c\text - replace the selected range with the text
   • s/2012/2013/ - replace the first occurrence of 2012 on first occurrence on each line
   • s/2012/2013/g - replace ’’ ’’ every occurrence on each line
4. examples
   • sed 's/Nick\|nick/Gary/g' < salaryprojections.txt
   • ^ this would replace any ‘Nick’ or ‘nick’ with ‘Gary’ at all points in the file
   • standard replace format: sed 's/^/ /' file.txt > file.new.txt ### Examining Processes
5. ps command
   • The internal Unix command for process control is ps
   • By default, ps reports back what is happening inside of the current terminal/tty;
   • shows PID, TTY, TIME, and CMD
     • PID is the Process ID number for a running program
     • TTY is the Teletypewriter device which the process runs from; reports stdout and stderr to this by default
     • TIME is the cumulative processor time used by the program on Linux
   • ps supports traditional Unix options and GNU flags, which can be somewhat confusing
   • Options for ps
     • Display all processes running ps -A/-e both cause all processes on the system to be listed and output is identical
     • Display processes belonging to a specific user: ps -u user
     • Display extra fields as a “full format” ps -f can be combined with other options
       • UID User ID shows the username of the user running the command
       • PPID parent process ID, the process ID of the process that spawned the process
       • C processor utilization as % of usage over the lifetime of the process
       • STIME start time of the process as per system time (only in 24h clock format if current day; otherwise uses MMDD format)
     • Display long format ps -l
       • F Flags field
         • 0 = no special flags for this process
         • 1 = forked process, but did not call exec
         • 4 = executable was executed with superuser privileges
       • S State codes
         • D Uninterruptible sleep (usually IO)
         • R Running or runnable (on run queue)
         • S interruptible Sleep (waiting for an event to complete)
         • T Stopped, either by a job control signal or because it is being traced.
         • X dead (should never be seen)
         • Z Defunct (“zombie”) process, terminated but not reaped by its parent.
       • UID uses system number rather than human readable name
       • PRI Priority is used by the kernel, rather than in userspace, to determine the next in queue for processing
       • NI Niceness of a process describes how accommodating a process is to sharing time on the system.
         • This number ranges from +19 to -20, with 0 as the default.
         • In the scheduling system, niceness is used to determine how much processing to give to each process when it asks for it.
       • ADDR the memory address of the process
       • SZ the size of the image in pages
         • how much virtual memory is in use by the process.
         • Includes text, data, and stack space
         • page size is a kernel variable and can be read by getconf PAGE_SIZE
       • WCHAN address of where a function is sleeping (only kernel functions)
       • RSS amount of physical memory in use
   • User-oriented format ps -u displays information in a more readable manner
     • %CPU percent of processor time used by process divided by real time
     • %MEM actual percentage of non-reserved physical RAM
     • VSIZE virtual memory size used
   • Virtual memory format ps -v displays information including number of:
     • MAJFL major page faults of the process, indicating times it has used swap space
     • TRS physical memory devoted to stack
     • DRS physical memory devoted to non-executable code
   • Recieve a process hierarchy ps -AH see parent-child relationships visually
   • Choose your own adventure mode ps -o uid,pid,ppid,command
     • this command used to determine certain columns
   • Piping ps can be piped as per normal to other programs:
     • Ex; ps -u darseife \| grep bash
     • pgrep is a common utility that behaves similarly to the previous command
6. top
   • top is used to view processes as they execute in real time
     • -d used to choose update frequency in seconds
     • -p pid to monitor specifically
     • -n run for N iterations
     • -r command to renice a process within top
     • -s to change the update rate within top
   • htop can also be used as a next generation to top
7. nice and renice
   • The nice command is used to run a command with an adjusted niceness
     • generally can’t give negative values to programs without superuser access
     • nice -n 12 ./script.sh will add 12 to the default priority (0)
   • renice is used to adjust niceness of running processes.
     • renice 0 -p 2772 3562 renices the processes 2772 and 3562 to 0
     • renice 1 -u darseife renices the processes of user darsiefe to 1
8. Killing processes
   1. kill
      • stopping a process currently running -kill -s signal PID
        • PID PID by default, kill sends a signal to a number of process IDs
        • signal 1 is SIGHUP terminates interactive programs and causes daemons to reread config files
        • signal 9 is SIGKILL stops all processes without performing a graceful exit or shutdown
        • signal 15 is SIGTERM asks the process to terminate and to close out open files, run its shutdown script, etc.
      • kill will only kill processes owned by the user, unless run with superuser access
   2. killall
      • kill multiple things
      • killall vi kills all running instances of vi for the user; when run as a superuser, this will kill any running instance of vi
      • killall -s 9 vi kills all running instances of vi for the user by sending the SIGKILL signal to it; when run as a superuser, this will kill any running instance of vi.
      • killall -i vi uses interactive mode to ask which processes to kill
      • if you are running killall as root, try to use -i to prevent loss of other people’s work
9. Additional process control
   • If you are running a program inside of a terminal, [Ctrl + Z] will pause and go to background
     • To restart the process, run the command fg in the same terminal
     • With multiple processes paused, fg 1 or fg 2 will bring back process 1 or 2
   • Processes can be started in the background with & at the end of the line; unlike paused processes, these will continue to run.

Linux Time and Scheduling

1. Overview
   • Computer systems in general rely on a pair of clocks to operate
   • Two clocks
     • Hardware clock - initialized in BIOS/UEFI and kept updated via battery while computer is off
     • Software clock - initialized at boot from BIOS/UEFI time, and is used to track time while running
   • Typically, the clock of the computer is set using coordinated universal time (UTC)
     • adjusted for use based on timezones and localization settings like DST before entering userspace
   • Both the hardware clock and the software clock are not terribly reliable and tends to drift over time
   • This can be a problem on x86_64 systems in general because many network services rely on accurate timekeeping
   • This is made worse when having hundreds or thousands of servers requesting time from a single source could be bad
2. Unix Time Commands
   • date outputs the current software clock time
   • date -u outputs the UTC time
   • date 102715022019 sets the time to exactly Oct 27, 15:02, 2019 in local time
   • hwclock used to retrieve the value of the hardware clock; must be done as root
   • setting the hardware clock
     • hwclock --set--date 102715022019 is a manual set
     • hwclock -hctosys set the hardware clock to the system clock
     • hwclock --synctohc synchronizes system clock to hardware
   • these are called by daemons and init on startup and shutdown
   • both the system time and the hardware clock still suck and need a replacement
3. NTP Network Time Protocol
   • NTP relies on a tree of servers (each level is called a stratum) from a hardware atomic clock server
   • The further down the tree you are the less accurate the time is.
   • One NTP server requesting time from a center will exchange packet via timestamp and the delay between communications is estimated
   • It is best to pick NTP servers with a short delay
   • Typically, a cluster of computer is requested from a local time server, which in turn is the only computer requesting times outside of the network
4. timesyncd
   • Traditionally, ntpd is the daemon that has provided client time synchronization
   • Newer versions of Ubuntu use a service called timesyncd
     • uses a slightly different protocol than NTP, called SNTP (simplified network time protocol)
     • configuration information is at /etc/systemd/timesyncd.conf
     • timedatectl is a way to view the current SNTP settings
   • also more information related to this accessible with sudo systemctl status systemd-timesyncd
5. Running jobs in the future: cron
   • cron like timesyncd or ntpd, cron is a daemon that runs continuously and looks for events to cause it to spring into action
   • Most daemons are network services, but cron responds to temporal events
   • A cron daemon wakes up once per minute, and examines its config files in:
     • /var/spool/cron/
     • /etc/cron.d/
     • /etc/crontab
   • Executes command specified:
     • System jobs - run as root, and perform system maintenance tasks (rotate logs, clean out /tmp, remove old files, update system)
     • User jobs - run jobs in the future for users
   • /etc/crontab
     • System cronjobs are controlled by this file
     • Normally, begins with environment variable configuration, then several lines like:
       • On a typical system 02 4 * * * root run-parts /etc/cron.daily
       • On Ubuntu Desktop 25 6 * * * root test -x /usr/sbin/anacron || cd / && run-parts --report /etc/cron.daily
     • Fields for cron, in order
       1. the minute of the hour
       2. the hour of the day
       3. day of the month
       4. month of the year
       5. day of the week
     • different scripts for items are run from:
       • /etc/cron.daily
       • /etc/cron.weekly
       • /etc/cron.monthly
6. anacron a specific implementation of cron
   • anacron is not intended to be a replacement for cron, it is a supplement
   • cron works great on systems that are always on
   • anacron is smart in that it can tell if things are run
     • if things are missed, it plays catchup
     • light-night log rotation, for example, might never run
     • anacron is intended to address this
       • when run it checks to see when it last executed each program
       • if a period greater than the programs interval has passed, the program gets run by anacron
     • config file /etc/anacrontab
       • format: 1 5 cron.daily run-parts --report /etc/cron.daily
         • first column: delay in minutes
         • second column: frequency in days
7. crontab edit cronjobs for users
   • cron jobs for the system are under /etc/ but user ones are kept in /var
   • crontab -e edit your user cronfile
   • root can edit other users with crontab -u username -e
   • /etc/cron.allow and /etc/cron.deny exist for whitelisting or blacklisting users from using cron
     • an empty /etc/cron.deny allows ALL USERS if present
     • an empty /etc/cron.allow blocks ALL USERS if present
   • if neither of these exist, then only root may use cron
8. at a handy simple task timer
   • cron and anacron can be overkill for simple tasks that need to be done once
   • example:
     • at 5am Oct 31
     • at> /home/darren/run-backup.sh
     • at> echo "job done"
     • at> mail -s "backup job finished" root
     • at> ctrl+d to finish
   • at now +1 hour will run 1 hour from now relative to current system time
   • other examples:
     1. at midnight
     2. at noon
     3. at teatime
   • aterm to remove an upcoming command
   • atq used to list pending commands

Log Files

1. Intro
   • Linux machines log files that record key details about system operation and are typically stored in /var/log
   • Logging on Linux system is generally performed by the syslog daemon, syslogd
     • However, some daemons and programs perform logging independently of syslog
   • Logging on Linux can be done locally to the computer, but in a networked environment, logs may be sent to a central logging server and by email
   • Logging is a security action and is typically a part of the system worth hardening because it is exploitable by an attacker to hide their break-in
   • Logs should be rotated, aka moved out of the system, to prevent /var/log from being filled completely
2. syslog
   • The basic idea of a system logger is to provide a unified means of a handling log files
   • The daemon runs in the background and and accepts data delivered from servers and other programs
   • Once the log daemon receives a message, it is classified and directed to an appropriate file
   • syslogd might be managing several services at the same time and providing different rule-based logging based on priority of the messages received
   • syslogd relies on the config file /etc/syslog.conf
     • Ubuntu uses /etc/rsyslog.conf for rocket-fast syslog
     • the configuration file supports a ruleset, located at /etc/rsyslog.d/ or /etc/syslog.d/ respectively
     • standard logging is done by daemon, with the format facility.priority action
       • example: mail.* /var/log/mail.log
     • facility parameter is the name of the type of program the daemon that generates messages to be logged
       1. auth/priv security messages
       2. cron from the cron daemon
       3. daemon catch-all for various systems servers
       4. kern kernel messages
       5. lpr printer and CUPS messages
       6. mail mail subsystems
       7. news oldschool newsgroups
       8. syslog internally generated messages
       9. uucp unix-to-unix copy daemon
       10. local[0-7] predefined facilities for custom configurations
     • priority parameters include:
       • deb debugging messages
       • info normal operations
       • notice abnormalities that do not effect functionality
       • warn tells that an error will occur is action is not taken
       • err standard error for the daemon, non-urgent failure of the daemon causing an error message
       • crit critical error causing a secondary system to fail urgently
       • alert critical error causing a primary system to fail urgently
       • emerg Panic message - system unusable
     • the action parameter is a file or location on the network that will accept the message
       • could be a file in /var/log
       • could also be a tty: /dev/tty8
       • could also be a computer on the network: @logger.ndus.edu
   • When a program sends a message to the system logger, it includes the priority: the message is only logged if it is of a certain level or higher
     • Exceptions:
       • * all codes
       • = (mail.=crit) applies to this priority only
     • You can also specify multiple facilities with the same priority by using a comma
     • Multiple facilities separated by commas followed by a priority all follow the same priority
     • A semicolon is used to specify a number of logging priorities kern.info;kern.err everything between info and error
3. logger - logging something else
   • The logger command allows a user to log something else other than established facilities
   • logger shutting down for system maintenance
     • will log to /var/log/syslog as Nov 6 15:50:00 darseife-virtualbox darseife: shutting down for system maintenance
   • -i records the PID of the logger process when run (useful when run from script)
   • -s print the log message to standard error in addition to logging
   • -f /foo logs to /foo instead of the system log
   • -p specify both a facility and a priority for the system log message; -p local3.info

Firewalling

1. TCP Wrappers
   • TCP wrappers depend on two config files to work properly; these are /etc/hosts.allow and /etc/hosts.deny
   • the most common config for these is default deny
     • ALL:ALL in /etc/hosts.deny
     • ALL:127.0 [::1] in /etc/hosts.allow
     • specific services from a specific host enabled as the form sshd:192.168.X.Y in /etc/hosts.allow
     • all services from a domain as ALL:.minotstateu.edu in /etc/hosts.allow
   • Also used is specific allow
     • nothing is listed in /etc/hosts.allow
     • specifically removed IP addresses in the form ALL:165.234.X.Y in /etc/hosts.deny
   • Secondary service-level blocking can be used to apply rules to services in the form /etc/sshd.hosts
2. iptables
   • iptables is the kernel level netfilter firewall
   • iptables -L lists the current ruleset
   • iptables -A is used to APPEND new rules to a chain (takes a chain name)
   • iptables -I inserts a rule (takes a chain name and a position)
     • example iptables -A INPUT -ptcp -dport ssh -j ACCEPT
   • rule options
     • -p the protocol used
     • -d the destination port, can accept a range of port
     • -j the action to take, includes ACCEPT, DENY, DROP
     • -s specify a source address, can be either 192.168.0.1 or 192.168.0.0/24
     • -m require the rule to match a specific property
       • ex iptables -m conntrack allows filter rule to match based on connection state
     • --ctstate define a list of connection states for the rule to match on
       1. NEW - new connections
       2. RELATED - new but related to a previous on
       3. ESTABLISHED - a preexisting connection already in progress
       4. INVALID - any malformed traffic or unidentified traffic
     • -j tells iptable what to do with the packet given the rule
       • ACCEPT - routes as normal as per IP header
       • DROP - kills the packet without giving reply
       • REJECT - sends an explicit error message
3. Uncomplicated Firewall (ufw)
   • ufw enable to turn on ufw
   • ufw disable to turn off ufw
   • ufw status view current rules for ufw
   • ufw status verbose reports logging level + access levels for each rule
   • ufw allow 80/tcp sets a rule to allow tcp packets on port 80
   • ufw deny 22/udp sets a rule to reject udp packets on port 22
   • ufw deny ssh sets a rule to reject all ssh protocol packets
   • ufw allow 6000-6007/tcp sets a rule with a range to allow tcp packets from 6000-6007
   • ufw allow from 15.15.15.1 to any sets a rule allowing all traffic from 15.15.15.1
   • ufw status numbered returns a numbered list of rules in the set
   • ufw delete 2 removes rule number 2 in a list
   • ufw logging {on/off} turns the logging on or off; usually caught by syslogd/rsyslog; logging is set to low by default
   • ufw logging {low/medium/high/full sets logging verbosity of ufw;
     • note that logging is rate limited at medium and not rate limited at high

Boot/Initialization

1. Slackware - SysV Init
   • System 5 style rc script via init
   • PID 1 = init which is the process that starts all other processes when the system boots
   • init is responsible for processing /etc/inittab to see how to start the system and at what runlevel
   • Things init does
     • Runs the script /etc/rc.d/rc.s to determine to prepare the system before loading to the desired runlevel
     • Enables virtual memory for the system and mounts filesystems in /etc/fstab
     • Loads kernel modules
     • Near the end of the script rc.s, init will run scripts related to the runlevel the system is entering -for example, rc3.d is run at runlevel 3
2. SystemD
   • Ubuntu, CentOS, RedHat, and others have moved from SysV init process to the SystemD initialization process
   • Rather than running many shell scripts during init, the systemD process uses one monolithic program to control startup of services
   • The way the services are started are controlled by a configuration file
   • systemd uses units and targets rather than scripts and runlevels
     • there are 11 different types of units:
       • mount unit defines a mountpoint on the system named with the mount path with slashes replaced as dashes
       • swapspace
       • device a device in the linux device tree
       • socket fifo pipe to an internet port with an associated service
       • service a service or application on the system
     • targets are used to stop and start the service and group units together
     • systemd identifies units by name and type
       • example ssh could have an ssh.service and an ssh.socket
   • systemctl - can list current units loaded on the system
   • config files - /lib/systemd/system
   • modify how a unit functions - /etc/systemd/system