Unix crash course

1. Introduction to Unix

Filesystem

1. Directories
   1. first and bottom of the directory of the file structure tree is called root /
   2. /root in addition, there is a file directory, within the root directory called /root (“slash root”), which acts as root’s home directory
   3. /home normal users’ directories
   4. /boot contains the boot directory, including kernel, initial RAMdisk, and bootloader
   5. /dev lists all devices of the computer
   6. /etc configuration files for nearly all programs that need a state
   7. /tmp temporary files for the system
   8. /var variable-length files used by programs
      • /var/spool old-school webmail location
      • /var/www location of http server files
   9. /usr files available to users, including binaries for daily user tasks in /usr/bin
   10. /sbin binaries used only by root
   11. /bin binaries for the system everyone needs to use
2. Commands
   1. cd change directories
   2. ~ alias used to shortcut the current user directory
   3. Ctrl+R search history of commands in bash; also available as output from history
   4. !! runs the previous command
   5. ls list in the contents of current directory
   6. .: the current directory
   7. .. : go up one directory#User and Group Configuration

The Shell explained

1. A shell is a command interpreter where the user interacts with the system via command line interface (CLI)
2. Different shells currently in-use A. Bourne-Again SHell - located at /bin/bash is the main shell and is used was a default on most Linux systems B. sh - The Bourne Shell, old school version does not require any libraries. Rarely seen as a user-interface shell; typically a symbolic link to another shell binary. Created by Stephen Bourne, first included in Unix V7 C. tcsh Tee Shell. Based on the earlier C-shell from Unix. Has some additional features built into it. D. csh C-shell, a BSD component not often seen on Linux. Created by Bill Joy at UC Berkeley. E. ksh Korn shell from Bell Labs in the 1980s. The idea was to incorporate features from the Bourne Shell along with C-programming syntax. F. zsh 1990s, incorporates additional ideas into ksh.
3. Built-in Commands A. cd changes directory - cd ~ or cd /home/user are equivalent B. ls - lists files in a directory - default behavior just lists files - -a lists all files including hidden ones - -l long format - -d just directories - -F uses a special character to rep filetypes - -R recursive listing C. Single characters are substituted by ?, wildcards by * D. Exiting the system - exit exits the current shell - logout only works for login shells E. Remote access - ssh or secure shell provides the basic command to create a shell across an encrypted pipe - ssh username@remote.host logs in as user username on a computer remote.host - several programs provide ssh access as a feature, such as PuTTY - scp or secure copy provides file transfer over ssh - scp /path/to/localmachine/ username@remote.host:/path/to/destination - scp username@remote.host:/path/from/remotemachine /path/to/localdestination

Man pages and getting help

1. displays the manual pages for any packages on the system
2. typing [/] will allow you to search through the page
3. allows you to move via [spacebar] or [pgup/pgdn]
4. man is broken up into seperate chapters: man 5 would show chapter 5
   1. executable programs and shell
   2. system calls
   3. library calls
   4. device calls
   5. file formats
   6. games
   7. misc. (macro packages)
   8. sys admin commands
   9. kernel routines
5. man -k * : to find any pages matching *
6. most programs also allow for --help or -? to be passed for a quick rundown on commands and usage

Unix filesystem permissions

1. filesystem permissions saved as a series of 3 bit numbers
2. format: d,rwx,rwx,rwx for read-write-execute
   1. d - denotes a Directory or file
   2. first triad is for the User of the file
   3. second triad is for Group members of the file
   4. third triad is for Everyone else
3. Read-write-execute for files
   1. Read allows the file to be opened
   2. Write allows for the file to be edited
   3. Execute allows for the file to run as a program
4. Read-write-execute for directories
   1. Read allows for the directory to view filenames in the directory
   2. Write allows files to be added or deleted from a dir
   3. Execute allows the directory to be visited, “permission to traverse through”; must have this to execute programs within a directory
5. chmod changes file and directory permissions
   1. two modes for chmod
      1. symbolic mode chmod {u,g,o}[+,-,=]{r,w,x}
      2. absolute mode - chmod 777 /path/to/file` * 0 no permissions * 1 execute * 2 write * 4 read F. chown - change file owner and group
      3. chown user /foo - changes foo’s owner to user
      4. chown root:root /foo - changes foo’s owner to root, also changes foo’s group to root G. echo - displays any text you enter after the command
      5. primarially used in shell scripts or ‘programs’ that utilize a combination of of built-in commands, variables, and executables to complete tasks
      6. variables reached from echo by invoking the variable name as $NAME

Redirection symbols

1. prog > - redirects std output to the file, truncating and rewriting
2. prog 2> - redirect std error to the file, truncating and rewriting
3. &> - redirects both std output and error to file
4. >> - non-destructively appends to end of file rather than rewriting
5. - "backtick" process commands in the middle of another command a. A tool similar to xargs b. text within backticks are treated as a separate command line who's results are substituted on this command line c. things inside of backticks are attempted to be used as a command d. example: rm find / - user darren `
6. $() - is used like a backtick; more resistant to weird syntax

Piping data between programs

1. programs on a UNIX system frequently use pipes to redirect each other’s output.
2. for instance you might use a text-filtering command on output from a file reading command.
3. allows small programs to operate on a flow of information in a row

4. - this is an unnamed pipe

5. mkfifo - allows for the creation of named pipes
   1. mkfifo fifo1
   2. ls -l > fifo1
   3. cat < fifo1
6. tee - read from standard input and write to standard output and files
   1. splits standard input so that is it is both displayed via standard output and also captured to one of more files
   2. allows for logging and redirection while the process is also shown on screen
   3. overwrites to a file as per ‘>’ by default rather than like ‘>>’
   4. example
      • echo $PATH | tee file1 file2
      • file1, file2, and stdoutput all have same information
7. find - searches for files in directory hierarchy
8. xargs - builds a command from standard input, using command name and arguments to that command and then executes the command on each line of standard input
   1. example: find / - user darren | xargs -d %rm
9. file - returns the type of file and metadata information regarding files
10. join - combines two text files by matching the contents of specified fields
    1. by default, join uses spaces as the limiters (unless -t CHAR is given)
    2. example: join -1 3 -2 2
       • this joins the third field of the first file and the second field of the second file
11. paste A. Paste is like cat rotated 90 degrees. When operating on multiple files, it will merge the lines of the files.
12. expand - converts tabs to spaces A. by default assumes you want 8 spaces per tab; -t changes B. unexpand - converts spaces to tabs
13. od - dump files into octal and other formats A. default usage is dumping binary file into octal binary format B. -x for 2byte hex output
14. sort - sorts lines of text files A. writes sorted concatenation of all files as output B. by default, uses first column C. Important options 1. -f : ignore case 2. -m : sort by three digit month abbreviation; ex, jan 3. -n : numeric sort 4. -v : sort by versio 5. -h : compare human readable numbers 6. -r : reverses the sort 7. -R : sorts randomly but groups like files 8. -d : dictionary-order 9. -k : sort via key, sorts by a field number, can be comma-seperated listq D. the opposite of this is ‘shuf’
15. split - splits a file into 2 or more files A. needs to have an output file prefix B. where do you want to split 1. -b : size in bytes 2. -l : lines C. example: split -l 2 listening1.txt numbers -> output is: ‘numbersaa’ and ‘numbersab’ D. by default: splits every 10,000 lines
16. tr - trnaslate or delete characters A. changes individual characters coming in through standard input and outputs via standard output B. options 1. [:lower:] - all lower case chars 2. [:upper:] - all upper case chars 3. [::]
17. cut - extracts portions of lines of input and prints out as output A. used to remove items within lines, delimited by spaces or tabs B. options 1. -b : cuts the specified list by byte 2. -c : cuts the specified characters 3. -f : cuts specified list of fields from input 4. -d : specifies the delimiter for fields C. EXAMPLE: cut -f 7 -d : /etc/passwd
18. uniq - take an input list and report or omit repeated lines A. only if they are next to each other, so you have to sort first B. EXAMPLE: cut -f 7 -d : /etc/passwd | sort | uniq

Tar and Packaging

1. tar A. the name ‘tar’ comes from its original purpose: Tape ARchive B. you can bundle files as .tar, ‘tarballs’, which contain a series of file entries and terminating bytes between files C. each entry has a file descriptor and the binary contents of the file D. file descriptor, or header, contains the name of a file, a checksum of that file, and permissions data E. does not compress data, but instead keeps it in one file F. tar options: -c: Creates a tar archive -v: Verbose output -x: eXtract from a tar archive -f: specify a Filename -A: Appends two archives together -M: create or extract a multivolume archive -j: specify bzip2 compression -J: specify xz compression -z: specify gzip compression –lzma: specify lzma compression -t: Test an archive

More Unix Commands

1. fmt - format A. some files have crazy long lines of text B. reformats to a certain -width C. by default, truncates after 75 characters and cleans up paragraphs D. anything with 2+ blankline removed, and adds indention E. -s : prevents truncation of paragraphs
2. nl - line number A. nl does count whitespace by default B. identical to cat -b (but not cat -n)
3. pr - prepare a file for printing A. includes header, footer, and page breaks B. create colummns: pr -3 file.txt C. -l sets the length of lines D. -o choose the header text
4. regular expressions - sed and awk A. simliar to wildcard expansion: ex; ls .txt B. grep and sed are two examples of regEx programs C. unix system programs have two different forms of regular expressions: basic and extended D. types of expresssions: 1. exact text match: grep “blah” returns all lines with “blah” 2. bracketed expressions: b[aeiou]g - returns bag, beg, big, bog, bug 3. bracketed with range: a[2-4]z - returns a2z, a3z, a4z 4. matching single char: a.z (allows for anything between a and z) 5. matching begin/end of line: ^ beginning, $ end 6. any of these can be combined with the wildcard: ex: A.* Lincoln (with 0 or more occurances of a pattern.) ex: A[ae]* Lincoln (with 0 or more occurances of a-e after A. 7. matching at least one of: + symbol ex. A[ae]+ Lincoln - returns at least Aa or Ae Lincoln 8. multiple possibilities or: | needs to be set up with ’ ex. ‘Darren (F|Fredrick) Seifert’ 9. grep -E invokes regex functionality

Shell Scripting

1. The first line of a shell script tells which program to use to interpret the contents: #!/bin/bash or another shell A. the first two characters are (#!) are special code to tell the kernel this is a script - crunchbang, shebang, hashbang, poundbang B. the remaining lines beginning with # are considered comments # this is a comment
2. Classic Hello World

# hello.sh
#!/bin/bash
# Print out hello world
echo 'Hello World!'

3. When you want to run a shell script, it needs to be made executable before it is an executable file: ‘# chmod u+x ./file.sh’
4. Can be invoked with ./file.sh
5. Alternative routes: A. bash file.sh works whether or not #!/bin/bash or chmod +x has been given B. exec file.sh is used to run scripts from within other scripts and to stop the tty instance after it is done (used to run scripts within scripts)
6. Passing Parameters A. much like C++ programs, bash scripts can be passed parameters B. accessing these, the dollar sign is used, much like PHP or Perl, to call the number of the parameter 1. $0 - the name of the running script itself 2. $1 - the first parameter 3. and so on… 4. $? - returns the exit status of the last run command 5. $$ - returns the process ID (PID) of the current script 6. $USER - returns the username of the user running the script 7. $SECONDS - returns the runtime of the current script 8. $RANDOM - generates a random number (int 0 - 32000) 9. $LINENO - current line of script C. creation of new variables 1. variablename=value # no spaces ever 2. if you do want spaces, they must be escaped by quotes 3. double quotes recommended for variable substitution later 4. for instance varname=“value $1” allows for insertion of other variables into variables whereas varname=’value ${\textstyle 1'willnotworkD.savingoutputintovariables1.outputfromprogramscanbesavedintoavariable2.example:curdir=}$(pwd) will run pwd and return the standard output as a variable 3. if you want to capture the standard error for a command, it can be saved as error=$(</tmp/error) 4. alternative use: curdir=pwd note the backtics

# foo.sh
#!/bin/bash
# An example script
cp $1 $2
#verify it worked
echo "Details for $2"
ls -l $2

• handling user input and output can use ‘echo’ to create interactive scripts:

# plususer.sh
#!/bin/bash
echo -n 'Please enter user name for account:'
read username # this saves the stdin input as username
read -p 'Username' $username # -n no-newline
echo $username
read -sp 'Password:' password # -s stealth

• scripts can also handle files as standard input like other commands

# salesreport.sh
#!/bin/bash
#a sales report generated from standard input
echo Data Summary:
cat /dev/stdin | cut -d ' ' -F2,3 | sort

• variables can be made global via export

# script.sh
#!/bin/bash
var1=blah
var2=foo
echo $0 :: var1 : $var1 , var2 : $var2
export var1
exec ./script2
echo $0 :: var1 : $var1 , var2 : $var2

script2.sh
#!/bin/bash
echo $0 :: var1 : $var1 , var2 : $var2
var1=flop
var2=bleh
echo $0 :: var1 : $var1 , var2 : $var2