Collection Of Bash Tips
October 22, 2020 by Areg Sarkissian
This post list a collection of bash commands and scripts that I have collected over the years to be used as a reference.
I also list resources on various bash topics at the end of the post.
list processes running on system
Use the ps command to list processes running on the system:
ps aux | grep php
Flags:
a = show rows processes for users attached to terminal session u = display the process’s user/owner x = also show rows of processes not attached to a terminal
Using any the above flags will display all columns.
Instead we can use the -Ao flag with only the specific columns names that we want displayed:
# only display uid,user,pid and command columns in output
ps -Ao uid,user,pid,command | grep php
Check servers running on ports
Command to show network statistics:
netstat -anv
Flags:
-a Displays all active TCP connections and the TCP and UDP ports on which the computer is listening. -n Displays active TCP connections, however, addresses and port numbers are expressed numerically -v Displays active TCP connections and includes the process ID (PID) for each connection
Example to list servers listening on port 80:
netstat -anv | grep "\.80" | grep LISTEN
Example to list server using port 9000:
netstat -anv | grep 9000
Watching processes
The watch command can be used along with the ps aux command to poll and display process statistics continuously:
Watch can be used to monitor other commands whose output would change over time.
Install on Mac
brew install watch
Examples:
#Watch command to watch the foo process
watch 'ps aux | grep foo'
#this wont work since it will execute watch over 'ps' command instead of result of entire command chain
watch ps aux | grep foo
#watch all processes
watch 'ps aux'
tailing files
The tail command allows you to tail updates to files. Useful in monitoring log files
Example of how it works:
Create and tail a new test file:
touch testfile
tail -F testfile
launch another terminal tab and change to the testfile file directory:
echo 'abcd' >> testfile
# check the content of the file
cat testfile
Switch back to the previous tab an see the output of tail command.
bash command tips
Various bash command tips
stdout, stderr and redirects
Many bash commands use stdin, stdout and stderr that are represented using file descriptor numbers:
stdin => 0 stdout => 1 stderr => 2
File Descriptor symbols allow redirects to point to the stdout and stderr file descriptors.
Below are the file descriptor symbols for stdin and stdout:
# filedescriptor 1 symbol for stdout
&1
# filedescriptor 2 symbol for stder
&2
Redirect symbols point the specified or default descriptor to a destination file where the content written to that descriptor will be sent to:
Below are the redirect symbols for stdin and stdout:
#redirect stdout file descriptor 1
1>
#redirect stdout file descriptor 1. File descriptor 1 is assumed as the default.
>
#redirect stderr file descriptor 2
2>
These redirect symbols will overwrite the content of the destination they redirect to, if the command is re executed. If we want to redirect and append the outputs of the command we need to use the append version of the symbols shown below:
#redirect and append stdout file descriptor 1
1>>
#redirect and append stdout file descriptor 1. File descriptor 1 is assumed as the default.
>>
#redirect and append stderr file descriptor 2
2>>
We can use the stdout redirect symbol to redirect output of a command written to stdout to a file instead using the default stdout redirect symbol:
some_command > some_file
Or the equivalent using the explicit stdout redirect symbol:
some_command 1> some_file
We can also redirect any errors from a command written to stderr to a file instead.
some_command_with_errors 2> some_file
Some examples below:
#echo text to standar output
echo 'abcd'
#echo text to standard output that is redirected to a file
echo 'abcd' > testfile
echo 'xyz' >> testfile
#echo text to standard output that is redirected to a file
echo 'abcd' 1> testfile
echo 'xyz' 1>> testfile
#redirect error output of the failed ls command to the test file
ls -0 2> testfile
ls -0 2>> testfile
We can redirect the stderr to the stdout file descriptor as well
using the expression 2>&1 which is the concatenation of the 2> stderr redirect to the &1 stdout file descriptor.
This will be useful in commands with multiple redirects that we will see in the next section.
Multiple redirects
The redirect symbol > or append redirect symbols >> can be used more then once at the end of the command line to redirect stdin or stderr.
#echo text to standard output, standard error is redirected to standard output and standard output is redirected to file
#the first redirect symbol redirets the output of some_command to somefile
# the secont redirect symbol in 2>&1 redirects the standard error represented by the file descriptor 2 to the standard
#outout represented by the &1
some_command > somefile 2>&1
#the reverse will not work to send the stderror content into somefile since the the stderror is redirected to stdout before
#the standard out is redirected to somefile.
some_command 2>&1 > somefile
#There is a shorthand that is the equivalent:
some_command &> somefile
Below are examples:
#The following three echo commands are equivalent:
echo 'abcd' > testfile 2>&1
echo 'abcd' 1>testfile 2>&1
echo 'abcd' &> testfile
#The following three echo commands are equivalent:
#here the stdout and std in are redirected to /dev/null instead of a file
echo 'abcd' > /dev/null 2>&1
echo 'abcd' 1> /dev/null 2>&1
echo 'abcd' &> /dev/null
#Of course the above will not generate any erros. so a more practical example
#can use the ls command with an invalid flag to generate an error:
#redirect to file and redirect error to output to same file
ls -0 > testfile 2>&1
#or to suppress any output
ls -0 > /dev/null 2>&1
#If we send stdin and stdout to same text file the effect is not the same as the ouput from each will not get interleaved
ls -0 >testfile 2>testfile
#redirect to separate files (note testfile1 will be empty since we only have stderr outout)
ls -0 >testfile1 2>testfile2
#a valid ls command will have an empty testfile1 file because we will only have stdout output)
ls >testfile1 2>testfile2
#A more practical examlple is this command to run the laravel scheduler as a cron job
* * * * * cd /PATH-TO-LARAVEL-APP && php artisan schedule:run >> /dev/null 2>&1
dump all environment variables
env
Variable interpolation
To interpolate the variable $var use ${var}.
The {} brackets wrapped around text portion of a variable $var means that the contents of the variable var is interpolated with the surrounding text.
Otherwise $varabcd would be assumed the $varabcd variable, instead of the $var variable followed by the text abcd like ${var}abcd would.
Command evaluation
To evaluate a command and interpolate its output with surrounding text use the $() wrapper around the command.
The format is $(command):
Example:
#interpolate the path of the python executable determined using the `which python` command.
export PYTHON_HOME=\$(which python)
Open html file or URL in default browser
$ open index.html $ open https://www.google.com
Insert newline into file
The echo command will by default add a newline after the text we echo to a file. So everytime we append another string to the file using echo, the new string will appear on the next line.
echo "hello" > test
echo "world" >> test
cat test
Adding the \n character at the end of the string will insert an additional newline.
echo "hello\n" > test
echo "world" >> test
cat test
We can add as many \n characters at the end of the string to insert more newlines
echo "hello\n\n\n\n" > test
echo "world" >> test
cat test
The newline can placed anywhere within the string as well to insert newline between text
echo "hello\nworld" > test
cat test
echo "\n\nhello\nworld" > test
cat test
multiline interactive input
By using the EOL symbol with the cat command we can interactively add more lines until the closing EOL symbol on the last line completes the command.
the stdin redirect symbol << is used to redirect input lines typed in interactively in the terminal as stdin that then get redirected to stdout using the stdout symbol >. The EOL as the session terminator keyword. Any other word can be used in its place as long as the word at both ends match. The keywords will not be added to the content of the file.
Note that if we had used the
<stdin redirection symbolcat > test < EOL, thenEOLwill be treated as a filename that is redirected by stdin, not an interactive session.
Create file and add multiple lines:
cat > test << EOL
> hello
> world
> EOL
cat test
Keep appending more lines:
cat >> test << EOL
> once
> more
> EOL
cat test
Overwrite content:
cat > test << EOL
> hello
> again
> EOL
cat test
We can use any characters as the terminator as long as both sides match.
cat > test << MYTerm
> hello
> world
> MYTERM
cat test
multiline input in a shell script
First we create a shell script file:
touch multilinescript
Then we add the following to the multilinescript script file:
We can actually be recursive and interactively add content to the multilinescript file using the iteractive multiline technique we learned in last section!
#!/bin/bash
cat > test << EOL
hello
world
script
EOL
Run the script:
#give execute permissions to owner
chmod u+x multilinescript
#execute the script
./multilinescript
The result will be a file names test created with following content:
hello
world
script
cat to files
#create a file with some text
echo 'abcd' > test
#print to output
cat test
#create a new file test1 from test file
cat test > test1
#print to output
cat test1
#append into test 1
cat test >> test1
#print to output
cat test1
#create a new file test2 from test1 file
cat test1 > test2
#print to output
cat test2
#print concatenated files to output
cat test1 test2
#concatenate both files into new file 3
cat test1 test2 > test3
#print to output
cat test3
#append concatenated file to test 3
cat test1 test2 >> test3
#print to output
cat test3
cat standard input
We saw how use the cat command to output to a file, by interactive input through stdin.
We can also use the cat command to get text from stdin and use the stdin redirect to redirect stdin to a file.
Instead of explicitly having to specify /dev/stdin, The cat command also accepts a single dash character as a shorthand that is equivalent to /dev/stding.
Both forms are shown in the example below:
cat ~/.ssh/id_rsa.pub | ssh root@<YOUR_IP> 'cat /dev/stdin >> ~/.ssh/authorized_keys'
#shorthand with `cat -`
cat ~/.ssh/id_rsa.pub | ssh root@<YOUR_IP> 'cat - >> ~/.ssh/authorized_keys'
The cat command writes the id_rsa.pub to standard output of our local machine which is piped
into the ssh command as standard input of the ssh command. Cat is then run on the remote machine via ssh
command line argument that redirects the stdin to append the authorized_keys file using the >> append redirection symbol.
run command in background
command &
command &>/dev/null &
ps -eaf | grep php
# -A flag means Select all processes, including those of other users.
ps -Ao uid,user,pid,command | grep php
#pid=1713
kill 1713
Curl command piping
Variations of the same command
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash
curl -o - https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash
Wget command piping
Variations of the same command
wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash
wget -qO - https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash
wget -q -O- https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash
wget -q -O - https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash
bash script tips
Various script file tips
Give executable permissions Script
Assume we have a script file called myscript:
#make the script file executable
chmod +x myscript
#execute the script
./myscript
Give executable permissions Script for owner user only
Assume we have a script file called myscript:
#make the script file executable
chmod u+x myscript
#execute the script
./myscript
set environment variable used by executed script
FOO=bar bash -c ‘command1 args’
| FOO=bar bash -c ‘command1 args | command2’ |
Setting Script language
starting a script file with the #! line tells the shell process running the script the interpreter to use to execute the script.
Use explicit bash interpreter:
#!/bin/bash
Alternate form, using the default bash interpreter:
#!/usr/bin/env bash
PHP script example:
#!/usr/local/bin/php
Script error flags
Use the set command to set the error handling strategy of the script.
Include it after the first line #!/usr/bin/env bash.
#exit script anywhere the in the command pipeline on error or unset variable
set -euo pipefail
#also prints out commmands and arguments values that are executed
set -euxo pipefail
“-e” ensures that your script stops on first command failure. “-u” ensures that your script exits on the first unset variable encountered. “-o pipefail” ensures that if any command in a set of piped commands failed, the overall exit status is the status of the failed command
Script command line arguments
$*, $@, $# ,$0, $1 are used in bash script to refer to script and arguments
script $arg1 $arg2
When $\* and $@ are quoted:
"$*" quotes the entire list of arguments "$arg1 $arg2" output as a string
"$@" quotes all arguments individually "$arg1" "$arg2" output as a string
When $\* and $@ are not quoted they result the argument list as is and not quoted.
Should not use without quotes because it will break if arguments contain spaces or wildcards.
Example of a bash alias I setup in my .zshrc file that uses the "$*" arguments.
function gg {
#quote all arguments as a single message
#so we can write commit message without quotes
#gg this is my commit message sans quotes
#note: commit message can not have quotes or left and right brackets
if ! [ $# -eq 0 ]
then
git add -A
git commit -am "$*"
git status
fi
}
This allows me to type in my git commit messages like so:
gg my new commit
Which will transform to:
git commit -am "my new commit"
This is because $* representing the entire my new commit arguments is wrapped in quotes.
Also note the script checks that the number of arguments to the gg command is not zero using:
#number of arguments
$#
We could refer to the script text gg using the $0.
We could also refer to the my argument using $1, the new argument using $2 and so on.
permissions
echo $USER
sudo chown -R $USER:$USER /var/www
sudo chmod -R 755 /var/www
cd /var/www/myapp
sudo chown -R www-data: storage bootstrap
grep
cd /usr/local/bin && la | grep code
cd /usr/local/bin && la | grep subl
cd /usr/local/bin && la | grep pstorm
find
root@localhost:~# find / \( -iname "php.ini" -o -name "www.conf" \)
/etc/php/7.0/fpm/php.ini
/etc/php/7.0/fpm/pool.d/www.conf
/etc/php/7.0/cli/php.ini
input processing
#non exported variable used in current shell
instr="a,b,c,d,e"
echo "${instr//,/$\n}" ## Shell parameter expansion with commas replaced by newlins using echo
$ tr ',' '\n' <<<"$instr" ## Shell parameter expansion with commas replaced by newlins using tr
instr="a,b,c,d,e"
echo "${instr//,/ }" ## Shell parameter expansion with commas replaced by spaces
resources
how-to-read-and-set-environmental-and-shell-variables-on-a-linux-vps
the-first-two-statements-of-your-bash-script-should-be
safer_bash_scripts_with_set_euxo_pipefail
allowing-for-errors-in-bash-when-you-have-set-e-defined
here-is-why-you-should-quote-your-variables-in-bash
step-by-step-breakdown-of-dev-null