Shell is a program that is capable of understanding shell commands. Bash shell is one of the most propular shell programs.
Terminal is just a graphical interface between the shell program and command. This is where we write commands.
Command are instruction that are written in the terminal and submitted to the shell for interpretation.
GNU Project (http://www.gnu.org/)
- Free software is open source but not all open source is free software
- Richard Stallman began development of GNU operating system in 1983
- The freedom is provided by GNU public license (GPL)
- In 1991 Linus Torvalds created the Linux Kernel and published it under GPL v2.0
wget https://ftp.gnu.org/gnu/coreutils/coreutils-8.28.tar.xz
tar -xvJf coreutils-8.28.tar.xz
cd coreutils-8.28/
bash configure
# configure compuler and create Makefilemake
# compilation probablysudo make install
install the software
lsb_release -a
Ubuntu software repositories (https://help.ubuntu.com/community/Repositories/Ubuntu)
- is like a library of software
- Used for Installing softwares
- Used for Updating softwares to updated version
- Four different software repositores:
- Main: Cononical supported free and open source software
- Universe: Community maintained free and open source
- Restricted: Proprietary drivers for devices (can be open source)
- Multiverse: Software restricted by copyright or legal issues (can be open source)
Note: Fedora only supports free softwares
Ref: How-To Geek
- Checksums may also be called “hashes.”
- The algorithm uses a cryptographic hash function that takes an input and produces a string (a sequence of numbers and letters) of a fixed length
- deals with installing dependencies and makes life easier
- list of all available packages is located at
var/lib/apt/lists
(this folder is calledapt-cache
) sudo apt-get update
# updates the cache of packages and dependenciessudo apt-get upgrade
# upgrades every software in the local machine to the newest versionsudo apt-get install x11-apps
# install x11-apps in the local machine
nano /etc/apt/sources.list
and uncomment all of the lines in front of deb-srcsudo apt-get update
# updates the cachesudo apt-get install dpkg-dev
sudo apt-get source <package_name>
# downloads source code in pwd
sudo apt-get remove <package_name>
# uninstall package_name from the system but configuration files are left in the systemsudo apt- 8000 get purge <package_name>
# uninstall package_name and all of the configuration files. (preferred way)sudo apt-get autoremove
# remove packages those were previously installed as dependencies but no longer required by any packagessudo apt-get clean
# deletes all compressed package archivessudo apt-get autoclean
# deletes package archives that are no longer accessible
- Search the package name ex: 'foliate'
- If installed it will give fullname with
ii
- Use the fullname to uninstall:
sudo dpkg -P com.github.johnfactotum.foliate
Note: package manager downloads archive files (package archives) prior to installing the package into the system. These are located at /var/cache/apt/archives
- Install with snap:
sudo snap install <package_name>
Example:sudo snap install rambox
- Update packages installed with snap:
snap refresh
- Uniinstall packages:
sudo snap remove rambox
- 0 = standard input; 1 = standard output; 2 = standard error
- redirection of standard output breaks the pipleline
tee
command helps continue flowing the output whereas output redirection halts the output upto that stage- pipe
|
is used for connecting starndard output of one command to standard input of anohter command - all command doesn't accept standard input, they only accept command line arguments,
xargs
are used in these cases ~
is home directory for current user i.e /home/utshaw- directories or files starting with dot
.
are hidden files - single dot
.
is current directory - double dot
..
is previous directory - press double tab to show all completion possible with current amount of text
- file extension doesn't matter in linux unlike windows
- In order to determine the file type linux reads piece of code inside the top of each file known as header
directory/*
means everything inside the directory but not itselfdirectory/
means directory itself + everything inside the directory
- EUID: Effective User ID
- UID: User ID
- When the setuid bit is set, the file will execute with the permission of the owner of the file & only then EUID=owner's id, UID=user executing the script; other times EUID=UID
- This is rarely seen as modern linux OS doesn't allow scripts to be setuid
- root's UID=0
- if a command/program/script exits successfully then the exit status is 0 otherwise, the exit status is non-zero
- system/application accounts have lower uids (<1000)
- normal user has id>=1000
- ids are set in file /etc/login.defs
Stackoverflow post about shell-builtin
- builtin commands are part of the shell, and are implemented as part of the shell's source code.
- see the help page by
help <shell_built-in>
; ex:help read
# read is a shell-built in
^ symbol = ctrl
M- symbol = alt
Nano configuration file location: /etc/nanorc
- Note: bash scripts can be used for scheduling
- name the file as script.sh
- the first line will be #!/bin/bash (Shebang line); the rest of the script will be interpreted by bash script
- chmod +x script.sh
- In order to run the script from anywhere we need to add the path to $PATH environment variable
- In the end of the ~/.bashrc file add this line:
PATH="$PATH:$HOME/bin"
- The above line will cause all of the files in
$HOME/bin
location to be avialable from terminal - bash shell keeps hash to remember all program locations to execute.
hash -r
resets that hash for all remembered locations
NUM=$(( 1 + 2 ))
echo "${NUM}" # 3
NUM=$(( 6 * 8 ))
echo "${NUM}" # 48
NUM=$(( 6 / 4 ))
echo "${NUM}" # 1 integer division; doesn't support floatin point arithmetic
REM=$(( 6 % 4 ))
echo "${REM}" # 2 -> remainder of 2
DICEA='3'
DICEB='6'
NUM=$(( DICEA + DICEB ))
echo "${NUM}" # 9
MY_NUM=9
(( MY_NUM++ ))
echo "${MY_NUM}" # 10
(( MY_NUM-- ))
echo "${MY_NUM}" # 9
(( MY_NUM+=5 ))
echo "${MY_NUM}" # 14
OTHER_NUM=$(( MY_NUM+25 ))
echo "${OTHER_NUM}" # 39
- It is a bash shell built-in
- strip directory and suffix from filenames
basename /farhan/tanvir/utshaw/file.txt
# Output: file.txt
- strip last component from file name
dirname /farhan/tanvir/utshaw/file.txt
# Output: /farhan/tanvir/utshaw
cron - daemon to execute scheduled commands (Vixie Cron)
Each user gets a cron tab that lists which commands or scripts will be automatically run by the user
-
crontab -e
# edit crontab of current user -
format:
m h dom mon dow command
<minute[0-59]>
<hour[0-23]>
<day_of_the_month[1-31]>
<month[[1-12][JAN-DEC]]>
<day_of_the_week[[0-6][SUN-SAT]]>20 11 * * * <command>
means it will run at 11:20AM every day,month, year
20 11 10 JUN SUN <command>
this command will run at 11:20AM of JUN only if 10th JUN is Sunday
* * means any minute any day
* * * * * <command>
# the command will run every minute every hour
0,15,30,45 * * * * echo "Hello23"
# evry 15 minutes run this command
*/15 \* \3 \* \* echo "Hello23"
# evry 3 days every 15 minutes run this command
59 23 * JAN,DEC * echo Hello
# run the command only on JAN and DEC each day at 23:59
**Note: adding to environment variable $PATH won't let cron command know the command. **59 23 * * FRI bash ~/bin/script
# run the script file named /home/<user_name>/script
- Note: aliases can't be used for scheduling aliases are mainly used as shorter form of the original longer command steps:
- create a
.bash_aliases
file in home - alias aliasName="<long_command>" i.e
alias getPythonVersion="python --version"
- save the file
- restart terminal
- enter aliasName i.e
getPythonVersion
*
means abything i.e*.txt
matches any file with.txt
extension?
means any character occuring once i.e?,txt
means any file named 'A.txt' or 'B.txt'[]
means choose any one from inside of the square (must choose exactly one)file[123].txt
matchesfile1.txt
,file2.txt
,file3.txt
but doesn't selectfile12.txt
orfile.txt
echo {1..10}
outputs1 2 3 4 5 6 7 8 9 10
echo {A,B,C}-{1,2,3}
outputsA-1 A-2 A-3 B-1 B-2 B-3 C-1 C-2 C-3
mkdir {jan, feb, mar, apr, may}_{2019,2020}
# creates 5*2 directories
- Make tarball from the files using
tar
command - Compress tarball using
gzip
orbzip2
command to show current terminal file location
grep searches for PATTERN in each FILE.
grep returns lines that contain a piece of text wildcards also works
grep [OPTIONS] PATTERN [FILE...]
options:
- -C NUM, -NUM, --context=NUM Print NUM lines of output context.
- -v, --invert-match Invert the sense of matching, to select non-matching lines.
- -i, --ignore-case Ignore case distinctions, so that characters that differ only in case match each other.
grep command is canse-sensitive
grep -rnw './' -e '<text_to_find>'
# search for all files that contain a specific text
grep -rnwi './' -e '<text_to_find>'
# search for all files that contain a specific text case-insensitive
grep e hello.txt
# outputs the line(s) containing letter e in hello.txt
grep you -c gadsby.txt
# outputs number of references of word 'you' in gadsby.txt file
grep -i gadsby gadsby.txt
# case insensitive search
grep -i "our boys" gadsby.txt
# sentence search
grep -vi e gadsby.txt
# outputs those lines without the letter e
grep -ic "e" hello.txt gadsby.txt
# multiple file search
How do I find all files containing specific text on Linux?
search for specific keyword: man -k "your keyword here"
man -k "list directory contents"
Output:
dir (1) - list directory contents
ls (1) - list directory contents
ntfsls (8) - list directory contents on an NTFS filesystem
vdir (1) - list directory contents
reads from standard input and writes into standaed output
cat 1> output.txt
# redirects standard out
8000
put; cleans the taget file and writes into it
cat > output.txt
# redirects standard output; cleans the taget file and writes into it
cat >> output.txt
# redirects standard output; appends into the target file
cat 0< input.txt
# redirects standard input; reads from the input.txt and loads them into cat program
cat < input.txt
# redirects standard input; reads from the input.txt and loads them into cat program
cat 0< input.txt 1> output.txt
# redirects standard input; reads from the input.txt and redirects them into output.txt
cat 0< input.txt > /dev/pts/3
# redirects std input and std output; reads from input.txt and sends them to another terminal
cat file1.txt file2.txt file3.txt > compile.txt
# concatenating all of the outputs and puts them into compile.txt
cat file[1-3]txt > compile.txt
# concatenating all of the outputs and puts them into compile.txt
- Number that represents an open file
- Every new process starts with three file descriptor
- FD 0 STDIN
- FD 1 STDOUT
- FD 2 STDERR
- Linux represents everything as file
- Resources like keyboard, montior are treated like a file
- FDs are like pointers of sources of data or where data is displayed
tac - concatenate and print files in reverse
tac file.txt
# outputs the lines of the file in reverse order. the last line will be first. the first line will be last horizontally same vertically reversed
rev - reverse lines characterwise
rev file.txt
# vertically same horizontally reversed each line
less - opposite of more
head - output the first part of files
by default shows the first ten lines
head -n 5 file.txt
# outputs first 5 lines of file.txt
find . | head -n 2
# outputs only first 2 results
tail - output the last part of files
remove sections from each line of files
example:
date | cut -d " " -f 1 | cat > date.txt
# pipes from date to cut then get the first field based on space delimiter and then pipes to date.txt
read from standard input and write to standard output and files
example:
date | tee fulldate.txt | cut -d " " -f 1
# two pipes from date one gets the output to be written to fulldate.txt , the other goes to cut command
build and execute command lines from standard input
example: echo doesn't support standard input only supports command line argument
date | xargs echo
# prints the date in standard output
cat fileToDelete.txt | xargs rm
# delete file with filename written in fileToDelete.txt
determine file type
file [file_name]
Example:
file .bash_aliases
Output:
.bash_aliases: ASCII text
remove files or directories
rm -r delFolder
# deletes delFolder
rm -r delFolder/del{1,2,3}
# # deletes del,1 del2, del3 folders
rm -r delFolder/
# deletes delFolder and everything inside of it
rm -r delFolder/*
# deletes everything inside delFolder but not itself
rm -ri delFolder/
# interactive deletion: asks before deletion of each file or folder
rm delFile
# deletes delFile
rmdir - remove empty directories
rmdir delFolder
# removes delFolder only if it is empty
mkdir - make directories
mkdir -p aFolder/bFolder/cFile
# creates the full path along the way of creating folders and files
touch - change file timestamps
generally used to create empty file[s]
touch aFolder/file{1,2,3}.txt
# creates three empty files inside aFolder
cp - copy files and directories
cp [OPTION]... [-T] SOURCE DEST
cp [OPTION]... SOURCE... DIRECTORY
cp file1.txt file2.txt
# file1 is source file2 is destination
cp file1.txt file2.txt directory/
# file1 , file2 are sources directory is destination directory
cp directory/* .
# copy everything from directory to current folder
cp -r sourceDir/* destDir
# copy everything from sourceDir to destDir
sort - sort lines of text files
oprtions:
- -n, --numeric-sort compare according to string numerical value
- -M, --month-sort
- -r, --reverse reverse the result of comparisons
- -h, --human-numeric-sort compare human readable numbers (e.g., 2K 1G)
Note: -M and -n options can't be used together
sort file.txt
# sorts the content of file.txt alphabetically
sort -r file.txt
# sorts the content of file.txt in reverse alphabetical way
sort -n numbers.txt
# sort based on value of the numbers
sort -u numbers.txt
# sort and outputs only unique
ls -l | head -n 20 | sort -k 5 -n
# sorts files based on size
ls -lh | head -n 20 | sort -k 5 -hr
# human readable format sort
ls -lh /etc | head -n 20 | sort -k 6 -M
# sort by month
- temporarily substitute users
command -options arguments
which -a python3
# prints all matching executables in the PATH
mv - move (rename) files
mv oldName.txt newName.txt
# renames to newName.txt
mv oldFolder newFolder
# renames to newFolder
mv directory/* .
#moves everything from directory folder to current directory
locate - find files by name
locate command uses a database which gets updated once a day. this can show outdated result
in order to update the database use: sudo updatedb
options: -e: Print only entries that refer to files existing at the time locate is run.
locate utshaw
# searches for file or directories with case sensitive name utshaw in their path
locate -i uTsHaW
# searches for file or directories with case insensitive name utshaw in their path
locate -c uTsHaW
# count the number of occurrences of utshaw found
locate -e utshaw
# outputs actually existing file
sudo updatedb
# updates the database locate
command uses for searching
locate --existing --follow utshaw
# checks if the file really exists and omit broken symbolic links
find - search for files in a directory hierarchy
find
command doesn't require database to operate it's always upto date
locate
only outputs files whereas find
outputs files and folders
find
# outputs every files and directories from the current directory tree recursively
find . -maxdepth 1
# det maximum depth for search
find . -type f
# outputs only files
find . -type d
# outputs only directories
find -type f -size +100k
# outputs files with size > 100k
find -type f -size -100k
# outputs files with size < 100k
find -type f -size +100k -size -5M
# outputs files with size > 100k & <5M
sudo find -maxdepth 3 -type f -size +10k -size -12k -exec cp {} ~/Desktop/test/ \;
# copies each of those files to test folder
find -type f -name "test.txt"
# case-sensitive searches for test.txt from current directory tree
find -type f -name "test*"
# case-sensitive searches for files starting with test from current directory tree
find -type f -iname "test*"
# case-insensitive searches for files starting with test from current directory tree
find -type f -name "*.py"
# finds all files with extension py
find -type f -mmin -10
# finds all files those were modified less than ten minutes ago
find -type f -mmin +1 -mmin -10
# finds all files those were modified greater than one minute ago and less than ten minutes ago
tar - an archiving utility
options:
- -c, --create Create a new archive.
- -v, --verbose Verbosely list files processed.
- -f lets tar command accept files
- -t List the contents of an archive.
- -C, --directory=DIR Change to DIR before performing any operations.This option is order-sensitive,
i.e. it affects all options that follow.
tar -xvzf ideaIU-2019.2.4.tar.gz -C ~/
# extracts to home folder
tar -cvf ourArchive.tar file[1-3].txt
# create a new archive named ourArchive.tar
tar -tf ourArchive.tar
# outputs contents of ourArchive.tar
tar -xvf ourArchice.tar
# extracts ourArchive.tar
tar -cvzf ourArchive.tar.gz diary.txt utshaw.txt diary.txt
# create gzip file in one command
tar -cvjf ourArchive.tar.bz2 diary.txt utshaw.txt diary.txt
# create bz2 file in one command
tar -xvzf ourArchive.tar.gz
# unzips from gz file
tar -xvjf ourArchive.tar.bz2
# unzips from bz2 file
tar -xvJf ourArchive.tar.xz
# extract from tar.xz file
gzip is faster but with less compression power means file size won't be that small
gzip ourArchive.tar
# compress the tar file with gz extension
gunzip ourArchive.tar
# un-compress the gz file and give back tar file
bzip2 is slower but has higher compression power can compress to very small files
bzip2 ourArchice.tar
# compress the tar file with .bz2 extension
bunzip2 ourArchice.tar.bz2
# un-comress the bz2 file and gives back tar file
zip - package and compress (archive) files
zip ourZip.zip file1.txt file2.txt
# creates zip file named ourZip.zip from file1.txt and file2.txt
wc - print newline, word, and byte counts for each file
wc -l
# outputs number of lines the command received in the standard input
sudo apt-get install tmux
- Create new pane: Ctrl+B+%
- Switch between pane: Ctrl + B + -> / <-
- Quit a pane:
exit
- Create new window: Ctrl + b c
- Go to windows 1 : Ctrl + b 1
- Go to windows 0 : Ctrl + b 0
- Rename window:
Ctrl + b ,
then rename enter - Detach session:
Ctrl + b, d
- View tmux sessoins:
tmux ls
- Attach to running session:
tmux attach -t <session_name>
- Ex:
tmux attach -t 0
- Rename tmux session:
tmux rename-session -t <session_name> <renamed_session>
- Ex:
tmux rename-session -t 0 git
- Create a session with name
tmux new -s <name>
- Killing session:
tmux kill-session -t <session_name>
Use case scenario
tmux # start a session 0
source venv/bin/activate
export GOOGLE_APPLICATION_CREDENTIALS=/home/gcp_key.json
cd python; sudo python3 driver.py
Ctrl + b , d # detaching from session 0
tmux attach -t 0 # check the running session 0
- https://youtu.be/qE77MbDnljA Usage:
rsync -av get-docker.sh mirage@192.168.1.18:~/Desktop/
xclip -sel c < file
- Folow instructions youtube-dl
- Install ffmpeg
sudo apt-get install ffmpeg
- Cheat sheet: Linux Training Academy
- man page: https://www.youtube.com/watch?v=uJnrh9hAQR0
- crontab: https://crontab.guru/