Linux

Linux

Commands

• brief "cheat sheet" with common examples of how to use a shell command: curl cheat.sh/command

Processes

• kill all python processes: pkill -f "python" or for pid in $(ps -C python -o pid=); do kill -9 $pid ; done
• kill all processes for user: pkill -u <username>
• sort Linux ‘ps’ output by memory (RAM), from high to low: ps aux --sort -rss
• get PID from search in the process command + args: pgrep -f "string"
• process information from PID: ps -p $PID -o pid,vsz=MEMORY -o user,group=GROUP -o comm,args=ARGS
• debugging: pstack <pid>
• list services: systemctl list-units --type=service

Resources

• display CPU and IO statistics for devices and partitions (refresh every second): iostat -x -t 1
• print out number of cores/ processors: nproc --all
• display file status (size; access, modify and change time, etc) of a file (e.g. filename.txt): stat filename.txt
• collect and summarize all hardware info of your machine (json output): lshw -json >report.json

File system

• clone permissions from source folder to destination: chmod --reference=source destination
• find files greater than 10 MB in size and running a command on those files: find / -size +10M -exec command {} \;
• check which file make the device busy on umount: lsof /mnt/dir
• list all files with alphabet in its filename: ls -l [a-z]*
• list all .sh and .py files: ls *.{sh,py}
• empty a big directory (dir_to_be_emptied):

  EMPTYDIR=$(mktemp -d)
  rsync -a --delete $EMPTYDIR/ dir_to_be_emptied/

• replace symbolic links with a deep copy of the file: find ./ -type l -exec sh -c 'cp --remove-destination $(readlink "{}") "{}"' \;
• syncs your files to cloud storage: rclone
• copy a large folder: rclone copy --transfers 15 --checkers 15 --drive-chunk-size=16384 --fast-list --size-only --progress --buffer-size 500M --multi-thread-streams=20 --stats-one-line source destination

SSH

• generate ssh keys: ssh-keygen
• copy ssh key to another hosts to login without password: ssh-copy-id -i .ssh/id_rsa.pub [email protected]
• create a ssh tunnel:

  ssh -f -L 9000:targetservername:8088 [email protected] -N
  #-f: run in background; -L: Listen; -N: do nothing
  #the 9000 of your computer is now connected to the 8088 port of the targetservername through 192.168.14.72
  #so that you can see the content of targetservername:8088 by entering localhost:9000 from your browser.

• Tunnel all your traffic over Websocket or HTTP2 - Bypass firewalls/DPI: https://github.com/erebe/wstunnel
• Multiple jumphost config ~/.ssh/ssh_config:

Host myserver
	HostName myserver.example.com
	User virag
	IdentityFile /users/virag/keys/myserver-cert.pub
	ProxyJump jump
Host bastion
	#Used because HostName is unreliable as IP address changes frequently
	HostKeyAlias bastion.example
	User external
Host jump
	HostName jump.example.com
	User internal
	IdentityFile /users/virag/keys/jump-cert.pub
	ProxyJump bastion

Users

• add user to sudo users: sudo usermod -a -G sudo <username>
• enable passwordless for members of group sudo: sudo visudo => %sudo ALL=(ALL:ALL) NOPASSWD:ALL
• turn off the login banner (message of the day => /etc/motd): touch ~/.hushlogin
• use “passwd” as a database to get the information of the . It displays the username, user’s id, and folder name: getent passwd <username>

Grep

• grep for precise string: grep -r -w -o <string> .
• search for multiple patterns: grep -e value1 -e value2 files

Systemd

• auto start service at boot: sudo systemctl enable application.service
• check logs: sudo journalctl -u application.service

Sysctl

Modify kernel parameters at runtime.

• list all sysctl controlled parameters: sudo sysctl -a
• persist config changes: sudo vi /etc/sysctl.conf

Disk

• check space in use: df -h
• check inodes in use: df -ih
• list disks: lsblk
• determine whether there is a file system on the volume: sudo file -s /dev/sdc. If the output shows simply data there is no file system on the device. and you must create one
• create a file system on the volume: sudo mkfs -t xfs /dev/sdc

Diff

• Diff creating a patch: diff -Nur originalfile newfile > patchfile
• Apply the patch: patch -p1 < patchfile. Note, the --dry-run option can be used.
• Diff compressed files: zdiff file1.txt.gz file2.txt.gz, bzdiff is associated with bzip2, and xzdiff associated with xz
• find common/differing lines: comm -12 fileA fileB

Utils

• split file in 4G chunks split -b 4096m CentOS7.tgz CentOS7.tgz-split-
• combine splitted file - double question marks(??) match any two-character extension in the file name: cat CentOS7.tgz-split-?? > CentOS7.tgz
• apply command to all files .h in the folder: for file in *.h; do <command> $file; done
• append 2nd column from file2 to file1: paste <(cat file1.txt) <(cat file2.txt | awk '{ print $2}')
• print first and seventh field of every line: awk -F: '{ print $1 $7 }' /etc/passwd
• convert all uppercase filenames to lowercase: rename 'y/A-Z/a-z/' *
• rename using find: find / -type f -name 'howtouselinux*' -exec mv {} {}_renamed \;
• rename all *.bak file as *.txt: rename .bak .txt *.bak
• convert mystring to uppercase: echo mystring | tr “[a-z]” “[A-Z]” or echo mystring | tr “[:lower:]” “[:upper:]”
• copy file while converting characters to upper-case: dd if=/tmp/group of=/tmp/GROUP conv=ucase
• backup device: dd if=/dev/input/DEVICE-HERE of=/dev/OUTPUT/DEVICE-HERE bs=64K conv=noerror,sync
• remove newline: tr --delete '\n' <input.txt >output.txt
• sleep indefinitely until a signal is received: trap 'echo "Interrupted"; exit' INT; sleep infinity
• allocate 1GB ram: cat /dev/zero | head -c 1G | tail
• run a command with a time limit. Example, run top for 5 seconds: timeout 5s top

Redirection

• both stdout and stderr will be dumped into /dev/null: ls -lR /tmp >& /dev/null
• pipe string message: printf "message\n" | cat /dev/stdin
• process substitution (which generates a file): cat <(printf "message\n")
• pipe yes to another command that prompts for confirmation: yes | command

Shell

• change the default shell to bash: chsh -s /bin/bash

Sed

• remove CTRL+M: sed -e "s/\r//g" file > newfile
• convert 01/02/ to JAN/FEB/: sed -e 's/01/JAN/' -e 's/02/FEB/'
• remove lines with string, (-i) for inline: sed "/string/d" filename, case insensitive: sed "/string/Id" filename

Date

• yesterday date: yesterday=$(date -v-1d +"%Y-%m-%d")
• add (180) days to date (2021-05-29) and get new date: date -d "2021-05-29+180 days"
• get current epoch time: date +%s
• convert a epoch time (1284763671) digit to Coordinated Universal Time: date -ud@1284763671

Cron

• view crontab entries of current users: crontab -l
• to add or update jobs in crontab: crontab -e
• schedule a cron to execute at 2am daily: 0 2 * * * /bin/sh backup.sh
• schedule tasks to execute daily: @daily /scripts/script.sh

Network

• verify access to a running webserver on port 80: nc -vz www.linux.com 80
• check if remote file exists: wget --spider --timeout=10 --tries=1 https://raw.githubusercontent.com/romitagl/kgraph/master/README.md
• print content to screen: wget -qO- www.linux.com
• check process using port 9000: sudo lsof -i:9000
• display routing table: netstat -rn or ip -c r | column -t
• monitor network bandwidth utilization and rate estimator: bmon

Linux Networking & Security Fundamentals Course: https://github.com/daveprowse/lnsf

Tcpdump

• sudo tcpdump -i eth0 'tcp port 8080' -nnvXSs 0
• sudo tcpdump -i lo -s 65535 -A port 8080 -w /tmp/trace.cap
• tcpdump -i eth0 dst platform.clickatell.com

Curl

• post data from file: curl -X POST localhost:8080/admin/schema -d '@schema.graphql'
• authentication: curl -u user:pass -O ftp://remote_url/file-to-download.zip
• download file (e.g. Amazon CLI): curl -L "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
• get with parameters: curl -G --data-urlencode "key=value" --data-urlencode "key2=value2" http://localhost:8000/endpoint
• SSL certificate problem - unable to get local issuer certificate: export CURL_CA_BUNDLE='/etc/ssl/certs/ca-certificates.crt'. More info at: https://curl.se/docs/sslcerts.html

Wget

• follow original filename: wget --content-disposition $url

Certificates

• extract the CA cert for a particular server: openssl s_client -showcerts -servername server -connect server:443 > cacert.pem type "quit", followed by the "ENTER" key
• to see the data in the certificate: openssl x509 -inform PEM -in certfile -text -out certdata

Vi

• find and replace string: :%s/<search_string>/<replace_string>/g
• delete all lines :1,$d
• display line numbers: set number
• enable syntax highlighting: syntax on
• highlight the current line: set cursorline
• yaml settings in ~/.vimrc - append the following lines:

set nu
set ic
set expandtab
set shiftwidth=2
set tabstop=2

Sessions Management

screen

# Create session and attach:
screen

# Create a screen and name it 'test'
screen -S test

# Detached session foo:
screen: ^a^d

# List sessions:
screen -ls

# Attach last session:
screen -r

# Attach to session foo:
screen -r foo

# Kill session foo:
screen -r foo -X quit

# Scroll:
# Hit your screen prefix combination (C-a / control+A), then hit Escape.
# Move up/down with the arrow keys (↑ and ↓).  

# Redirect output of an already running process in Screen:
# (C-a / control+A), then hit 'H'  

# Store screen output for Screen:
# Ctrl+A, Shift+H  
# You will then find a screen.log file under current directory.  

tumx

# Create session and attach:
tmux

# Attach to session foo:
tmux attach -t foo

# Detached session foo:
^bd

# List sessions:
tmux ls

# Attach last session:
tmux attach

# Kill session foo:
tmux kill-session -t foo

# Create detached session foo:
tmux new -s foo -d

# Send command to all panes in tmux:
Ctrl-B
:setw synchronize-panes

# Some tmux pane control commands:
Ctrl-B
#   Panes (splits), Press Ctrl+B, then input the following symbol:
#   %  horizontal split
#   "  vertical split
#   o  swap panes
#   q  show pane numbers
#   x  kill pane
#   space - toggle between layouts

#   Distribute Vertically (rows):
select-layout even-vertical
#   or
Ctrl+b, Alt+2

# Distribute horizontally (columns):
select-layout even-horizontal
#   or
Ctrl+b, Alt+1

# Scroll
Ctrl-b then \[ then you can use your normal navigation keys to scroll around.
Press q to quit scroll mode.

Distros

• Vanilla OS is an immutable and atomic Ubuntu Linux-based distribution

Books

• Advanced Linux Programming: https://mentorembedded.github.io/advancedlinuxprogramming/alp-folder/