Operating system usually segregates virtual memory into user space and kernel space. Primarily, this separation serves to provide memory protection and hardware protection from malicious or errant software behaviour.
+-------------------------------------------------------------------------+
| |User | GUI, Servers, Shell, GIMP, Blender, Mozilla |
| |applications| |
| |-----------------------------------------------------------------|
|User |System |init |System |Window |Graphics:|Other |
|mode |components |daemon: |daemons: |manager: | |libraries:|
| | |systemd |polkitd |X11 |AMD |GTK |
| |-----------------------------------------------------------------|
| |C standard | fopen, execv, malloc,... (up to 2000 sub) |
| |library | glibc aims to be fast, musl aims to be lightw |
|-------------------------------------------------------------------------|
| | | stat, splice, dup, read, open, ioctl, write |
| | | mmap, close, exit,... (about 380 system calls) |
| | | System Call Interface (SCI), aims to be POSIX |
| | |----------------------------------------------------|
|Kernel | Linux |Process |IPC |Memory |Virtual |Networking|
|mode | kernel |scheduling|subsystem|management|files |subsystem |
| | |subsystem | |subsystem |subsystem| |
| | |----------------------------------------------------|
| | | Other components: ALSA, DRI, evdev, klibc, LVM |
| | | device mapper, Linux Network Scheduler, Netfilter |
| | | Linux Security Modules: SELinux, TOMOYO, AppArmor |
|-------------------------------------------------------------------------|
| Hardware (CPU, Main Memory, Data Storage Devices, Network Ports) |
+-------------------------------------------------------------------------+
Is an entity that can run processes and own files. Users exist primarily to support permissions and boundaries. Every user-space process has a user owner, and processes are said to run as the owner. A user may terminate or modify the behavior of its own processes (within certain limits), but it cannot interfere with other users’ processes.
The root user is an exception to the preceding rules because root may terminate and alter another user’s processes and access any file on the local system. For this reason, root is known as the superuser.
Groups are sets of users. The primary purpose of groups is to allow a user to share file access to other members of a group.
Is the main memory that the kernel allocates for user processes. If a process makes a mistake and crashes, the consequences are limited and can be cleaned up by the kernel.
Because a process is simply a state (or image) in memory, user space also refers to the memory for the entire collection of running processes.
The bottom level of user space tends to consist of small components that perform single, uncomplicated tasks (Network configuration, Communication Bus, Diagnostic Logging,..). The middle level has larger components such as mail, print, and database services. Finally, components at the top level perform complicated tasks that the user often controls directly (Web Browser, User Interface, …).
init.- Essential low-level services, such as
udevdandsyslogd. - Network configuration.
- Mid- and high-level services (cron, printing, and so on).
- Login prompts, GUIs, and high-level applications, such as web servers.
You may know some software like Firefox, GIMP, etc, but what they really are is just lots and lots of files that have been compiled into one. The people (or sometimes a single person) that write this software are known as upstream providers, they compile their code and write up how to get it installed.
These upstream providers work on getting new software out and update existing software. When they are ready to release it to the world, they send their package to package maintainers who handle getting this piece of software in the hands of the users. These package maintainers review, manage, and distribute this software in the form of packages. So instead of downloading packages/software one by one from random sites (like on Windows), there is something better – called package repositories in Debian.
Repositories are just a central storage location for packages. Many online repositories hold lots of packages. Your machine doesn’t know where to look for these repositories unless you explicitly tell it where to look. Debian already comes with pre-approved sources to get packages from and this is how it installs all the base packages you see on your system (if a user did a net-install). On a Debian system, this sources file is the /etc/apt/sources.list file. Your machine will know to look there and check for any source repositories you added.
The basic unit of software in a packaging system is the package file. A package file is a compressed collection of files that comprise the software package. A package may consist of numerous programs and data files that support the programs. In addition to the files to be installed, the package file also includes metadata about the package, such as a text description of the package and its contents. Additionally, many packages contain pre- and post-installation scripts that perform configuration tasks before and after the package installation.
Package files are created by a person known as a package maintainer, often (but not always) an employee of the distribution vendor. The package maintainer gets the software in source code form from the upstream provider (the author of the program), compiles it, and creates the package metadata and any necessary installation scripts. Often, the package maintainer will apply modifications to the original source code to improve the program’s integration with the other parts of the Linux distribution.
While some software projects choose to perform their own packaging and distribution, most packages today are created by the distribution vendors and interested third parties. Packages are made available to the users of a distribution in central repositories that may contain many thousands of packages, each specially built and maintained for the distribution.
A distribution may maintain several different repositories for different stages of the software development life cycle. For example, testing, development and also have related third-party repositories.
Programs are seldom “standalone”; rather they rely on the presence of other software components to get their work done. Common activities, such as input/output for example, are handled by routines shared by many programs. These routines are stored in what are called shared libraries, which provide essential services to more than one program. If a package requires a shared resource such as a shared library, it is said to have a dependency. Modern package management systems all provide some method of dependency resolution to ensure that when a package is installed, all of its dependencies are installed, too.
Package management systems usually consist of two types of tools.
- Low-level tools which handle tasks such as installing and removing package files, like
dpkg. - High-level tools that perform metadata searching and dependency resolution, like
apt.
apt provides a high-level command line interface for debian’s package management system, with all the necessary tools in one place. apt aims to provide a simple, efficient way of handling packages for end users.
All packages that are included in the official Debian distribution are free according to the Debian Free Software Guidelines. This assures free use and redistribution of the packages and their complete source code. The official Debian distribution is what is contained in the main section of the Debian archive.
As a service to our users, we also provide packages in separate sections that cannot be included in the main distribution due to either a restrictive license or legal issues. They include:
- contrib
- packages in this area are freely licensed by the copyright holder but depend on other software that is not free.
- non-free
- packages in this area have some onerous license condition restricting use or redistribution of the software.
- non-free-firmware
- this section is restricted to device firmware. While packages in this area basically have the same underlying license restrictions as those in non-free, a special exception has been made for them: even if not being DFSG-free, packages from non-free-firmware are allowed to be included in the official installation images.
Note that same packages might appear in several distributions, but with different version numbers.
Package lists in the:
- stable distribution
- this is the latest official release of the Debian distribution. This is stable and well tested software, which changes only if major security or usability fixes are incorporated. The current “stable” distribution is version 12, codenamed bookworm. It was initially released as version 12.0 on June 10th, 2023.
- testing distribution
- this area contains packages that are intended to become part of the next stable distribution. There are strict criteria a package in unstable must obey before it can be added to testing. Note that “testing” does not get the timely security updates from the security team. The main advantage of using this distribution is that it has more recent versions of software. The current “testing” distribution is trixie.
- unstable distribution
- this area contains the most recent packages in Debian. Once a package has met our criterion for stability and quality of packaging, it will be included in testing. “unstable” is also not supported by the security team. The “unstable” distribution is always called sid.
Corollaries to this in the commercial world are Development, Testing, and Production.
Debian Unstable is not strictly a release, but rather a rolling development version of the Debian distribution containing the latest packages that have been introduced into Debian.
While other release code names progress in time from being “testing” to being “stable”, Sid is forever doomed to being unstable. Sid will always be the unstable branch. When the current “testing” repository becomes mature and is released, “testing” becomes the latest “stable” release. From there, a new “testing” repository will be created with the next planned code name, and packages will continue to trickle down from Sid into “testing” just as before.
Sid is where packages go after they’ve been uploaded by their maintainer, and cleared for release by the FTP master. When packages have met certain criteria, they are automatically moved from Sid to the current “testing” repository. The “Unstable” repository is updated every 6 hours.
Sid exclusively gets security updates through its package maintainers. The Debian Security Team only maintains security updates for the current “stable” release.
Ways to get set up on Sid:
- Use the current “stable” installer to install a minimal stable system (recommended).
- Change your apt sources to point to “unstable”.
- Run
apt updateandapt full-upgrade.
- Use the current “testing” installer to install a minimal (next) stable system.
- Change your apt sources to point to “unstable”.
- Run
apt updateandapt full-upgrade.
- Use the Unstable “mini.iso” image.
- Download the “mini.iso” for your CPU architecture located here: https://d-i.debian.org/daily-images/ under */daily/netboot/
- During the installation choose “Advanced options” -> “Expert install”.
- In the step “Choose a mirror of the Debian archive” choose version “sid - unstable”
If you are already using a stable system use the method described above for upgrading to unstable changing your apt sources from “stable” (or the codename for the current stable) to “unstable”.
The entries in this file normally follow this format:
deb http://site.example.com/debian distribution comp1 comp2 comp3
deb-src http://site.example.com/debian distribution comp1 comp2 comp3
The first word on each line, deb or deb-src, indicates the type of archive. Deb indicates that the archive contains binary packages (deb), the pre-compiled packages that we normally use. Deb-src indicates source packages, which are the original program sources plus the Debian control file (.dsc) and the diff.gz containing the changes needed for packaging the program.
The next entry on the line is a URL to the repository that you want to download the packages from.
The ‘distribution’ can be either the release code name / alias ( stretch, buster, bullseye, bookworm, sid) or the release class (oldoldstable, oldstable, stable, testing, unstable) respectively.
- main
- consists of DFSG-compliant packages, which do not rely on software outside this area to operate. These are the only packages considered part of the Debian distribution.
- contrib
- packages contain DFSG-compliant software, but have dependencies not in main (possibly packaged for Debian in non-free).
- non-free
- contains software that does not comply with the DFSG.
deb http://deb.debian.org/debian bookworm main non-free-firmware
deb-src http://deb.debian.org/debian bookworm main non-free-firmware
deb http://deb.debian.org/debian-security/ bookworm-security main non-free-firmware
deb-src http://deb.debian.org/debian-security/ bookworm-security main non-free-firmware
deb http://deb.debian.org/debian bookworm-updates main non-free-firmware
deb-src http://deb.debian.org/debian bookworm-updates main non-free-firmware
If you need the contrib, non-free and non-free-firmware components, add contrib non-free non-free-firmware after main. For example, for Debian 12/Bookworm:
deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware
deb-src http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware
deb http://deb.debian.org/debian-security/ bookworm-security main contrib non-free non-free-firmware
deb-src http://deb.debian.org/debian-security/ bookworm-security main contrib non-free non-free-firmware
deb http://deb.debian.org/debian bookworm-updates main contrib non-free non-free-firmware
deb-src http://deb.debian.org/debian bookworm-updates main contrib non-free non-free-firmware
# Append the following line to the bottom of the file:
deb http://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware
deb-src http://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware
# Now that you have added the repository, update APT's cache to include
# the backports in the list of available packages:
apt update
deb http://deb.debian.org/debian/ unstable main
deb-src http://deb.debian.org/debian/ unstable main
deb http://ftp.br.debian.org/debian/ unstable main
deb-src http://ftp.br.debian.org/debian/ unstable main
# or
deb ftp.br.debian.org/debian/ unstable main
deb-src ftp.br.debian.org/debian/ unstable main
Debian is distributed (aka mirrored) on hundreds of servers worldwide, all offering the same content. This way we can provide better access to our archive.
You can find the following content on our mirror servers:
- Debian Packages (debian/)
- the Debian package pool: this includes the vast majority of .deb packages, installation material, and the sources.
- CD Images (debian-cd/)
- the repository of CD images: Jigdo files and ISO image files. See the list of Debian mirrors that include the debian-cd/ archive.
- Old Releases (debian-archive/)
- the archive of older Debian versions, released in the past.
Data compression is the process of removing redundancy from data.
Compression algorithms fall into two general categories.
- Lossless
- Lossless compression preserves all the data contained in the original. This means that when a file is restored from a compressed version, the restored file is exactly the same as the original, uncompressed version.
- Lossy
- Lossy compression, on the other hand, removes data as the compression is performed to allow more compression to be applied. When a lossy file is restored, it does not match the original version; rather, it is a close approximation. Exam- ples of lossy compression are JPEG (for images) and MP3 (for music).
A common file-management task often used in conjunction with compression is archiving. Archiving is the process of gathering up many files and bundling them together into a single large file.
Any jobs under the purview of the system administrator should be in /etc, since they are configuration files. If you have a root cron job for daily, weekly, or monthly runs, put them in /etc/cron.{daily,weekly,monthly}. These are invoked from /etc/crontab, and will run in alphabetic order, which serializes them.
On the other hand, if you have a cron job that (a) needs to run as a special user, or (b) needs to run at a special time or frequency, you can use either /etc/crontab, or, better yet, /etc/cron.d/whatever. These particular files also have an extra field that allows you to stipulate the user account under which the cron job runs.
In either case, you just edit the files and cron will notice them automatically. There is no need to run a special command.
Debian Backports provides new packages with new features on supported Debian stable releases.
Note (debian the beginners handbook): … what do you mean exactly by “backports” ? Nothing to do, in fact, with the “backdoors” used to spy on computers running proprietary systems… The backport is a mechanism allowing an application currently hold in the Debian development repositories, to be ported back to the “stable” version.
As a matter of Backports policy, packages in the stable-backports suite are taken from Debian testing; however, in rare cases such as security updates, packages are taken from unstable. Backports Policy permits this exclusively when such packages will be part of the next Debian stable release. In any case, the new packages are recompiled for use on Debian stable. Backports are expected to make use of the versions of libraries available in Debian stable whenever possible; however, when required, the backporting of additional dependencies is permitted.
Backports cannot be tested as extensively as Debian stable, and are thus supported on a best-effort basis; there is a risk of incompatibilities with other components in Debian stable, so backports should be used with care! That said, Debian Backports Policy does not allow backports of libraries that would break all dependent packages in stable (eg: new Qt 5.x releases), and by virtue of this, Debian Backports are considered generally safe when used as intended on an individual package basis.
The coinstallability of all available backports is not tested, and it is strongly recommended to opt-into the use of specific backported packages on an as-needed basis.
As root, or using sudo, open your sources.list file (Nano is the recommended editor for new users):
sudo apt edit-sources
Append the following line to the bottom of the file:
deb http://deb.debian.org/debian bookworm-backports main
Run apt update.
The backports repository is low-priority by default. So, if you want to install a backported package, you will have to state that explicitly.
For example:
sudo apt install cockpit/bookworm-backports
This will attempt to install Cockpit from bookworm-backports, preferring dependencies from stable. Sometimes backports depend on other backports, and it is necessary to specify these dependencies aswell.
(Dependency is just a stand in for a theoretical backport dependency)
sudo apt install cockpit/bookworm-backports dependency/bookworm-backports
Another option is to use the -t target release flag, but this can sometimes lead to unnecessarily downloading dependencies from backports.
sudo apt -t bookworm-backports install cockpit
The -t option here specifies bookworm-backports as the target release. This would install a newer version of Cockpit and all its reverse dependencies from bookworm-backports instead of the older one from Debian stable release.
There are three main functions that make up an e-mail system. First there is the Mail User Agent (MUA) which is the program a user actually uses to compose and read mails. Then there is the Mail Transfer Agent (MTA) that takes care of transferring messages from one computer to another. And last there is the Mail Delivery Agent (MDA) that takes care of delivering incoming mail to the user’s inbox.
These three functions can be performed by separate programs, but they can also be combined in one or two programs. It is also possible to have different programs handle these functions for different types of mail.
On Linux and Unix systems mutt is historically a very popular MUA. Like most traditional Linux programs it is text based. It is often used in combination with exim or sendmail as MTA and procmail as MDA.
With the increasing popularity of graphical desktop systems, the use of graphical e-mail programs like GNOME’s evolution, KDE’s kmail or Mozilla’s thunderbird has becoming more popular. These programs combine the function of a MUA, MTA and MDA, but can — and often are — also be used in combination with the traditional Linux tools.
Even if you are planning to use a graphical mail program, it would be useful, to have a traditional MTA/MDA installed and correctly set up on your Debian GNU/Linux system. Reason is that various utilities running on the system can send important notices by e-mail to inform the system administrator of (potential) problems or changes.
For this you can install exim4 and mutt with apt install exim4 mutt. exim4 is a combination MTA/MDA that is relatively small but very flexible. By default it will be configured to only handle e-mail local to the system itself and e-mails addressed to the system administrator (root account) will be delivered to the regular user account created during the installation.
When system e-mails are delivered they are added to a file in /var/mail/$username. The e-mails can be read using mutt.
As mentioned earlier, the installed Debian system is only set up to handle e-mail local to the system, not for sending mail to others nor for receiving mail from others.
If you intend to use a graphical mail program and use a mail server of your Internet Service Provider (ISP) or your company, there is not really any need to configure exim4 for handling external e-mail. Just configure your favorite graphical mail program to use the correct servers to send and receive e-mail (how is outside the scope of this manual).
However, in that case you may need to configure individual utilities to correctly send e-mails. One such utility is reportbug, a program that facilitates submitting bug reports against Debian packages. By default it expects to be able to use exim4 to submit bug reports.
To correctly set up reportbug to use an external mail server, please run the command reportbug --configure and answer “no” to the question if an MTA is available. You will then be asked for the SMTP server to be used for submitting bug reports.
If you would like your system to also handle external e-mail, you will need to reconfigure the exim4 package or another alternative.
sudo dpkg-reconfigure exim4-config
After entering that command (as root), you will be asked if you want split the configuration into small files. If you are unsure, select the default option.
Next you will be presented with several common mail scenarios. Choose the one that most closely resembles your needs.
- internet site
- your system is connected to a network and your mail is sent and received directly using SMTP. On the following screens you will be asked a few basic questions, like your machine’s mail name, or a list of domains for which you accept or relay mail.
- mail sent by smarthost
- in this scenario your outgoing mail is forwarded to another machine, called a “smarthost”, which takes care of sending the message on to its destination. The smarthost also usually stores incoming mail addressed to your computer, so you don’t need to be permanently online. That also means you have to download your mail from the smarthost via programs like fetchmail.
In a lot of cases the smarthost will be your ISP’s mail server, which makes this option very suitable for dial-up users. It can also be a company mail server, or even another system on your own network.
- mail sent by smarthost; no local mail
- this option is basically the same as the previous one except that the system will not be set up to handle mail for a local e-mail domain. Mail on the system itself (e.g. for the system administrator) will still be handled.
- local delivery only
- this is the option your system is configured for by default.
- no configuration at this time
- choose this if you are absolutely convinced you know what you are doing. This will leave you with an unconfigured mail system — until you configure it, you won’t be able to send or receive any mail and you may miss some important messages from your system utilities.
If none of these scenarios suits your needs, or if you need a finer grained setup, you will need to edit configuration files under the /etc/exim4 directory after the installation is complete. More information about exim4 may be found under /usr/share/doc/exim4; the file README.Debian.gz has further details about configuring exim4 and explains where to find additional documentation.
Note that sending mail directly to the Internet when you don’t have an official domain name, can result in your mail being rejected because of anti-spam measures on receiving servers. Using your ISP’s mail server is preferred. If you still do want to send out mail directly, you may want to use a different e-mail address than is generated by default. If you use exim4 as your MTA, this is possible by adding an entry in /etc/email-addresses.
Is simply a state (or image) in memory.
Running programs that the kernel manages.
In Linux, some processes are divided into pieces called threads. A thread is very similar to a process—it has an identifier (thread ID, or TID), and the kernel schedules and runs threads just like processes. However, unlike separate processes, which usually don’t share system resources such as memory and I/O connections with other processes, all threads inside a single process share their system resources and some memory.
Many processes have only one thread. A process with one thread is single-threaded, and a process with more than one thread is multithreaded. All processes start out single-threaded. This starting thread is usually called the main thread. The main thread may start new threads, making the process multithreaded, similar to the way a process can call fork() to start a new process.
The primary advantage of a multithreaded process is that when the process has a lot to do, threads can run simultaneously on multiple processors, potentially speeding up computation. Although you can also achieve simultaneous computation with multiple processes, threads start faster than processes, and it’s often easier or more efficient for threads to intercommunicate using their shared memory than it is for processes to communicate over a channel, such as a network connection or a pipe.
Some programs use threads to overcome problems managing multiple I/O resources. Traditionally, a process would sometimes use fork() to start a new subprocess in order to deal with a new input or output stream. Threads offer a similar mechanism without the overhead of starting a new process.
A process that connects to the terminal is called a foreground job. A job is said to be in the foreground because it can communicate with the user via the screen and the keyboard.
On the other hand, a process that disconnects from the terminal and cannot communicate with the user is called a background job. If the background job requires interaction with the user, it will stop and wait until establishing a connection to the terminal.
We can place the jobs that do not require interaction from the user as they run (like sorting a large file) in the background. This allows the user to access the terminal and continue to work, instead of waiting for a long job to finish.
+-----------------+
$ cmd --->| Foreground |--------------\
+-----------------+ \
| ^ ^ \
ctrl-z| fg| \ \
| | \fg \
v | \ L
+-----------------+ \ +----------+
| Stopped |--kill---job#-->| Killed |
+-----------------+ / +----------+
| ^ / ^
bg| stop job#| / /
| | / /
v | / /
+-----------------+ /
$ cmd & -->| Background |--------------/
+-----------------+
- HUP Hangup (1)
- this is a vestige of the good old days when terminals were attached to remote computers with phone lines and modems. The signal is used to indicate to programs that the controlling terminal has “hung up.” The effect of this signal can be demonstrated by closing a terminal session. The foreground program running on the terminal will be sent the signal and will terminate. This signal is also used by many daemon programs to cause a reinitialization. This means that when a daemon is sent this signal, it will restart and reread its configuration file. The Apache web server is an example of a daemon that uses the
HUPsignal in this way. - INT Interrupt (2)
- this performs the same function as a
Ctrl-csent from the terminal. It will usually terminate a program. - KILL Kill (9)
- this signal is special. Whereas programs may choose to handle signals sent to them in different ways, including ignoring them all together, the
KILLsignal is never actually sent to the target program. Rather, the kernel immediately terminates the process. When a process is terminated in this manner, it is given no opportunity to “clean up” after itself or save its work. For this reason, theKILLsignal should be used only as a last resort when other termination signals fail. - TERM Terminate (15)
- this is the default signal sent by the kill command. If a program is still “alive” enough to receive signals, it will terminate.
- CONT Continue (18)
- this will restore a process after a
STOPorTSTPsignal. This signal is sent by thebgandfgcommands. - STOP Stop (19)
- this signal causes a process to pause without terminating. Like the
KILLsignal, it is not sent to the target process, and thus it cannot be ignored. - TSTP Terminal stop (20)
- this is the signal sent by the terminal when
Ctrl-zis pressed. Unlike theSTOPsignal, theTSTPsignal is received by the program, but the program may choose to ignore it. - SEGV Segmentation violation (11)
- this signal is sent if a program makes illegal use of memory, that is, if it tried to write somewhere it was not allowed to write.
- WINCH Window change (28)
- this is the signal sent by the system when a window changes size. Some programs , such as
topandlesswill respond to this signal by redrawing themselves to fit the new window dimensions. - QUIT (3)
- quit.
The basic idea is that you place several processes into a cgroup, which allows you to manage the resources that they consume on a group-wide basis. For example, if you want to limit the amount of memory that a set of processes may cumulatively consume, a cgroup can do this.
After creating a cgroup, you can add processes to it, and then use a controller to change how those processes behave. For example, there is a cpu controller allowing you to limit the processor time, a memory controller, and so on.
There are two versions of cgroups, 1 and 2. Aside from a somewhat different feature set, the structural differences between the versions can be summed up as follows:
- cgroups v1
- each type of controller (
cpu,memory, and so on) has its own set of cgroups. A process can belong to one cgroup per controller, meaning that a process can belong to multiple cgroups. For example, in v1, a process can belong to a cpu cgroup and a memory cgroup. - cgroups v2
- a process can belong to only one cgroup. You can set up different types of controllers for each cgroup.
To visualize the difference, consider three sets of processes, A, B, and C. We want to use the cpu and memory controllers on each of them.
The next figure shows the schematic for cgroups v1. We need six
cgroups total, because each cgroup is limited to a single controller.
+-----------------+ +--------------------+
| CPU controllers | | Memory controllers |
| +-----------+ | | +-----------+ |
| | cgroup A1 | | | | cgroup A2 | |
| +-----------+ | | +-----------+ |
| +-----------+ | | +-----------+ |
| | cgroup B1 | | | | cgroup B2 | |
| +-----------+ | | +-----------+ |
| +-----------+ | | +-----------+ |
| | cgroup C1 | | | | cgroup C2 | |
| +-----------+ | | +-----------+ |
+-----------------+ +--------------------+
The next figure shows how to do it in cgroups v2. We need only three
cgroups, because we can set up multiple controllers per cgroup.
+---------------------+ +---------------------+ +---------------------+
| cgroup A | | cgroup B | | cgroup C |
| +-----------------+ | | +-----------------+ | | +-----------------+ |
| | CPU controller | | | | CPU controller | | | | CPU controller | |
| +-----------------+ | | +-----------------+ | | +-----------------+ |
| +-----------------+ | | +-----------------+ | | +-----------------+ |
| |Memory controller| | | |Memory controller| | | |Memory controller| |
| +-----------------+ | | +-----------------+ | | +-----------------+ |
+---------------------+ +---------------------+ +---------------------+
Unix processes use I/O streams to read and write data. Processes read data from input streams and write data to output streams. The source of an input stream can be a file, a device, a terminal window, or even the output stream from another process.
Standard output is similar. The kernel gives each process a standard output stream where it can write its output. Standard input and output are often abbreviated as stdin and stdout.
There is a third standard I/O stream, called standard error (stderr); it’s an additional output stream for diagnostics and debugging.
Although lots of sh manual pages don’t mention this, the shell reads arguments from left to right.
cat food 2>&1 > file
On the first command line, the shell sees 2>&1 first. That means “make the standard error fd2 go to the same place as the standard output fd1 is going.” There’s no effect because both fd2 and fd1 are already going to the terminal. Then > file redirects fd1 (stdout) to file. But fd2 (stderr) is still going to the terminal.
cat food > file 2>&1
On the second command line, the shell sees > file first and redirects stdout to file. Next 2>&1 sends fd2 (stderr) to the same place fd1 is going - that’s to the file. And that’s what you want.
n>&m
The Bourne shell operator n>&m rearranges the files and file descriptors. It says “make file descriptor n point to the same file as file descriptor m.”
You can use more than one of those n>&m operators. The shell reads them left-to-right before it executes the command.
To swap standard output and standard error - make stderr go down a pipe and stdout go to the screen. This is one place the other file descriptors, 3 through 9, come in handy. They normally aren’t used. You can use one of them as a “holding place,” to remember where another file descriptor “pointed.” For example, one way to read the operator 3>&2 is “make 3 point the same place as 2.” After you use 3>&2 to grab the location of 2, you can make 2 point somewhere else. Then, make 1 point where 2 used to (where 3 points now).
command 3>&2 2>&1 1>&3 | ...
var=`grep "Joe" file1 file2`
# grep: file2: No such file or directory
echo "$var"
# file1: Joe Jones 423-4567
var=`grep "Joe" file1 file2 3>&2 2>&1 1>&3`
# 3>&2 crea un FD3 que apunte a donde apunta el FD2 (en este caso a la
# pantalla), es decir, el FD3 ahora apunta a la pantalla. Luego 2>&1
# coloca el FD2 que apunte a el FD1 (en este caso al backquotes), es
# decir, ahora el FD2 no apunta a donde estaba antes (la pantalla) sino
# al backquotes. Luego 1>&3 coloca el FD1 que apunte a el FD3 (en este
# caso la pantalla), es decir, el FD1 y el FD3 ambos apuntan a la
# pantalla.
# file1: Joe Jones 423-4567
echo "$var"
# grep: file2: No such file or directory
Open files are automatically closed when a process exits. But it’s safer to close the files yourself as soon as you’re done with them. That way, if you forget and use the same descriptor later for something else (for instance, use F.D. 3 to redirect some other command, or a subprocess uses F.D. 3 ), you won’t run into conflicts. Use m<&- to close input file descriptor m and m>&- to close output file descriptor m. If you need to close standard input, use <&-; >&- will close standard output.
Is a mechanism for inter-process communication using message passing. A pipeline is a set of processes chained together by their standard streams, so that the output text of each process (stdout) is passed directly as input (stdin) to the next one. The second process is started as the first process is still executing, and they are executed concurrently.
This pipeline is a anonymous pipes, where data written by one process is buffered by the operating system until it is read by the next process, and this uni-directional channel disappears when the processes are completed.
Each ”|” tells the shell to connect the standard output of the command on the left to the standard input of the command on the right by an inter-process communication (IPC) mechanism called an (anonymous) pipe, implemented in the operating system. Pipes are unidirectional; data flows through the pipeline from left to right.
In bash (and other shells such as sh), pipelines create subshells. These are copies of the shell and its environment that are used to execute the command in the pipeline.
Subshells in Unix-like systems create copies of the environment for the processes to use while they execute. When the processes finishes, the copy of the environment is destroyed. This means that a subshell can never alter the environment of its parent process.
Ejemplo de Daniel Clemente:
Prueba un echo Hola | nano (no explicaré el | aquí), verás que no puedes escribir (lógico, porque la entrada la coge del echo Hola, no de STDIN). Ahora prueba un cat | nano, irá perfecto porque cat coge la entrada y la manda por la salida (pruébalo suelto, un cat). Pues, ¿por qué no pruebas a ponerlo todo como un sólo comando? Haz (echo Hola; cat) | nano
PS: esto lo he visto en exploits que dan shell; si un echo $shellcode | ./programa crea una shell pero no te permite escribir, usa este truco.
Is an extension to the traditional pipe concept, and is one of the methods of inter-process communication (IPC). A traditional pipe is ”unnamed” and lasts only as long as the process. A named pipe, however, can last as long as the system is up, beyond the life of the process. It can be deleted if no longer used. Usually a named pipe appears as a file, and generally processes attach to it for IPC.
Instead of a conventional, unnamed, shell pipeline, a named pipeline makes use of the filesystem. It is explicitly created using mkfifo() or mknod(), and two separate processes can access the pipe by name — one process can open it as a reader, and the other as a writer.
Examples:
- Create a named pipe (also known as a FIFO), start one command writing to the named pipe in the background, then run the other command with the named pipe as input.
mkfifo /tmp/sort2.fifo
sort file2 > /tmp/sort2.fifo &
sort file1 | diff - /tmp/sort2.fifo
rm /tmp/sort2.fifo
- One can create a pipe and set up gzip to compress things piped to it:
mkfifo my_pipe
gzip -9 -c < my_pipe > out.gz &
# In a separate process shell, independently, one could send the data to
# be compressed:
cat file > my_pipe
#The named pipe can be deleted just like any file:
rm my_pipe
- A named pipe can be used to transfer information from one application to another without the use of an intermediate temporary file. For example, you can pipe the output of gzip into a named pipe like so (here out.gz is from above example but it can be any gz):
mkfifo -m 0666 /tmp/namedPipe
gzip -d < out.gz > /tmp/namedPipe
#Then load the uncompressed data into a MySQL table[3] like so:
LOAD DATA INFILE '/tmp/namedPipe' INTO TABLE tableName;
Without this named pipe one would need to write out the entire uncompressed version of file.gz before loading it into MySQL. Writing the temporary file is both time consuming and results in more I/O and less free space on the hard drive.
- Ejemplo de Daniel Clemente:
Con Linux podemos crear fifos, y es muy sencillo: haz mkfifo fi y habrás creado uno (un ls -l lo muestra como tal). Ahora haz un cat fi (se quedará parado) y en otra terminal un echo Hola >fi . Verás que el cat que estaba esperando datos ya los ha recibido; el fifo ha hecho su trabajo.
Puedes hacer cosas muy raras con fifos: por ejemplo, imagínate que dices a cdrecord que te grabe la ISO mififo (se quedaría esperando a que le entre contenido al fifo) y después haces un wget servidorveloz.com/linux.iso -O mififo (esto especifica el fichero de destino). ¡Estarías grabando un CD al mismo tiempo que lo descargas! Usa la imaginación para descubrir otros utilidades…
Printing on Unix-like systems goes way back to the beginning of the operating system. In those days, printers and how they were used were much different from today.
Like computers, printers in the pre-PC era tended to be large, expensive, and centralized. The typical computer user of 1980 worked at a terminal connected to a computer some distance away. The printer was located near the computer and was under the watchful eyes of the computer’s operators.
When printers were expensive and centralized, as they often were in the early days of Unix, it was common practice for many users to share a printer. To identify print jobs belonging to a particular user, a banner page displaying the name of the user was often printed at the beginning of each print job. The computer support staff would then load up a cart containing the day’s print jobs and deliver them to the individual users.
Character-Based Printers The printer technology of the 80s was very different from today in two respects. First, printers of that period were almost always impact printers. Impact printers use a mechanical mechanism that strikes a ribbon against the paper to form character impressions on the page. Two of the popular technologies of that time were daisy-wheel printing and dot-matrix printing.
The second, and more important characteristic of early printers was that printers used a fixed set of characters that were intrinsic to the device. For example, a daisy-wheel printer could only print the characters actually molded into the petals of the daisy wheel. This made the printers much like high-speed typewriters. As with most typewriters, they printed using monospaced (fixed width) fonts. This means that each character has the same width. Printing was done at fixed positions on the page, and the printable area of a page contained a fixed number of characters. Most printers printed ten characters per inch (CPI) horizontally and six lines per inch (LPI) vertically. Using this scheme, a US-letter sheet of paper is 85 characters wide and 66 lines high. Taking into account a small margin on each side, 80 characters was considered the maximum width of a print line. This explains why terminal displays (and our terminal emulators) are normally 80 characters wide. Using a monospaced font and and an 80 character wide terminal provides a What You See Is What You Get (WYSIWYG) view of printed output.
Data is sent to a typewriter-like printer in a simple stream of bytes containing the characters to be printed. For example, to print an “a”, the ASCII character code 97 is sent. In addition, the low-numbered ASCII control codes provided a means of moving the printer’s carriage and paper, using codes for carriage return, line feed, form feed, and so on. Using the control codes, it’s possible to achieve some limited font effects, such as boldface, by having the printer print a character, backspace, and print the character again to get a darker print impression on the page.
The development of GUIs led to major changes in printer technology. As computers moved to more picture-based displays, printing moved from character-based to graphical techniques. This was facilitated by the advent of the low-cost laser printer which, instead of printing fixed characters, could print tiny dots anywhere in the printable area of the page. This made printing proportional fonts (like those used by typesetters), and even photographs and high-quality diagrams, possible.
However, moving from a character-based scheme to a graphical scheme presented a formidable technical challenge. Here’s why: the number of bytes needed to fill a page using a character-based printer can be calculated this way (assuming 60 lines per page each containing 80 characters):
60 X 80 = 4,800 bytes
In comparison, a 300 dot per inch (DPI) laser printer (assuming an 8 by 10 inch print area per page) requires this many bytes:
(8 X 300) X (10 X 300) / 8 = 900,000 bytes
Many of the slow PC networks simply could not handle the nearly one megabyte of data required to print a full page on a laser printer, so it was clear that a clever invention was needed.
That invention turned out to be the page description language (PDL). A page description language is a programming language that describes the contents of a page. Basically it says, “Go to this position, draw the character ‘a’ in 10 point Helvetica, go to this posi-tion…” until everything on the page is described. The first major PDL was PostScript from Adobe Systems, which is still in wide use today. The PostScript language is a complete programming language tailored for typography and other kinds of graphics and imaging. It includes built-in support for 35 standard, high-quality fonts, plus the ability to accept additional font definitions at runtime. At first, support for PostScript was built into the printers themselves. This solved the data transmission problem. While the typical PostScript program was very verbose in comparison to the simple byte stream of character-based printers, it was much smaller than the number of bytes required to represent the entire printed page.
A PostScript printer accepted a PostScript program as input. The printer contained its own processor and memory (oftentimes making the printer a more powerful computer than the computer to which it was attached) and executed a special program called a PostScript interpreter, which read the incoming PostScript program and rendered the results into the printer’s internal memory, thus forming the pattern of bits (dots) that would be transferred to the paper. The generic name for this process of rendering something into a large bit pattern (called a bitmap) is raster image processor (RIP).
As the years went by, both computers and networks became much faster. This allowed the RIP to move from the printer to the host computer, which, in turn, permitted high-quality printers to be much less expensive.
Many printers today still accept character-based streams, but many low-cost printers do not. They rely on the host computer’s RIP to provide a stream of bits to print as dots. There are still some PostScript printers, too.
Modern Linux systems employ two software suites to perform and manage printing. The first, Common Unix Printing System (CUPS) provides print drivers and print-job management, and the second, Ghostscript, a PostScript interpreter, acts as a RIP.
CUPS manages printers by creating and maintaining print queues.
Is a program that runs commands, like the ones that users enter into a terminal window. These commands can be other programs or built-in (like exec()) features of the shell. The shell also serves as a small programming environment.
The simple glob character *, tells the shell to match any number of arbitrary characters.
The shell matches arguments containing globs to filenames, substitutes the filenames for those arguments, and then runs the revised command line. The substitution is called expansion because the shell substitutes all matching filenames for a simplified expression.
Another shell glob character, the question mark ?, instructs the shell to match exactly one arbitrary character.
If you don’t want the shell to expand a glob in a command, enclose the glob in single quotes (”). For example, the command echo '*' prints a star.
Note:
It is important to remember that the shell performs expansions before running commands, and only then. Therefore, if a * makes it to a command without expanding, the shell won’t do anything more with it; it’s up to the command to decide what it wants to do.
The shell can store temporary variables, called shell variables, containing the values of text strings. Shell variables are very useful for keeping track of values in scripts, and some shell variables control the way the shell behaves.
To assign a value to a shell variable, use the equal sign (=) and to access this variable, use $.
Is like a shell variable, but it’s not specific to the shell. All processes on Unix systems have environment variable storage. The main difference between environment and shell variables is that the operating system passes all of your shell’s environment variables to programs that the shell runs, whereas shell variables cannot be accessed in the commands that you run.
You assign an environment variable with the shell’s export command.
Interactive shells are those you use to run commands from a terminal, and they can be classified as login or non-login.
Note: noninteractive shells (such as those that run shell scripts) usually don’t read any startup files.
- Login Shells
Traditionally, a login shell is what you get when you first log in to a system with the terminal using a program such as /bin/login. Logging in remotely with SSH also gives you a login shell. The basic idea is that the login shell is an initial shell. You can tell if a shell is a login shell by running echo $0; if the first character is a -, the shell’s a login shell.
When bash runs as a login shell, it runs /etc/profile. Then it looks for a user’s .bash_profile, .bash_login, and .profile files, running only the first one that it sees.
As strange as it sounds, it’s possible to run a noninteractive shell as a login shell to force it to run startup files. To do so, start the shell with the -l or --login option.
- Non-Login Shells
A non-login shell is an additional shell that you run after you log in. It’s simply any interactive shell that’s not a login shell. Windowing system terminal programs (xterm, GNOME Terminal, and so on) start non-login shells unless you specifically ask for a login shell.
Upon starting up as a non-login shell, bash runs /etc/bash.bashrc and then runs the user’s .bashrc.
# COMMAND PATH
PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
PATH=$HOME/bin:$PATH # at the front so that executables there
# take precedence over the system versions.
# If you need the system executables, add /sbin and /usr/sbin .
# PS1 is the regular prompt.
# Substitutions include:
# \u username \h hostname \w current directory
# \! history number \s shell name \$ $ if regular user
# PROMPT
PS1='\u\$ '
# EDITOR and VISUAL determine the editor that programs such as less
# and mail clients invoke when asked to edit a file.
EDITOR=vi
VISUAL=vi
# PAGER is the default text file viewer for programs such as
# man (that shows text one page at a time).
PAGER=less
# These are some handy options for less.
# A different style is LESS=FRX
# (F=quit at end, R=show raw characters, X=don't use alt screen)
LESS=meiX
# You must export environment variables.
export PATH EDITOR VISUAL PAGER LESS
# By default, give other users read-only access to most new files.
# MASK
umask 022
# You can share this .bashrc file with .bash_profile via a symbolic
# link, or you can make the relationship even clearer by creating
# .bash_profile as this one-liner:
# . $HOME/.bashrc
The getty program attaches to terminals and displays a login prompt. On most Linux systems, getty is uncomplicated because the system uses it only for logins on virtual terminals. In a process listing, it usually looks something like this (for example, when running on /dev/tty1):
$ ps ao args | grep getty
#/sbin/agetty -o -p -- \u --noclear tty1 linux
On many systems, you may not even see a getty process until you access a virtual terminal with something like CTRL-ALT-F1. This example shows agetty, the version that many Linux distributions include by default.
After you enter your login name, getty replaces itself with the login program, which asks for your password. If you enter the correct password, login replaces itself (using exec()) with your shell. Otherwise, you get a “Login incorrect” message. Much of the login program’s real authentication work is handled by PAM.
You’ll rarely even use getty and login, because most users now log in either through a graphical interface such as gdm or remotely with SSH, neither of which uses getty or login.
A command can be one of four different things:
- an executable program
- like
/usr/bin. - a command built into the shell itself
bashsupports a number of commands internally called shell builtins. Thecdcommand, for example, is a shell builtin.- a shell function
- shell functions are miniature shell scripts incorporated into the environment.
- an alias
- aliases are commands that we can define ourselves, built from other commands.
Note:
When square brackets appear in the description of a command’s syntax, they indicate optional items. A vertical bar character indicates mutually exclusive items. In the case of the cd command above:
cd [-L|[-P[-e]]] [dir]
This notation says that the command cd may be followed optionally by either a -L or a -P and further, if the -P option is specified the -e option may also be included followed by the optional argument dir.
Is a number that a process uses together with the system libraries and kernel to identify and manipulate a file.
Most system programs write their diagnostic output as messages to the syslog service. The traditional syslogd daemon performs this service by waiting for messages and, upon receiving one, sending it to an appropriate channel, such as a file or a database. On most contemporary systems, journald (which comes with systemd) does most of the work.
The system logger is one of the most important parts of the system. When something goes wrong and you don’t know where to start, it’s always wise to check the log. If you have journald, you’ll do this with the journalctl command.
A log message typically contains important information such as the process name, process ID, and timestamp. There can also be two other fields: the facility (a general category) and severity (how urgent the message is).
At the bottom of any graphical display mechanism is the framebuffer, a chunk of memory that the graphics hardware reads and transmits to the screen for display. A few individual bytes in the framebuffer represent each pixel of the display, so the idea is that if you want to change the way something looks, you need to write new values to the framebuffer memory.
The approach that the X Window System takes is to have a server (called the X server) that acts as a sort of “kernel” of the desktop to manage everything from rendering windows to configuring displays to handling input from devices, such as keyboards and mice. The X server doesn’t dictate the way anything should act or appear. Instead, X client programs handle the user interface. Basic X client applications, such as terminal windows and web browsers, make connections to the X server and ask to draw windows. In response, the X server figures out where to place the windows and where to render client graphics, and it takes a certain amount of responsibility for rendering graphics to the framebuffer. The X server also channels input to a client when appropriate.
The name Wayland refers to a communications protocol between a compositing window manager and graphical client program. Unlike X, Wayland is significantly decentralized by design. There’s no large display server managing the framebuffer for a number of graphical clients, and there’s no centralized authority for rendering graphics. Instead, each client gets its own memory buffer (think of this as sort of a sub-framebuffer) for its own window, and a piece of software called a compositor combines all of the clients’ buffers into the necessary form for copying to the screen’s framebuffer.
One of the more unusual aspects of Wayland systems is the mechanism for drawing window decorations, such as title bars. In X, the window manager did it all, but in the initial implementations of Wayland, this was left up to the client applications, which sometimes led to windows having many different kinds of decorations on the same screen. Now there’s a part of the protocol called XDG-Decoration that allows the client to negotiate with the window manager to see if the window manager is willing to draw decorations.
A major difference between X and Wayland systems is in the window manager, the piece of software that determines how to arrange windows on the screen and is central to the user experience.
In X, the window manager is a client that acts as a helper to the server; it draws the windows’ decorations (such as title bars and close buttons), handles input events to those decorations, and tells the server where to move windows.
However, in Wayland, the window manager is the server, more or less. It is responsible for compositing all of the client window buffers into the display framebuffer, and it handles the channeling of input device events. As a result, it is required to do more work than a window manager in X, but much of that code can be common between window manager implementations.
Desktop applications include certain common elements, such as buttons and menus, called widgets. To speed up development and provide a common look, programmers use graphical toolkits to provide those elements. On operating systems like Windows or macOS, the vendor provides a common toolkit, and most programmers use that. On Linux, the GTK+ toolkit is one of the most common, but you’ll also frequently see widgets built on the Qt framework and others.
Toolkits usually consist of shared libraries and support files, such as images and theme information.
Although toolkits provide the user with a uniform outward appearance, some details of a desktop require a degree of cooperation between different applications. For example, one application may wish to share data with another or update a common notification bar on a desktop. To provide for those needs, toolkits and other libraries are bundled into larger packages called desktop environments. GNOME, KDE, and Xfce are some common Linux desktop environments.
Toolkits are at the core of most desktop environments, but to create a unified desktop, environments must also include numerous support files, such as icons and configurations, that make up themes. All of this is bound together with documents that describe design conventions, such as how application menus and titles should appear and how applications should react to certain system events.
Is an interprocess communication service used in many parts of the system. D-Bus is important because it serves as an interprocess communication mechanism that allows desktop applications to talk to each other, and because most Linux systems use it to notify processes of system events, such as inserting a USB drive.
D-Bus itself consists of a library that standardizes interprocess communication with a protocol and supporting functions for any two processes to talk to each other. By itself, this library doesn’t offer much more than a fancy version of normal IPC facilities, such as Unix domain sockets. What makes D-Bus useful is a central “hub” called dbus-daemon. Processes that need to react to events can connect to dbus-daemon and register to receive certain kinds of events.
Connecting processes also create the events. For example, the process udisks-daemon monitors udev for disk events and sends them to dbus-daemon, which then retransmits the events to applications interested in disk events.
The applications such as web browsers and the terminal window normally stand alone, but they often use interprocess communication to become aware of pertinent events. For example, an application can express interest when you attach a new storage device or when you receive new email or an instant message. This communication usually occurs over D-Bus.
Overseeing one or more virtual machines on a computer is a piece of software called a hypervisor or virtual machine monitor (VMM), which works similarly to how an operating system manages processes. There are two types of hypervisors, and the way you use a virtual machine depends on the type.
- the type 2 hypervisor
- is the most familiar, because it runs on a normal operating system such as Linux. For example, VirtualBox is a type 2 hypervisor, and you can run it on your system without extensive modifications.
- type 1 hypervisor
- is more like its own operating system (especially the kernel), built specifically to run virtual machines quickly and efficiently. This kind of hypervisor might occasionally employ a conventional companion system such as Linux to help with management tasks. Creating an instance of an operating system on a cloud service such as AWS is creating a virtual machine on a type 1 hypervisor.
In general, a virtual machine with its operating system is called a guest. The host is whatever runs the hypervisor. For type 2 hypervisors, the host is just your native system. For type 1 hypervisors, the host is the hypervisor itself, possibly combined with a specialized companion system.
Virtual machines are great for insulating an entire operating system and its set of running applications, but sometimes you need a lighter-weight alternative. Container technology is now a popular way to fulfill this need.
A container can be loosely defined as a restricted runtime environment for a set of processes, the implication being that those processes can’t touch anything on the system outside that environment. In general, this is called operating system–level virtualization.
It’s important to keep in mind that a machine running one or more containers still has only one underlying Linux kernel. However, the processes inside a container can use the user-space environment from a Linux distribution different than the underlying system.
The restrictions in containers are built with a number of kernel features. Some of the important aspects of processes running in a container are:
- They have their own cgroups.
- They have their own devices and filesystem.
- They cannot see or interact with any other processes on the system.
- They have their own network interfaces.
Pulling all of those things together is a complicated task. It’s possible to alter everything manually, but it can be challenging; just getting a handle on the cgroups for a process is tricky. To help you along, many tools can perform the necessary subtasks of creating and managing effective containers. Two of the most popular are Docker and LXC.
First you need to create an image, which comprises the filesystem and a few other defining features for a container to run with.
Note: It’s easy to confuse images and containers. You can think of an image as the container’s filesystem; processes don’t run in an image, but they do run in containers. This is not quite accurate (in particular, when you change the files in a Docker container, you aren’t making changes to the image), but it’s close enough for now.
Install Docker on your system (your distribution’s add-on package is probably fine), make a new directory somewhere, change to that directory, and create a file called Dockerfile containing these lines:
FROM alpine:latest
RUN apk add bash
CMD ["/bin/bash"]
This configuration uses the lightweight Alpine distribution. The only change we’re making is adding the bash shell, which we’re doing not just for an added measure of interactive usability but also to create a unique image and see how that procedure works. It’s possible (and common) to use public images and make no changes to them whatsoever. In that case, you don’t need a Dockerfile.
Note: Anything done with the RUN command in a Dockerfile happens during the image build, not afterward, when you start a container with the image. The CMD command is for the container runtime; this is why it occurs at the end.
Build the image with the following command, which reads the Dockerfile in the current directory and applies the identifier hlw_test to the image:
docker build -t hlw_test .
# Sending build context to Docker daemon 2.048kB
# Step 1/3 : FROM alpine:latest
# latest: Pulling from library/alpine
# cbdbe7a5bc2a: Pull complete
# Digest:
# sha256:9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4b9a54
# Status: Downloaded newer image for alpine:latest
# ---> f70734b6a266
In this step, Docker has created a new image with the identifier
f70734b6a266 for the basic Alpine distribution image (it’s not the
final image). An image that isn’t intended to be a final product is
called an intermediate image.
# Step 2/3 : RUN apk add bash
# ---> Running in 4f0fb4632b31
# fetch http://dl-
# cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
# fetch http://dl-
# cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
# (1/4) Installing ncurses-terminfo-base (6.1_p20200118-r4)
# (2/4) Installing ncurses-libs (6.1_p20200118-r4)
# (3/4) Installing readline (8.0.1-r0)
# (4/4) Installing bash (5.0.11-r1)
# Executing bash-5.0.11-r1.post-install
# Executing busybox-1.31.1-r9.trigger
# OK: 8 MiB in 18 packages
# Removing intermediate container 4f0fb4632b31
# ---> 12ef4043c80a
This part of our configuration is the bash shell package installation
in Alpine. After setting up the (temporary) container with ID
4f0fb4632b31 , Docker ran the apk command inside that container to
install bash, and then saved the resulting changes to the filesystem
into a new intermediate image with the ID 12ef4043c80a .
#Step 3/3 : CMD ["/bin/bash"]
#---> Running in fb082e6a0728
#Removing intermediate container fb082e6a0728
#---> 1b64f94e5a54
#Successfully built 1b64f94e5a54
#Successfully tagged hlw_test:latest
Finally, Docker makes the final changes required to run a bash shell
when starting a container from the new image.
In this example, you now have a final image with the ID 1b64f94e5a54, but because you tagged it (in two separate steps), you can also refer to it as hlw_test or hlw_test:latest. Run docker images to verify that your image and the Alpine image are present:
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hlw_test latest 1b64f94e5a54 1 minute ago 9.19MB
alpine latest f70734b6a266 3 weeks ago 5.61MB
Running Docker Containers Let’s jump right into it and start container with the image that you just built:
docker run -it hlw_test
# You should get a bash shell prompt where you can run commands in
# thecontainer. That shell will run as the root user. (-it options
# (interactive, connect a terminal))
# If you’re the curious type, you’ll probably want to take a look around
# the container. Run some commands, such as mount and ps , and explore
# the filesystem in general.
ps aux
#PID USER TIME COMMAND
# 1 root 0:00 /bin/bash
# 6 root 0:00 ps aux
initasks every process to shut down cleanly.- If a process doesn’t respond after a while, init kills it, first trying a
TERMsignal. - If the
TERMsignal doesn’t work, init uses theKILLsignal on any stragglers. - The system locks system files into place and makes other preparations for shutdown.
- The system unmounts all filesystems other than the root.
- The system remounts the root filesystem read-only.
- The system writes all buffered data out to the filesystem with the sync program.
- The final step is to tell the kernel to reboot or stop with the
reboot(2)system call. This can be done by init or an auxiliary program, such asreboot,halt, orpoweroff.
Is the core of the operating system. The kernel is software residing in memory that tells the CPU where to look for its next task. Acting as a mediator, the kernel manages the hardware (especially main memory) and is the primary interface between the hardware and any running program.
There is a critical difference between how the kernel and the user processes run: the kernel runs in kernel mode, and the user processes run in user mode. Code running in kernel mode has unrestricted access to the processor and main memory. The memory area that only the kernel can access is called kernel space.
The Linux kernel can run kernel threads, which look much like processes but have access to kernel space.
One of the kernel’s tasks is to split memory into many subdivisions, and it must maintain certain state information about those subdivisions at all times. Each process gets its own share of memory, and the kernel must ensure that each process keeps to its share.
- The machine’s BIOS or boot firmware loads and runs a boot loader.
- The boot loader finds the kernel image on disk, loads it into memory, and starts it.
- The kernel initializes the devices and its drivers.
- The kernel mounts the root filesystem.
- The kernel starts a program called
initwith a process ID of1. This point is the user space start. initsets the rest of the system processes in motion.- At some point,
initstarts a process allowing you to log in, usually at the end or near the end of the boot sequence.
GRUB stands for Grand Unified Boot Loader. One of GRUB’s most important capabilities is filesystem navigation that allows for easy kernel image and configuration selection.
- The PC BIOS or firmware initializes the hardware and searches its boot-order storage devices for boot code.
- Upon finding the boot code, the BIOS/firmware loads and executes it. This is where GRUB begins.
- The GRUB core loads.
- The core initializes. At this point, GRUB can now access disks and filesystems.
- GRUB identifies its boot partition and loads a configuration there.
- GRUB gives the user a chance to change the configuration.
- After a timeout or user action, GRUB executes the configuration.
- In the course of executing the configuration, GRUB may load additional code (modules) in the boot partition. Some of these modules may be preloaded.
- GRUB executes a boot command to load and execute the kernel as specified by the configuration’s linux command.
- Select from multiple kernels.
- Switch between sets of kernel parameters.
- Allow the user to manually override and edit kernel image names and parameters (for example, to enter single-user mode).
- Provide support for booting other operating systems.
- Processes
- The kernel is responsible for determining which processes are allowed to use the CPU.
- Memory
- The kernel needs to keep track of all memory—what is currently allocated to a particular process, what might be shared between processes, and what is free.
- Device drivers
- The kernel acts as an interface between hardware and processes. It’s usually the kernel’s job to operate the hardware.
- System calls and support Processes
- normally use system calls to communicate with the kernel.
Process management describes the starting, pausing, resuming, scheduling, and terminating of processes.
In practice, each process uses the CPU for a small fraction of a second, then pauses; then another process uses the CPU for another small fraction of a second; then another process takes a turn, and so on. The act of one process giving up control of the CPU to another process is called a context switch.
The context switch answers the important question of when the kernel runs. The answer is that it runs between process time slices during a context switch.
To understand how the context switch works, let’s think about a situation in which a process is running in user mode but its time slice is up.
- The CPU interrupts the current process based on an internal timer, switches into kernel mode, and hands control back to the kernel.
- The kernel records the current state of the CPU and memory, which will be essential to resuming the process that was just interrupted.
- The kernel performs any tasks that might have come up during the preceding time slice (such as collecting data from input and output, or I/O, operations).
- The kernel is now ready to let another process run. The kernel analyzes the list of processes that are ready to run and chooses one.
- The kernel prepares the memory for this new process and then prepares the CPU.
- The kernel tells the CPU how long the time slice for the new process will last.
- The kernel switches the CPU into user mode and hands control of the CPU to the process.
The kernel must manage memory during a context switch, which can be a complex job. The following conditions must hold:
- The kernel must have its own private area in memory that user processes can’t access.
- Each user process needs its own section of memory.
- One user process may not access the private memory of another process. User processes can share memory.
- Some memory in user processes can be read-only.
When the process accesses some of its memory, the memory management unit MMU intercepts the access and uses a memory address map to translate the memory location from the process point of view into an actual physical memory location in the machine. The kernel must still initialize and continuously maintain and alter this memory address map. The implementation of a memory address map is called a page table.
Perform specific tasks, an interaction between a process and the kernel.
Two system calls, fork() and exec(), are important to understanding how processes start:
- fork()
- when a process calls
fork(), the kernel creates a nearly identical copy of the process. - exec()
- when a process calls
exec(program), the kernel loads and startsprogram, replacing the current process.
Other than init, all new user processes on a Linux system start as a result of fork(), and most of the time, you also run exec() to start a new program instead of running a copy of an existing process.
When you enter ls into a terminal window, the shell that’s running inside the terminal window calls fork() to create a copy of the shell, and then the new copy of the shell calls exec(ls) to run ls.
+-------+ +--------+ +-------+
| shell |--->| fork() |--->| shell |
+-------+ +--------+ | +-------+
|
| +------------+ +----------+ +----+
-->| copy shell |--->| exec(ls) |--->| ls |
+------------+ +----------+ +----+
The kernel also supports user processes with features other than traditional system calls, the most common of which are pseudodevices. Pseudodevices look like devices to user processes, but they’re implemented purely in software. This means they don’t technically need to be in the kernel, but they are usually there for practical reasons. For example, the kernel random number generator device /dev/random would be difficult to implement securely with a user process.
Technically, a user process that accesses a pseudodevice must use a system call to open the device, so processes can’t entirely avoid system calls.
The CPU has a memory management unit (MMU) to add flexibility in memory access. The kernel assists the MMU by breaking down the memory used by processes into smaller chunks called pages. The kernel maintains a data structure, called a page table, that maps a process’s virtual page addresses to real page addresses in memory. As a process accesses memory, the MMU translates the virtual addresses used by the process into real addresses based on the kernel’s page table.
A user process doesn’t actually need all of its memory pages to be immediately available in order to run. The kernel generally loads and allocates pages as a process needs them; this system is known as on-demand paging or just demand paging. To see how this works, consider how a program starts and runs as a new process:
- The kernel loads the beginning of the program’s instruction code into memory pages.
- The kernel may allocate some working-memory pages to the new process.
- As the process runs, it might reach a point where the next instruction in its code isn’t in any of the pages that the kernel initially loaded. At this point, the kernel takes over, loads the necessary page into memory, and then lets the program resume execution.
- Similarly, if the program requires more working memory than was initially allocated, the kernel handles it by finding free pages (or by making room) and assigning them to the process.
Is a computer program that runs as a background process, rather than being under the direct control of an interactive user.
In a Unix environment, the parent process of a daemon is often, but not always, the init process. A daemon is usually created either by a process forking a child process and then immediately exiting, thus causing init to adopt the child process, or by the init process directly launching the daemon.
Systems often start daemons at boot time that will respond to network requests, hardware activity, or other programs by performing some task. Daemons such as cron may also perform defined tasks at scheduled times.
Is a software suite that provides an array of system components for Linux operating systems.
The systemd init is one of the newest init implementations on Linux. In addition to handling the regular boot process, systemd aims to incorporate the functionality of a number of standard Unix services, such as cron and inetd.
Where systemd really stands out from its predecessors is its advanced service management capabilities. Unlike a traditional init, systemd can track individual service daemons after they start, and group together multiple processes associated with a service, giving you more power and insight into exactly what is running on the system.
systemd is goal-oriented. At the top level, you can think of defining a goal, called a unit, for some system task. A unit can contain instructions for common startup tasks, such as starting a daemon, and it also has dependencies, which are other units. When starting (or activating) a unit, systemd attempts to activate its dependencies and then moves on to the details of the unit.
When starting services, systemd does not follow a rigid sequence; instead, it activates units whenever they are ready. After boot, systemd can react to system events (such as the uevents outlined) by activating additional units.
Requests to activate, reactivate, and restart units are called jobs in systemd, and they are essentially unit state changes. These jobs also have nothing to do with the shell’s job control.
Following its integrated approach, systemd also provides replacements for various daemons and utilities, including the startup shell scripts, pm-utils, inetd, acpid, syslog, watchdog, cron and atd. systemd’s core components include the following:
- systemd
- is a system and service manager for Linux operating systems.
- systemctl
- is a command to introspect and control the state of the systemd system and service manager. Not to be confused with sysctl.
- systemd-analyze
- may be used to determine system boot-up performance statistics and retrieve other state and tracing information from the system and service manager.
On the system shown below, default.target groups the units necessary to start a GUI.
+----------------+
| default.target |
+----------------+
|
v
+-------------------+
| multi-user.target |
+-------------------+
/ | \
v v v
+--------------+ +--------------+ +--------------+
| basic.target | | cron.service | | dbus.service |
+--------------+ +--------------+ +--------------+
|
v
+----------------+
| sysinit.target |
+----------------+
One way that systemd is more ambitious than previous versions of init is that it doesn’t just operate processes and services; it can also manage filesystem mounts, monitor network connection requests, run timers, and more. Each capability is called a unit type, and each specific function (such as a service) is called a unit. When you turn on a unit, you activate it. Each unit has its own configuration file.
These are the most significant unit types that perform the boot-time tasks on a typical Linux system:
- service units
- control the service daemons found on a Unix system.
- target units
- control other units, usually by grouping them.
- socket units
- represent incoming network connection request locations.
- mount units
- represent the attachment of filesystems to the system.
Ordering: To activate units in a particular order, use the following dependency modifiers:
- Before
- the current unit will activate before the listed unit(s). For example, if
Before=bar.targetappears infoo.target, systemd activatesfoo.targetbeforebar.target. - After
- the current unit activates after the listed unit(s).
When you use ordering, systemd waits until a unit has an active status before activating its dependent units.
To accommodate the need for flexibility and fault tolerance, systemd offers several dependency types and styles.
- Requires
- strict dependencies. When activating a unit with a
Requiresdependency unit, systemd attempts to activate the dependency unit. If the dependency unit fails, systemd also deactivates the dependent unit. - Wants
- dependencies for activation only. Upon activating a unit, systemd activates the unit’s
Wantsdependencies, but it doesn’t care if those dependencies fail. - Requisite
- units that must already be active. Before activating a unit with a
Requisitedependency, systemd first checks the status of the dependency. If the dependency hasn’t been activated, systemd fails on activation of the unit with the dependency. - Conflicts
- negative dependencies. When activating a unit with a
Conflictdependency, systemd automatically deactivates the opposing dependency if it’s active. Simultaneous activation of conflicting units fails.
The Wants dependency type is especially significant because it doesn’t propagate failures to other units. The systemd.service(5) manual page states that this is how you should specify dependencies if possible, and it’s easy to see why. This behavior produces a much more robust system, giving you the benefit of a traditional init, where the failure of an earlier startup component doesn’t necessarily prohibit later components from starting.
The format for unit files is a section names in square brackets [] and variable and value assignments (options) in each section.
There are two sections, [Unit] and [Service]. The [Unit] section gives some details about the unit and contains description and dependency information. In the most service unit, you’ll find the details about the service in the [Service] section, including how to prepare, start, and reload the service.
# bus-daemon.service unit file for the desktop bus daemon.
[Unit]
Description=D-Bus System Message Bus
Documentation=man:dbus-daemon(1)
Requires=dbus.socket # requires the dbus.socket unit as a dependency
RefuseManualStart=yes
[Service]
ExecStart=/usr/bin/dbus-daemon --system --address=systemd: --
nofork --nopidfile --systemd-activation --syslog-only
ExecReload=/usr/bin/dbus-send --print-reply --system --
type=method_call --dest= org.freedesktop.DBus /
org.freedesktop.DBus.ReloadConfig
Adding units to systemd is primarily a matter of creating, then activating and possibly enabling, unit files. You should normally put your own unit files in the system configuration directory /etc/systemd/system so that you won’t confuse them with anything that came with your distribution and so that the distribution won’t overwrite them when you upgrade.
As an example of create two targets, one with a dependency on the other, follow these steps:
- Create a unit file named
test1.targetin/etc/systemd/system:
[Unit]
Description=test 1
- Create a
test2.targetfile with a dependency ontest1.target:
[Unit]
Description=test 2
Wants=test1.target # Wants keyword defines a dependency
- Activate the
test2.targetunit to see it in action and verify that both units are active:
systemctl start test2.target
systemctl status test1.target test2.target
- If your unit file has an
[Install]section, you need to “enable” the unit before activating it (the [Install] section is another way to create a dependency).
systemctl enable unit
Removing Units from systemd:
- Deactivate the unit if necessary:
systemctl stop test1.target
- If the unit has an
[Install]section, disable the unit to remove any symbolic links created by the dependency system. Disabling a unit that is implicitly enabled (that is, does not have an [Install] section) has no effect.
systemctl disable test1.target
You can then remove the unit file if you like.
Apart from defining dependencies in a dependent unit’s configuration file. It’s also possible to do this “in reverse”—that is, by specifying the dependent unit in a dependency’s unit file. You can do this by adding a WantedBy or RequiredBy parameter in the [Install] section. This mechanism allows you to alter when a unit should start without modifying additional configuration files (for example, when you’d rather not edit a system unit file).
To see how this works, consider the example units. We had two units, test1.target and test2.target, with test2.target having a Wants dependency on test1.target. We can change them so that test1.target looks like this:
[Unit] Description=test 1
[Install] WantedBy=test2.target
And test2.target is as follows:
[Unit] Description=test 2
Because you now have a unit with an [Install] section, you need to enable the unit with systemctl before you can start it. Here’s how that works with test1.target:
systemctl enable test1.target
# Created symlink
# /etc/systemd/system/test2.target.wants/test1.target →
# /etc/systemd/system/test1.target.
Notice the output here—the effect of enabling a unit is to create a symbolic link in a .wants subdirectory corresponding to the dependent unit (test2.target in this case). You can now start both units at the same time with systemctl start test2.target because the dependency is in place.
Note: Enabling a unit does not activate it.
To disable the unit (and remove the symbolic link), use systemctl as follows:
systemctl disable test1.target
# Removed /etc/systemd/system/test2.target.wants/test1.target.
The [Install] section is usually responsible for the .wants and .requires directories in the system configuration directory /etc/systemd/system. However, the unit configuration directory [/usr]/lib/systemd/system also contains .wants directories, and you may also add links that don’t correspond to [Install] sections in the unit files. These manual additions are a simple way to add a dependency without modifying a unit file that may be overwritten in the future (by a software upgrade, for instance), but they’re not particularly encouraged because a manual addition is difficult to trace.
A simple network echo service. The idea of an echo service is to repeat anything that a network client sends after connecting; ours will listen on TCP port 22222. We’ll start building it with a socket unit to represent the port, as shown in the following echo.socket unit file:
[Unit]
Description=echo socket
[Socket]
ListenStream=22222
Accept=true
Note that there’s no mention of the service unit that this socket supports inside the unit file. So, what is that corresponding service unit file?
Its name is [email protected]. The link is established by naming convention; if a service unit file has the same prefix as a .socket file (in this case, echo), systemd knows to activate that service unit when there’s activity on the socket unit. In this case, systemd creates an instance of [email protected] when there’s activity on echo.socket. Here’s the [email protected] unit file:
[Unit]
Description=echo service
[Service]
ExecStart=/bin/cat
StandardInput=socket
Note: If you don’t like the implicit activation of units based on the prefixes, or you need to link units with different prefixes, you can use an explicit option in the unit defining your resource. For example, use Socket=bar.socket inside foo.service to have bar.socket hand its socket to foo.service.
To get this example unit running, you need to start the echo.socket unit:
systemctl start echo.socket
Now you can test the service by connecting to your local TCP port 22222 with a utility such as telnet. The service repeats what you enter; here’s an example interaction:
$ telnet localhost 22222
# Trying 127.0.0.1...
# Connected to localhost.
# Escape character is '^]'.
Hi there.
# Hi there.
To stop the service, stop the socket unit like so:
systemctl stop echo.socket
Because the [email protected] unit supports multiple simultaneous instances, there’s an @ in the name (the @ specifier signifies parameterization). Why would you want multiple instances? Say you have more than one network client connecting to the service at the same time, and you want each connection to have its own instance. In this case, the service unit must support multiple instances because we included the Accept=true option in echo.socket. That option instructs systemd not only to listen on the port, but also to accept incoming connections on behalf of the service unit and pass it to them, creating a separate instance for each connection. Each instance reads data from the connection as standard input, but it doesn’t necessarily need to know that the data is coming from a network connection.
Note: Most network connections require more flexibility than just a simple gateway to standard input and output, so don’t expect to be able to create complex network services with a service unit file like the [email protected] unit file shown here.
If a service unit can do the work of accepting a connection, don’t put an @ in its unit filename, and don’t put Accept=true in the socket unit. In this case, the service unit takes complete control of the socket from systemd, which in turn does not attempt to listen on the network port again until the service unit finishes.
An alternative to creating a cron job for a periodic task is to build a systemd timer unit. For an entirely new task, you must create two units: a timer unit and a service unit. The reason for two units is that a timer unit doesn’t contain any specifics about the task to perform; it’s just an activation mechanism to run a service unit.
Let’s look at a typical timer/service unit pair, starting with the timer unit. Let’s call this loggertest.timer; as with other custom unit files, we’ll put it in /etc/systemd/system.
[Unit]
Description=Example timer unit
[Timer]
OnCalendar=*-*-* *:00,20,40
Unit=loggertest.service
[Install]
WantedBy=timers.target
This timer runs every 20 minutes, with the OnCalendar option resembling the cron syntax. In this example, it’s at the top of each hour, as well as 20 and 40 minutes past each hour.
The OnCalendar time format is year-month-day hour:minute:second. The field for seconds is optional. As with cron, a * represents a sort of wildcard, and commas allow for multiple values. The periodic / syntax is also valid; in the preceding example, you could change the *:00,20,40 to *:00/20 (every 20 minutes) for the same effect.
The associated service unit is named loggertest.service. We explicitly named it in the timer with the Unit option, but this isn’t strictly necessary because systemd looks for a .service file with the same base name as the timer unit file. This service unit also goes in /etc/systemd/system.
[Unit]
Description=Example Test Service
[Service]
Type=oneshot
ExecStart=/usr/bin/logger -p local3.debug I\'m a logger
The meat of this is the ExecStart line, which is the command that the service runs when activated. This particular example sends a message to the system log.
Note the use of oneshot as the service type, indicating that the service is expected to run and exit, and that systemd won’t consider the service started until the command specified by ExecStart completes. This has a few advantages for timers:
- You can specify multiple
ExecStartcommands in the unit file. - It’s easier to control strict dependency order when activating other units using
WantsandBeforedependency directives. - You have better records of start and end times of the unit in the journal.
+-----------------------------------------------------------------+
| +-----------------------------------------------------------+ |
| | Partition Table | |
| +-----------------------------------------------------------+ |
| | | |
| v v |
| +--------------------------------------------+ +-----------+ |
| | Partition | | Partition | |
| | +--------------------------------------+ | | | |
| | | +----------------------------+ | | | | |
| | | | Filesystem Data Structures | | | | | |
| | | +----------------------------+ | | | | |
| | | | | | | | |
| | | v | | | | |
| | | +--------------------------------+ | | | | |
| | | | File Data | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | +--------------------------------+ | | | | |
| | +--------------------------------------+ | | | |
| +--------------------------------------------+ +-----------+ |
+-----------------------------------------------------------------+
A device is typically accessible only in kernel mode because improper access could crash the machine. A notable difficulty is that different devices rarely have the same programming interface, even if the devices perform the same task.
- block device
- programs access data from a block device in fixed chunks. Disks can be easily split up into blocks of data. Because a block device’s total size is fixed and easy to index, processes have quick random access to any block in the device with the help of the kernel.
- character device
- character devices work with data streams. You can only read characters from or write characters to character devices. Character devices don’t have a size; when you read from or write to one, the kernel usually performs a read or write operation on it. Printers directly attached to your computer are represented by character devices.
- pipe device
- named pipes are like character devices, with another process at the other end of the I/O stream instead of a kernel driver.
- socket device
- are special-purpose interfaces that are frequently used for interprocess communication.
Small Computer System Interface is a set of standards for physically connecting and transferring data between computers and peripheral devices. The SCSI standards define commands, protocols, electrical, optical and logical interfaces.
The MBR table in example below contains primary, extended, and logical partitions. A primary partition is a normal subdivision of the disk; partition 1 is an example. The basic MBR has a limit of four primary partitions, so if you want more than four, you must designate one as an extended partition. An extended partition breaks down into logical partitions, which the operating system can then use as it would any other partition. In this example, partition 2 is an extended partition that contains logical partition 5.
parted -l
Model: ATA KINGSTON SM2280S (scsi)
Disk /dev/sda: 240GB
Sector size (logical/physical): 512B/512B Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 223GB 223GB primary ext4 boot
2 223GB 240GB 17.0GB extended
5 223GB 240GB 17.0GB logical linux-swap(v1)
Model: Generic Flash Disk (scsi)
Disk /dev/sdf: 4284MB
Sector size (logical/physical): 512B/512B Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 1050MB 1049MB myfirst
2 1050MB 4284MB 3235MB mysecond
If you run out of real memory, the Linux virtual memory system can automatically move pieces of memory to and from disk storage. This is called swapping because pieces of idle programs are swapped to the disk in exchange for active pieces residing on the disk. The disk area used to store memory pages is called swap space (or just swap).
To use an entire disk partition as swap, follow these steps:
- Make sure the partition is empty.
- Run
mkswap dev, wheredevis the partition’s device. This command puts a swap signature on the partition, marking it as swap space (rather than a filesystem or otherwise). - Execute
swapon devto register the space with the kernel.
After creating a swap partition, you can put a new swap entry in your /etc/fstab file to make the system use the swap space as soon as the machine boots. Here’s a sample entry that uses /dev/sda5 as a swap partition:
/dev/sda5 none swap sw 0 0
You can use a regular file as swap space if you’re in a situation where you would be forced to repartition a disk in order to create a swap partition. You shouldn’t notice any problems when doing this.
Use these commands to create an empty file, initialize it as swap, and add it to the swap pool:
dd if=/dev/zero of=swap_file bs=1024k count=num_mb
mkswap swap_file
swapon swap_file
Here, swap_file is the name of the new swap file, and num_mb is the desired size in megabytes.
To remove a swap partition or file from the kernel’s active pool, use the swapoff command. Your system must have enough free remaining memory (real and swap combined) to accommodate any active pages in the part of the swap pool that you’re removing.
Example:
## Check current swap status.
sudo swapon -s
## Create the swap file (1024k*7168=7340032->7GB)
sudo dd if=/dev/zero of=/swapfile bs=1024k count=7168
## Set Swap Permissions.
sudo chmod 600 /swapfile
## Set up the file as Linux swap area.
sudo mkswap /swapfile
## Enable the swap.
sudo swapon /swapfile
## Make the swap file permanent adding the following
## line at the end of the /etc/fstab file.
/swapfile none swap sw 0 0
## To verify that the swap is active.
# sudo swapon --show
## Remove Swap Space
# swapoff /swapfile
## Remove the swap file entry /swapfile none swap sw 0 0
## from the /etc/fstab file.
## Remove the swap file
# sudo rm /swapfile
Char devices are accessed through names in the filesystem. Those names are called special files or device files or simply nodes of the filesystem tree; they are conventionally located in the /dev directory. Special files for char drivers are identified by a c in the first column of the output of ls -l. Block devices appear in /dev as well, but they are identified by a b.
Each device node’s type (block or character) and numbers (known as the major and minor number) serve as identifiers for the kernel. You can view the current devices in the /proc/devices file.
- major number
- identifies the driver associated with the device. For example,
/dev/nulland/dev/zeroare both managed by driver1, whereas virtual consoles and serial terminals are managed by driver4. The kernel uses the major number at open time to dispatch execution to the appropriate driver.
- minor number
- is used only by the driver specified by the major number; other parts of the kernel don’t use it, and merely pass it along to the driver. It is common for a driver to control several devices; the minor number provides a way for the driver to differentiate among them (typically refers to an instance).
ls -l /dev/vd*
# brw-rw---- 1 root disk 253, 0 Feb 3 09:09 /dev/vda
# brw-rw---- 1 root disk 253, 1 Feb 3 09:09 /dev/vda1
# brw-rw---- 1 root disk 253, 2 Feb 3 09:09 /dev/vda2
# brw-rw---- 1 root disk 253, 3 Feb 3 09:09 /dev/vda3
# brw-rw---- 1 root disk 253, 16 Feb 3 09:09 /dev/vdb
# brw-rw---- 1 root disk 253, 32 Feb 3 09:09 /dev/vdc
# brw-rw---- 1 root disk 253, 33 Feb 3 09:09 /dev/vdc1
# In the first line 253 is the major number virblk identifier and the
# minor number is 0 which differentiates it from the others.
- You’ll start at the command prompt with the device name (first unmount it, if needed).
sudo fdisk /dev/sdd mwill display the program menu.- First, print the current table with the
poption.
Disk /dev/sdd: 4 GiB, 4284481536 bytes, 8368128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos
Disk identifier: 0x88f290cc
Device Boot Start End Sectors Size Id Type
/dev/sdd1 2048 8368127 8366080 4G c W95 FAT32 (LBA)
You can delete the existing ones pressing d.
Remember that fdisk doesn’t make changes until you explicitly write the partition table, so you haven’t yet modified the disk. If you make a mistake you can’t recover from, use the q option to quit fdisk without writing the changes.
- Now you’ll create the first 200MB partition with the
noption (the default values are quite often what you want). - When you’re finished laying out the partitions, use the
p(print) option to review.
Device Boot Start End Sectors Size Id Type
/dev/sdd1 2048 411647 409600 200M 83 Linux
/dev/sdd2 411648 8368127 7956480 3.8G 83 Linux
- When you’re ready to write the partition table, use the
woption.
Additionally,
- You can create a filesystem
ext4partition on/dev/sdd1with this command.sudo mkfs -t ext4 /dev/sdd1 - To mount the Fourth Extended filesystem found on the device
/dev/sdd1on/home/extra, use this command.sudo mount -t ext4 /dev/sdd1 /home/extra
Is a form of database; it supplies the structure to transform a simple block device into the sophisticated hierarchy of files and subdirectories that users can understand.
The Linux boot process is, for the most part, fairly straightforward. However, one component has always been somewhat confounding: initramfs, or the initial RAM filesystem. Think of it as a little user-space wedge that goes in front of the normal user mode start.
The Linux kernel does not talk to the PC BIOS interface or EFI to get data from disks, so in order to mount its root filesystem, it needs driver support for the underlying storage mechanism. Unfortunately, there are so many storage controller drivers that distributions can’t include all of them in their kernels, so many drivers are shipped as loadable modules. But loadable modules are files, and if your kernel doesn’t have a filesystem mounted in the first place, it can’t load the driver modules that it needs.
The workaround is to gather a small collection of kernel driver modules along with a few other utilities into an archive. The boot loader loads this archive into memory before running the kernel. Upon start, the kernel reads the contents of the archive into a temporary RAM filesystem (the initramfs), mounts it at /, and performs the user-mode handoff to the init on the initramfs. Then, the utilities included in the initramfs allow the kernel to load the necessary driver modules for the real root filesystem. Finally, the utilities mount the real root filesystem and start the true init.
A symbolic link is a file that points to another file or a directory, effectively creating an alias (shortcut).
Your system may also have links that point to other links, which are called chained symbolic links and can be a nuisance when you’re trying to track them down.
ln creates a hard link, giving an additional real filename to a single file. The new filename has the status of the old one; it points (links) directly to the file data instead of to another filename as a symbolic link does.
By default, every file has a single hard link that gives the file its name. When we create a hard link, we create an additional directory entry for a file. Hard links have two important limitations:
- A hard link cannot reference a file outside its own file system. This means a link cannot reference a file that is not on the same disk partition as the link itself.
- A hard link may not reference a directory.
A hard link is indistinguishable from the file itself. Unlike a symbolic link, when we list a directory containing a hard link we will see no special indication of the link. When a hard link is deleted, the link is removed but the contents of the file itself continue to exist (that is, its space is not deallocated) until all links to the file are deleted.
Not all filesystems represent storage on physical media. Most versions of Unix have filesystems that serve as system interfaces. That is, rather than serving only as a means to store data on a device, a filesystem can represent system information, such as process IDs and kernel diagnostics. This idea goes back to the /dev mechanism, which is an early model of using files for I/O interfaces.
Some of the special filesystem types in common use on Linux include:
- proc
- mounted on /proc.
- sysfs
- mounted on /sys.
- tmpfs
- mounted on /run and other locations. With
tmpfs, you can use your physical memory and swap space as temporary storage. You can mounttmpfswhere you like, using the size andnr_blockslong options to control the maximum size. However, be careful not to pour things constantly into atmpfslocation, because your system will eventually run out of memory and programs will start to crash. - squashfs
- a type of read-only filesystem where content is stored in a compressed format and extracted on demand through a loopback device. One example use is in the snap package management system that mounts packages under the
/snapdirectory. - overlay
- a filesystem that merges directories into a composite. Containers often use overlay filesystems.
A traditional Unix filesystem has two primary components: a pool of data blocks where you can store data and a database system that manages the data pool. The database is centered around the inode data structure.
An inode is a set of data that describes a particular file, including its type, permissions, and—perhaps most important—where in the data pool the file data resides. Inodes are identified by numbers listed in an inode table.
Filenames and directories are also implemented as inodes. A directory inode contains a list of filenames and links corresponding to other inodes.
User-level representation of a filesystem:
+------+
|(root)|
+------+
_____/\____
_____/ \_____
/ \
+----+ +----+
|dir1| |dir2|
+----+ +----+
/|\ /\
/ | \ / \
/ | \ / \
+-----+ +-----+ +-----+ +-----+ +-----+
|file1| |file2| |file3| |file4| |file5|
+-----+ +-----+ +-----+ +-----+ +-----+
Note: For any ext2/3/4 filesystem, you start at inode number 2, which is the root inode (try not to confuse this with the system root filesystem).
The link count field is the number of total directory entries (across all directories) that point to an inode.
There is one small exception in link counts. The root inode 2 has a link count of 4. However, Figure shows only three directory entry links. The “fourth” link is in the filesystem’s superblock because the superblock tells you where to find the root inode.
Inode structure of the filesystem shown in previous filesystem:
inode table
# link count type data pool
+------+---+------+ +-------------------+
| 2 | 4 | dir |------- | . inode 2 |
+------+---+------+ \------->| dir1 inode 12 |
+------+---+------+ | dir2 inode 7633 |
| 12 | 2 | dir |--- +-------------------+
+------+---+------+ \--- +-------------------+
+------+---+------+ \--- | . inode 12 |
| 13 | 1 | file |-- \--->| .. inode 2 |
+------+---+------+ \ | file1 inode 13 |
+------+---+------+ \ | file2 inode 14 |
| 14 | 1 | file |- \ | file3 inode 15 |
+------+---+------+ \ \ +-------------------+
+------+---+------+ \ \ +-------------------+
| 15 | 2 | file |- \ \-->| "a" |
+------+---+------+ \ \ +-------------------+
+------+---+------+ \ \ +-------------------+
| 16 | 1 | file |- \ \ | . inode 7633 |
+------+---+------+ \ \ \ | .. inode 2 |
+------+---+------+ \ \ \ | file4 inode 16 |
| 7633 | 2 | dir |--------------->| file5 inode 15 |
+------+---+------+ \ \ \ +-------------------+
\ \ \ +-------------------+
\ \ \-->| "b" |
\ \ +-------------------+
\ \ +-------------------+
\ \-->| "c" |
\ +-------------------+
\ +-------------------+
\-->| "d" |
+-------------------+
Note: Portable Filenames
To ensure that a filename is portable between multiple platforms (i.e., different types of computers and operating systems), care must be taken to limit which characters are included in a filename. There is a standard called the POSIX Portable Filename Character Set that can be used to maximize the chances that a filename will work across different systems. The standard is pretty simple. The only characters allowed are the uppercase letters A-Z, the lowercase letters a-z, the numerals 0-9, period ( . ), hyphen ( - ), and underscore ( _ ). The standard further suggests that filenames not begin with a hyphen.
A shell’s built-in umask (permissions mask) facility sets the default permissions. Include the umask command in one of your startup files to make certain that any program you run creates files with your desired permissions. There are two reasonable choices:
077This mask is the most restrictive permissions mask; it doesn’t give any other users access to new files and directories. This is often appropriate on a multi-user system where you don’t want other users to look at any of your files. However, when set as the default, this mask can sometimes lead to problems when your users want to share files but don’t understand how to set permissions correctly.022This mask gives other users read access to new files and directories. This can be a good choice on a single-user system because many daemons that run as pseudo-users won’t be able to see files and directories created with the more restrictive077umask.
Everywhere a 1 appears in the binary value of the mask, an attribute is unset.
| Original file | — rw- rw- rw- | 0666 |
| Mask | 000 000 010 010 | 0022 |
|---|---|---|
| Result | — rw- r– r– | 0644 |
Meaning that when you execute the program, it runs as though the file owner is the user instead of you. Many programs use this setuid bit to run as root in order to get the privileges they need to change system files. One example is the passwd program, which needs to change the /etc/passwd file.
Like the setuid bit, changes the effective group ID from the real group ID of the real user to that of the file owner. If the setgid bit is set on a directory, newly created files in the directory will be given the group ownership of the directory rather the group ownership of the file’s creator. This is useful in a shared directory when members of a common group need access to all the files in the directory, regardless of the file owner’s primary group.
This is a holdover from ancient Unix, where it was possible to mark an executable file as “not swappable.” On files, Linux ignores the sticky bit, but if applied to a directory, it prevents users from deleting or renaming files unless the user is either the owner of the directory, the owner of the file, or the superuser. This is often used to control access to a shared directory, such as /tmp.
# Viewing a List of Mounted File Systems
mount
# /dev/sda2 on / type ext4 (rw)
# device /dev/sda2 is mounted as the root file system, is of type ext4,
# and is both readable and writable
# Unmount the disc with the umount command
umount /dev/sdc
# Create a new mount point for the disk.
# First create a new directory.
mkdir /mnt/cdrom
# Mount the CD-ROM at the new mount point.
mount -t iso9660 /dev/sdc /mnt/cdrom
# Mount the flash drive (other example).
sudo mkdir /mnt/flash
sudo mount /dev/sdb1 /mnt/flash
To reformat the flash drive with a Linux native file system, rather than the FAT32 system for example. This involves two steps.
- (optional) Create a new partition layout if the existing one is not to our liking.
- Create a new, empty file system on the drive.
- First unmount it (if needed) and then invoke the
fdiskprogram:
sudo umount /dev/sdb1
sudo fdisk /dev/sdb
- Entering an
mwill display the program menu. - The first thing we want to do is examine the existing partition layout. We do this by entering
pto print the partition table for the device.
Command (m for help): p
Disk /dev/sdb: 16 MB, 16006656 bytes
1 heads, 31 sectors/track, 1008 cylinders
Units = cylinders of 31 * 512 = 15872 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 2 1008 15608+ b W95 FAT32
# We see a 16 MB device with a single partition (1) that uses 1,006 of
# the available 1,008 cylinders on the device. The partition is
# identified as a Windows 95 FAT32 partition.
We see that the ID b is used to specify the existing partition. To see a list of the available partition types, we refer to the program menu.
If we enter l at the prompt, a large list of possible types is displayed. Among them we see b for our existing partition type and 83 for Linux.
- We enter
t(change a partition’s system id) at the prompt and enter the new ID.
Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 83
Changed system type of partition 1 to 83 (Linux)
- This completes all the changes we need to make. Up to this point, the device has been untouched (all the changes have been stored in memory, not on the physical device), so we will write the modified partition table to the device and exit. To do this, we enter
wat the prompt.
Additionally, to create an ext4 file system on the device, we use the -t option to specify the ext4 system type, followed by the name of the device containing the partition we want to format.
sudo mkfs -t ext4 /dev/sdb1
To reformat the device to its original FAT32 file system, specify vfat as the file system
type.
sudo mkfs -t vfat /dev/sdb1
An access-control list (ACL) is a list of permissions associated with a system resource (object or facility). An ACL specifies which users or system processes are granted access to resources, as well as what operations are allowed on given resourcess. Each entry in a typical ACL specifies a subject and an operation. For instance, If a file object has an ACL that contains (Alice: read,write; Bob: read), this would give Alice permission to read and write the file and give Bob permission only to read it.
A filesystem ACL is a data structure (usually a table) containing entries that specify individual user or group rights to specific system objects such as programs, processes, or files. These entries are known as access-control entries (ACEs). Each accessible object contains an identifier to its ACL. The privileges or permissions determine specific access rights, such as whether a user can read from, write to, or execute an object. In some implementations, an ACE can control whether or not a user, or group of users, may alter the ACL on an object.
The following is a list of important considerations regarding directories and partitions.
- /
- root partition / must always physically contain /etc, /bin, /sbin, /lib, /dev and /usr, otherwise you won’t be able to boot. This means that you should provide at least 600–750MB of disk space for the root partition including /usr, or 5–6GB for a workstation or a server installation.
- /var
- variable data like news articles, e-mails, web sites, databases, the packaging system cache, etc. will be placed under this directory. The size of this directory depends greatly on the usage of your system, but for most people will be dictated by the package management tool’s overhead. If you are going to do a full installation of just about everything Debian has to offer, all in one session, setting aside 2 or 3 GB of space for /var should be sufficient. If you are going to install in pieces (that is to say, install services and utilities, followed by text stuff, then X, …), you can get away with 300–500 MB. If hard drive space is at a premium and you don’t plan on doing major system updates, you can get by with as little as 30 or 40 MB.
- /tmp
- temporary data created by programs will most likely go in this directory. 40–100MB should usually be enough. Some applications — including archive manipulators, CD/DVD authoring tools, and multimedia software — may use /tmp to temporarily store image files. If you plan to use such applications, you should adjust the space available in /tmp accordingly.
- /home
- every user will put his personal data into a subdirectory of this directory. Its size depends on how many users will be using the system and what files are to be stored in their directories. Depending on your planned usage you should reserve about 100MB for each user, but adapt this value to your needs. Reserve a lot more space if you plan to save a lot of multimedia files (pictures, MP3, movies) in your home directory.
Is normally a binary file that contains the kernel. A boot loader loads this file into memory and sets it in motion when the system boots. Once the boot loader starts the kernel, the main kernel file is no longer used by the running system.
Contains ready-to-run programs (also known as executables), including most of the basic Unix commands such as ls and cp.
This directory contains executable files that are part of the core operating system (those required by the system for emergency repairs, booting, and single user mode). These files need to be accessible before /usr gets mounted. (for instance, the mount command is in /bin/mount). But in modern systems it is the same as the /usr/bin.
This directory are specified by the Linux Filesystem Hierarchy Standard to contain only files supplied and maintained by the Linux distributor.
Contains kernel boot loader files. These files pertain only to the very first stage of the Linux startup procedure, so you won’t find information about how Linux starts up its services in this directory.
Is normally a binary file that contains the kernel A boot loader loads this file into memory and sets it in motion when the system boots. Once the boot loader starts the kernel, the main kernel file is no longer used by the running system.
The GRUB configuration directory.
Contains device files.
These are the current hard disks connected to Linux systems. These devices represent entire disks.
Optical storage drives. This devices are read only, and they are used only for reading from discs.
This device is the controlling terminal of the current process. If a program is currently reading from and writing to a terminal, this device is a synonym for that terminal. A process does not need to be attached to a terminal.
The first virtual console, which is the one you see when you press CTRL-ALT-F1 or ALT-F1.
The first pseudoterminal device.
Contains the input of the several kernel devices (mouse, keyboard, touch-pad,…).
This file is a system device often referred to as a bit bucket, which accepts input and does nothing with it.
Descarta toda la información que se escribe en o se redirige hacia él. A su vez, no proporciona ningún dato a cualquier proceso que intente leer de él, devolviendo simplemente un EOF o fin de archivo.
Host-specific system configuration.
This core system configuration directory contains the user password, boot, device, networking, and other setup files.
The configurations for a single machine, such as user information /etc/passwd and network details /etc/network, go into this directory. However, general application details, such as a distribution’s defaults for a user interface, don’t belong here. System default configuration files not meant to be customized also are usually found elsewhere, as with the prepackaged systemd unit files in /usr/lib/systemd.
Archivo que utiliza APT que enlista las “fuentes” en donde se encuentran los paquetes. El archivo puede contener varios tipos de líneas. APT sabe como interpretar líneas del tipo http, ftp, file (archivos locales, p.e., un directorio que contiene un CD-ROM) y ssh.
#servidor oficial
deb http://security.debian.org/debian-security bullseye-security main non-free contrib
#paquetes fuente (normalmente usados para probar o recompilar)
deb-src http://security.debian.org/debian-security bullseye-security main non-free contrib
deb http://deb.debian.org/debian bullseye-proposed-updates main non-free contrib
#servidor de actualizaciones de seguridad
deb http://deb.debian.org/debian/ bullseye main non-free contrib
#paquetes fuente (normalmente usados para probar o recompilar)
deb-src http://deb.debian.org/debian/ bullseye main non-free contrib #Added by software-properties
- La primera palabra en cada línea indica el tipo de archivo: deb paquetes binarios pre-compilados que normalmente se usan. deb-src paquetes fuente que son los códigos originales.
- El primer argumento es la URL raiz de los archivos Debian.
- El segundo argumento es el nombre de la distribución.
- El tercer argumento y los siguientes son la lista de nombres de área de archivo válidos del archivo de Debian: main cumple con DFSG y sin dependencias con non-free . contrib cumple con DFSG pero con dependencias con non-free . non-free no cumple con DFSG.
Nota: suite name(codename)= stable(bullseye), testing(bookworm), unstable(siempre: sid).
This plaintext file maps usernames to user IDs.
Each line represents one user and has seven fields separated by colons.
┌ Login name | ┌ Password | | ┌ User ID | | | ┌ Group ID | | | | ┌ Real name (GECOS) | | | | | ┌ Home directory │ | | | | | ┌ Shell ┌─┴─┐┌┴┐┌─┴─┐┌─┴─┐┌──────┴──────┐┌────┴────┐ ┌───┴───┐ juser:x:3119:1000:J. Random User:/home/juser:/bin/bash
- The first field is the username.
- The second field in passwd or shadow is the encrypted password. An
xindicates that the encrypted password is stored in the shadow file. An asterisk (*) indicates that the user cannot log in. If this password field is blank (::), no password is required to log in. - The user ID (UID), which is the user’s representation in the kernel.
- The group ID (GID) Groups determine file permissions and little else.
- The user’s real name (often called the GECOS field).
- The user’s home directory.
- The user’s shell
The shadow password file normally contains user authentication information, including the encrypted passwords and password expiration information that correspond to the users in /etc/passwd.
This file defines the group IDs (such as the ones found in the /etc/passwd file).
Each line in this file is a set of fields separated by colons.
An x in password file means that there’s a corresponding entry in /etc/gshadow, and this is also nearly always a disabled password, denoted with a * or !.
┌ Group name | ┌ Password | | ┌ Group ID | | | ┌ Additional members ┌┴┐┌┴┐┌┴┐┌─────┴─────┐ disk:*:6:juser,beazley
Configure the privileged users.
Example, this file gives user1 and user2 the power to run any command as root without having to enter a password:
User_Alias ADMINS = user1, user2
ADMINS ALL = NOPASSWD: ALL root ALL=(ALL) ALL
The first line defines an ADMINS user alias with the two users, and the second line grants the privileges. The ALL = NOPASSWD: ALL part means that the users in the ADMINS alias can use sudo to execute commands as root. The second ALL means “any command.” The first ALL means “any host.” (If you have more than one machine, you can set different kinds of access for each machine or group of machines.)
The root ALL=(ALL) ALL simply means that the superuser may also use sudo to run any command on any host. The extra (ALL) means that the superuser may also run commands as any other user. You can extend this privilege to the ADMINS users by adding (ALL) to the second /etc/sudoers line, as shown here:
ADMINS ALL = (ALL) NOPASSWD: ALL
List of filesystems and options.
Inside this file, each line corresponds to one filesystem and is broken into six fields. From left to right, these fields are:
- the device or UUID
- most current Linux systems no longer use the device in
/etc/fstab, preferring the UUID. - the mount point
- indicates where to attach the filesystem.
- the filesystem type
- you may not recognize
swapin this list; this is a swap partition. - options
- long options, separated by commas.
- backup information for use by the dump command
- the
dumpcommand is a long-obsolete backup utility; this field is no longer relevant. You should always set it to 0. - the filesystem integrity test order
- to ensure that
fsckalways runs on the root first, always set this to 1 for the root filesystem and 2 for any other locally attached filesystems on a hard disk or SSD. Use 0 to disable the bootup check for every other filesystem, including read-only devices, swap, and the/procfilesystem.
You’ll often see the defaults option. These options are defined as follows:
- defaults
- this sets the
mountdefaults: read-write mode, enable device files, executables, thesetuidbit, and so on. Use this when you don’t want to give the filesystem any special options but you do want to fill all fields in/etc/fstab. - errors
- this ext2/3/4-specific parameter sets the kernel behavior when the system has trouble mounting a filesystem. The default is normally
errors=continue, meaning that the kernel should return an error code and keep running. To have the kernel try the mount again in read-only mode, useerrors=remount-ro. Theerrors=panicsetting tells the kernel (and your system) to halt when there’s a problem with the mount. - noauto
- this option tells a
mount -acommand to ignore the entry. Use this to prevent a boot-time mount of a removable-media device, such as a flash storage device. - user
- this option allows unprivileged users to run
mounton a particular entry, which can be handy for allowing certain kinds of access to removable media. Because users can put asetuid-rootfile on removable media with another system, this option also setsnosuid,noexec, andnodev(to bar special device files). Keep in mind that for removable media and other general cases, this option is now of limited use, because most systems useubusalong with other mechanisms to automatically mount inserted media. However, this option can be useful in special cases when you want to grant control over mounting specific directories.
System unit configuration directory (local definitions).
You should normally put your own unit files in this directory so that you won’t confuse them with anything that came with your distribution and so that the distribution won’t overwrite them when you upgrade.
The contents of the rc*.d directories are actually symbolic links to files in yet another directory, /etc/init.d.
The rc stands for run commands, which many people refer to as scripts, programs, or services, and the 5 in rc5.d tells us that we’re talking about runlevel 5.
The rc5.d directory contain file like this S10sysklogd, S20ppp, S99gpm… in this files the capital S in a command name means that the command should run in start mode, and the number (00 through 99) determines where in the sequence rc starts the command. The rc*.d commands are usually shell scripts that start programs in /sbin or /usr/sbin.
Some rc*.d directories contain commands that start with K (for “kill,” or stop mode). In this case, rc runs the command with the stop argument instead of start. You’ll most likely encounter K commands in runlevels that shut down the system.
Contains a number of start/stop scripts for various services on the system (changing init states).
To start and stop services by hand, use the script in this init.d directory. For example, one way to start the httpd web server program manually is to run init.d/httpd start. Similarly, to kill a running service, you can use the stop argument (httpd stop, for instance).
The mechanism that System V init uses to run the init.d scripts has found its way into many Linux systems, regardless of whether they use System V init. It’s a utility called run-parts, and the only thing it does is run a bunch of executable programs in a given directory, in some kind of predictable order.
The default behavior is to run all programs in a directory, but you often have the option to select certain programs and ignore others.
Distributions, such as Debian and Ubuntu, have a more complicated run-parts program. Their features include the ability to run programs based on a regular expression (for example, using the S[0-9]{2} expression for running all “start” scripts in an /etc/init.d runlevel directory) and to pass arguments to the programs. These capabilities allow you to start and stop System V runlevels with a single command.
Configuration file for anacron.
Many common cron-activated system tasks are run as the superuser. However, rather than editing and maintaining a superuser’s crontab to schedule these, Linux distributions normally have an /etc/crontab file for the entire system. You won’t use crontab to edit this file, and in any case, it’s slightly different in format: before the command to run, there’s an additional field specifying the user that should run the job. (This gives you the opportunity to group system tasks together even if they aren’t all run by the same user.)
# this cron job runs at 6:42 AM as the superuser (root 1)
42 6 * * * root1 /usr/local/bin/cleansystem > /dev/null 2>&1
A star (*) in any field means to match every value. The example below runs spmake daily because the day of month, month, and day of week fields are all filled with stars, which cron reads as “run this job every day, of every month, of every day of the week.” (these examples are similar, usually goin in another file, they don’t specify the user who runs the command).
┌ Minute (0 - 59)
| ┌ Hour (0 - 23)
| | ┌ Day of month (1 - 31)
| | | ┌ Month (1 - 12)
| | | | ┌ Day of week (0 - 6) (Sunday=0 or 7)
| | | | | ┌ Command
┌┴┐┌┴┐┌┴┐┌┴┐┌┴┐┌────────┴────────┐
15 09 * * * /home/juser/bin/spmake
# run spmake only on the 14th day of each month
15 09 14 * * /home/juser/bin/spmake
# run spmake on the 5th and the 14th day of each month
15 09 5,14 * * /home/juser/bin/spmake
Some distributions store additional system crontab files in this directory. These files may have any name, but they have the same format as /etc/crontab.
The files here are usually scripts run by a specific cron job in /etc/crontab or /etc/cron.d.
Contains the Internet Protocol (IP) host names and addresses for the local host and other hosts in the Internet network. This file is used to resolve a name into an address (that is, to translate a host name into its Internet address).
127.0.0.1 localhost
10.23.2.3 atlantic.aem7.net atlantic
10.23.2.4 pacific.aem7.net pacific
::1 localhost ip6-localhost
# The other entries here illustrate a simple way to add hosts
# on a local subnet.
Thi is the traditional configuration file for DNS servers.
Is the traditional interface for controlling several name-related precedence settings on your system, such as user and password information, and it has a host lookup setting.
It can be complicated to remember all of the places that name lookups can happen, but if you ever need to trace something from the bottom up, start with this file.
#Dicta el orden de resolución de host.
hosts: files mdns4_minimal [NOTFOUND=return] dns myhostname
- Si se encuentra primero “files”, buscara primero en el archivo
/etc/hostsy devolvera todas las direcciones validas para él y sale. - Se invoca el método “dns”. Si la consulta al (DNS) de Internet identificado por el archivo
/etc/resolv.confencuentra el nombre de host, devuelve todas las direcciones válidas para él y sale.
Contains the service names and port assignments of specific z/OS UNIX applications, i.e. translates well-known port numbers into names.
This is the sshd server system-wide configuration file.
This is the ssh client system-wide configuration file.
Each file in this directory is a fontconfig configuration file. These files are normally installed in /usr/share/fontconfig/conf.avail and then symlinked here, allowing them to be easily installed and then enabled/disabled by adjusting the symlinks.
A global configuration script for login shell sessions that applies to all users.
A global configuration script for non-login shell sessions that applies to all users.
Interface configuration (if not controlled via network-manager).
Default values for many daemons and services.
Holds home (personal) directories for regular users.
A user’s personal startup file for login shell. This can be used to extend or override settings in the global configuration script.
If /home/user/_bash_profile is not found, bash attempts to read this script (login shell).
If neither /home/user/_bash_profile nor /home/user/_bash_login is found, bash attempts to read this file. (login shell) This is the default in Debian-based distributions, such as Ubuntu.
A user’s personal startup file for non-login shell. It can be used to extend or override settings in the global configuration script.
The user trash-bins (or wastebaskets) folder. Note: dot.local is actually .local.
The other two trash-bins are:
- The administrator wastebasket :
/root/.local/share/Trash - The external wastebaskets : locates on your external disks, they are usually named
/media/your_id/your_disk/.Trash_1000
An abbreviation for library, this directory holds library files containing code that executables can use. This directory should not contain static libraries.
Are modules that the kernel loads and unloads on demand during the course of normal system operation.
A base attachment point for removable media such as flash drives that is found in many distributions.
Mount point for mounting a file system temporarily.
It’s a virtual file system maintained by the Linux kernel. Provides system statistics through a browsable directory-and-file interface. The /proc directory contains information about currently running processes as well as some kernel parameters. The files are readable and will give us a picture of how the kernel sees the computer.
Each numbered directory inside /proc refers to the ID of a current process on the system; the files in each directory represent various aspects of that process. The directory /proc/self represents the current process. The Linux proc filesystem includes a great deal of additional kernel and hardware information in files like /proc/cpuinfo. Keep in mind that the kernel design guidelines recommend moving information unrelated to processes out of /proc and into /sys, so system information in /proc might not be the most current interface.
Show directories and files assosiate with <pid> process.
Conjunto de descriptores de archivos (File Descriptor) abiertos en un proceso PID. Cada proceso en UNIX debe tener tres descriptores de archivo POSIX estándar, correspondientes a los tres flujos estándar:
- El descriptor de archivo
0es el stdin . - El descriptor de archivo
1es el stdout . - El descriptor de archivo
2es el stderr .
Los otros descriptores de archivos pueden ser archivos que ha abierto el proceso para utilizar, como por ejemplo, un archivo llamado file abierto por el proceso less file .
En la implementación tradicional de Unix, los descriptores de archivos se indexan en una tabla de descriptores de archivos por proceso mantenida por el kernel, que a su vez se indexa en una tabla de archivos de todo el sistema abierta por todos los procesos, llamada tabla de archivos. Esta tabla registra el modo con el que se ha abierto el archivo (u otro recurso): para lectura, escritura, anexión y posiblemente otros modos. También se indexa en una tercera tabla llamada tabla de inodos que describe los archivos subyacentes reales. Para realizar la entrada o la salida, el proceso pasa el descriptor de archivo al kernel a través de una llamada al sistema y el kernel accederá al archivo en nombre del proceso. El proceso no tiene acceso directo al archivo ni a las tablas de inodos.
List the v1 and v2 cgroups for any process. Every line of output here starts with a number and is a different cgroup.
- Numbers 2–12 are for cgroups v1. The controllers for those are listed next to the number.
- Number 1 is also for version 1, but it does not have a controller. This cgroup is for management purposes only (in this case, systemd configured it).
- The last line, number 0, is for cgroups v2. No controllers are visible here. On a system that doesn’t have cgroups v1, this will be the only line of output.
- Names are hierarchical and look like parts of file paths. You can see in this example that some of the cgroups are named
/user.sliceand others/user.slice/user-1000.slice/session-2.scope. - The name
/testcgroupwas created to show that in cgroups v1, the cgroups for a process can be completely independent. - Names under
user.slicethat include session are login sessions, assigned by systemd. You’ll see them when you’re looking at a shell’s cgroups. The cgroups for your system services will be undersystem.slice.
cat /proc/self/cgroup
# 12:rdma:/
# 11:net_cls,net_prio:/
# 10:perf_event:/
# 9:cpuset:/
# 8:cpu,cpuacct:/user.slice
# 7:blkio:/user.slice
# 6:memory:/user.slice
# 5:pids:/user.slice/user-1000.slice/session-2.scope
# 4:devices:/user.slice
# 3:freezer:/
# 2:hugetlb:/testcgroup
# 1:name=systemd:/user.slice/user-1000.slice/session-2.scope
# 0::/user.slice/user-1000.slice/session-2.scope
# Let’s explore the cgroup setup of a shell.
cd /sys/fs/cgroup/user.slice/user-1000.slice/session-2.scope/
ls
# Lists the processes in the cgroup.
cat cgroup.procs
# Show the controllers currently in use for the cgroup.
cat cgroup.controllers
# Number of threads running in the cgroup.
cat pids.current
# Show the maximum amount of memory that the cgroup can consume.
cat memory.max
Is a link to the currently running process. This allows a process to look at itself without having to know its process ID.
Show the block and character devices for which your system currently has drivers. Each line consists of a number and name. The number is the major number of the device.
Show how much real memory is being used for caches and buffers.
Incluye gran cantidad de información adicional sobre el kernel y el hardware.
Version of the Linux kernel used, its name, the version of the compiler used.
Full partition information.
Contiene las particiones swap disponibles.
Show the parameters passed to your system’s currently running kernel.
Contains a variety of statistics about the system, such as the number of processes running, the number of interrupts, and the amount of time spent in each CPU state.
Contains the amount of time the system has been running.
Contiene interfaces para cambiar ciertos parámetros del kernel en tiempo de ejecución. (Se puede hacer lo mismo mediante el comando especializado sysctl(8) o su archivo de precarga/configuración /etc/sysctl.conf ).
This is the home directory for the root account.
Contains runtime variable data specific to the system, including certain process IDs, socket files, status records, and, in many cases, system logging.
El directorio /run se monta como tmpfs en el proceso de arranque inicial. Esto permite escribir en él incluso cuando el directorio ”/” está montado como de solo lectura. Esta es la nueva ubicación para el almacenamiento de archivos de estado transitorio y reemplaza varias ubicaciones descritas en Filesystem Hierarchy Standard.
Archivos de configuración generados en tiempo de ejecución que anulan los archivos de configuración instalados.
Essential system binaries.
Programs in /sbin directories relate to system management, so regular users usually do not have /sbin components in their command paths.
These are programs that perform vital system tasks that are generally reserved for the superuser.
Virtual directory for system information.
This directory is similar to /proc in that it provides a device and system interface.
To provide a uniform view for attached devices based on their actual hardware attributes, the Linux kernel offers the sysfs interface through a system of files and directories.
This path has different purposes than /dev/sda filename, which is also a directory. The /dev file enables user processes to use the device, whereas the /sys/devices path is used to view information and manage the device.
Contain all of the block devices available on a system. However, those are just symbolic links.
cgroups are accessed entirely through the filesystem, which is usually mounted as a cgroup2 filesystem under this directory.
A storage area for smaller, temporary files created by various programs. Some configurations cause this directory to be emptied each time the system is rebooted.
Any user may read to and write from /tmp, but the user may not have permission to access another user’s files there.
Secondary hierarchy. It contains a large directory hierarchy, including the bulk of the Linux system. /usr is where most of the user-space programs and data reside.
Is where administrators can install their own software (programs that are not included with the distribution). Its structure should look like that of / and /usr.
Is where administrators can place the source code directory of a program for installation.
Install a font manually.
Install a font manually by downloading the appropriate .ttf or otf files and placing them into /usr/local/share/fonts (system-wide), ~~/.local/share/fonts~ (user-specific) or ~~/.fonts~ (user-specific). These files should have the permission 644 (-rw-r–r–), otherwise they may not be usable.
Fonts can be added system-wide to Debian by installing the appropriate package. As default system-wide fonts will be installed into /usr/share/fonts by the package-manager.
Fonts can be added system-wide to Debian by installing the appropriate package. As default system-wide fonts will be installed into /usr/share/fonts by the package-manager.
Notes:
- For most uses, you’ll want TrueType (TTF) and OpenType (OTF) fonts – these packages start with
fonts-. - Some fonts might have a -variable build, which are known as (VF), variable font.
- Some non-free font downloader packages are in
contrib, which you will need to add to yoursourcesif not present.
Manually
Install a font manually by downloading the appropriate .ttf or otf files and placing them into /usr/local/share/fonts (system-wide), ~~/.local/share/fonts~ (user-specific) or ~~/.fonts~ (user-specific). These files should have the permission 644 (-rw-r–r–), otherwise they may not be usable.
Contains executable files that are not part of the core operating system. These are considered to be “system wide binaries”. But in modern systems it is the same as the /bin.
This directory are specified by the Linux Filesystem Hierarchy Standard to contain only files (executables for nearly all packaged user software) supplied and maintained by the Linux distributor.
Holds header files used by the C compiler.
Another system unit directory.
Contains manual pages.
Contains files that should work on other kinds of Unix machines with no loss of functionality. These are usually auxiliary data files that programs and libraries read as necessary.
This includes things such as default configuration files, icons, screen backgrounds, sound files, etc.
It typically contains architecture-independent (shared) data. This means it holds files that are not specific to the architecture of your machine, such as certain shell scripts, man pages, locale information, and other miscellaneous system-wide data.
This is particularly useful in networked environments where multiple machines can share a common data source, thereby saving storage and ensuring consistency.
Some packages dump their available documentation into this directory with no regard for online manual systems such as man or info. See this directory on your system if you find yourself searching for documentation.
This directory contains a lot of time zones and aliases for time zones.
To set your system’s time zone manually, either copy one of the files in /usr/share/zoneinfo to /etc/localtime (or make a symbolic link) or change it with your distribution’s time zone tool. The command-line program tzselect may help you identify a time zone file.
Each file in this directory is a fontconfig configuration file.
Contains a dictionaries.
The variable subdirectory date, where programs record information that can change over the course of time. System logging, user tracking, caches, and other files that system programs create and manage are here.
Contains log files, records of various system activity.
If you have a version of syslogd, this directory should contain many files, most created by your syslog daemon. However, there will be a few files here that are maintained by other services. In addition, there may be further subdirectories containing logs.
The main log file that show what the system is doing in near real-time. It’s used to store non-debug and non-critical system messages. This includes messages with log levels such as info, notice, and warn. It’s often referred to as the “general system activity” log.
Useful for real-time view to determine how the removable device is named when it is attached.
Tip: Using the tail -f /var/log/messages technique is a great way to watch what the system is doing in near real-time.
Is a standard logging facility. It collects messages of various programs and services including the kernel, and stores them, depending on setup, in a bunch of log files typically under /var/log. It logs everything except authentication-related messages.
In some datacenter setups there are hundreds of devices each with its own log; syslog comes in handy here too. One just sets up a dedicated syslog server which collects all the individual device logs over the network.
En este se puede encontrar los mensajes anacron.
Is where journald stores its (binary) logfiles.
The journal is a component of systemd. It’s a centralized location for all messages logged by different components in a systemd-enabled Linux system. This includes kernel and boot messages, messages coming from syslog, or different services.
The journal is controlled by the systemd-journald daemon. It collects information from different sources and loads the messages into the journal.
View the messages in the kernel ring buffer.
Containing the most recent log messages.
Find package information.
List of available packages from repositories.
Status of installed and available packages. This file contains information about whether a package is marked for removal or not, whether it is installed or not, etc. A package marked reinst-required is broken and requires re-installation.
Each user can have their own crontab file, usually found in this directory. Normal users can’t write to this directory; the crontab command installs, lists, edits, and removes a user’s crontab.
The easiest way to install a crontab is to put your crontab entries into a file and then use crontab file to install file as your current crontab. The crontab command checks the file format to make sure that you haven’t made any mistakes. To list your cron jobs, run crontab -l. To remove the crontab, use crontab -r.
After you’ve created your initial crontab, it can be a bit messy to use temporary files to make further edits. Instead, you can edit and install your crontab in one step with the crontab -e command. If you make a mistake, crontab should tell you where the mistake is and ask if you want to try editing again.
Entrega correos a la dirección de destino en el mismo host, el comando /usr/sbin/sendmail realiza la entrega local agregándolo a este archivo.
Para la dirección de destino en el host remoto, el comando /usr/sbin/sendmail realiza una transferencia remota del correo electrónico al host de destino encontrado por el registro DNS MX mediante SMTP.
Por ejemplo cuando se ejecuta un anacron la salida del comando la envia a este archivo.
Contain the unix domain socket files for D-Bus.
Almacena los paquetes obtenidos vía mecanismos APT hasta que estos sean borrados, por lo general mediante apt clean.
Estructura de un paquete binario .deb:
package-name_upstream-version-debian.revision_architecture.deb
chromium-common_106.0.5249.119-1~deb11u1_amd64.deb
Archivo html que sirve para probar el servidor local apache2.
Data for services provided by the system.
Add-on application software packages. This may contain additional third-party software.
Each formatted partition or device using a Linux file system, such as ext4, will have this directory. It is used in the case of a partial recovery from a file system corruption event. Unless something really bad has happened to our system, this directory will remain empty.
It provides a way to fsck (File System Check utility) to recover parts as small data structure.
When fsck asks you about reconnecting an inode, it has found a file that doesn’t appear to have a name. When reconnecting such a file, fsck places the file in the lost+found directory in the filesystem, with a number as the filename. If this happens, you need to guess the name based on the file’s contents; the original filename is probably gone.
Compiling is the process of translating source code (the human-readable description of a program written by a programmer) into the native language of the computer’s processor.
The computer’s processor (or CPU) works at an elemental level, executing programs in what is called machine language. This is a numeric code that describes extremely small operations, such as “add this byte,” “point to this location in memory,” or “copy this byte.” Each of these instructions is expressed in binary (ones and zeros). The earliest computer programs were written using this numeric code.
This problem was overcome by the advent of assembly language, which replaced the numeric codes with (slightly) easier to use character mnemonics such as CPY (for copy) and MOV (for move). Programs written in assembly language are processed into machine language by a program called an assembler. Assembly language is still used today for certain specialized programming tasks, such as device drivers and embedded systems.
We next come to what are called high-level programming languages. They are called this because they allow the programmer to be less concerned with the details of what the processor is doing and more with solving the problem at hand. The early ones (developed during the 1950s) include FORTRAN (designed for scientific and technical tasks) and COBOL (designed for business applications).
Programs written in high-level programming languages are converted into machine language by processing them with another program, called a compiler. Some compilers translate high-level instructions into assembly language and then use an assembler to perform the final stage of translation into machine language.
A process often used in conjunction with compiling is called linking. There are many common tasks performed by programs. Take, for instance, opening a file. Many programs perform this task, but it would be wasteful to have each program implement its own routine to open files. It makes more sense to have a single piece of programming that knows how to open files and to allow all programs that need it to share it. Providing support for common tasks is accomplished by what are called libraries. They contain multiple routines, each performing some common task that multiple programs can share. If we look in the /lib and /usr/lib directories, we can see where many of them live. A program called a linker is used to form the connections between the output of the compiler and the libraries that the compiled program requires. The final result of this process is the executable program file, ready for use.
Is a binary file that a processor can almost understand, except that there are still a few loose ends. First, the operating system doesn’t know how to start up an object file, and second, you likely need to combine several object files and some system libraries to make a complete program.
To build a fully functioning executable program from one or more object files, you must run the linker, the ld command in Unix. The make system is the traditional Unix standard for managing and automating compiles.
Is a collection of common precompiled components that you can build into your program, and it’s really not much more than a bundle of object files along with some header files.
Libraries come into play primarily at link time, when the linker program ld creates an executable from object files. Linking using a library is often called linking against a library.
A library file ending with .a (such as libcurses.a) is called a static library. When you link a program against a static library, the linker copies the necessary machine code from the library file into your executable. Once it does this, the final executable no longer needs the original library file when it runs, and because your executable has its own copy of the library code, the executable’s behavior is not subject to change if the .a file changes.
Linking a program against a shared library doesn’t copy the code into the final executable; it just adds references to names in the code of the library file. When you run the program, the system loads the library’s code into the process memory space only when necessary. Many processes can share the same shared library code in memory. And if you need to slightly modify the library code, you can generally do so without recompiling any programs. When updating software on your Linux distribution, the packages that you’re updating can include shared libraries. When your update manager asks you to reboot your machine, sometimes it’s doing so to be sure that every part of your system is using new versions of shared libraries.
A shared library has a suffix that contains .so (shared object), as in libc-2.15.so and libc.so.6.
C header files are additional source code files that usually contain type and library function declarations.
Double quotes in header files like #include "myheader.h" mean that the header file is not in a system include directory, and usually indicate that the include file is in the same directory as the source file. If you encounter a problem with double quotes, you’re probably trying to compile incomplete source code.
The C compiler doesn’t actually do the work of looking for the include files. That task falls to the C preprocessor, a program that the compiler runs on your source code before parsing the actual program. The preprocessor rewrites source code into a form that the compiler understands; it’s a tool for making source code easier to read (and for providing shortcuts).
Preprocessor commands in the source code are called directives, and they start with the # character. There are three basic types of directives:
- Include files
- an
#includedirective instructs the preprocessor to include an entire file. - Macro definitions
- a line such as
#define BLAH somethingtells the preprocessor to substitutesomethingfor all occurrences ofBLAHin the source code. - Conditionals
- you can mark out certain pieces of code with
#ifdef,#if, and#endif. The#ifdefMACROdirective checks to see whether the preprocessor macroMACROis defined, and#if conditiontests to see whetherconditionis nonzero.
The basic idea behind make is the target, a goal that you want to achieve. A target can be a file (a .o file, an executable, and so on) or a label. In addition, some targets depend on other targets; for instance, you need a complete set of .o files before you can link your executable. These requirements are called dependencies.
To build a target, make follows a rule, such as one specifying how to go from a .c source file to a .o object file. make already knows several rules, but you can customize them to create your own.
One last make fundamental concept to know is that, in general, the goal is to bring targets up to date with their dependencies.
The following very simple Makefile builds a program called myprog from aux.c and main.c:
/* main.c */
void hello_call();
int main() {
hello_call();
}
/* aux.c */
#include <stdio.h>
void hello_call() {
printf("Hello, World.\n");
}# object files
OBJS=aux.o main.o # macro definition sets OBJS variable to two object
all: myprog # (all) first target (is always the default)
# all depends on myprog
# myprog (is the rule for all and the target of OBJS)
myprog: $(OBJS) # Makefile uses the macro $(OBJS) in the dependencies
$(CC) -o myprog $(OBJS) # The whitespace before $(CC) is a tab
Note: The whitespace before $(CC) in the Makefile file is a tab. make is very strict about tabs.You must insert a tab before any system command, on its own line.
This Makefile assumes that it has two C source files named aux.c and main.c in the same directory. Running make on the Makefile yields the following output, showing the commands that make is running:
make
cc -c -o aux.o aux.c
cc -c -o main.o main.c
cc -o myprog aux.o main.o
Makefile dependencies
+--------+
| myprog |
+--------+
---/\---
--/ \--
+--------+ +-------+
| main.o | | aux.o |
+--------+ +-------+
| |
v v
+--------+ +-------+
| main.c | | aux.c |
+--------+ +-------+
- CFLAGS
- C compiler options. When creating object code from a
.cfile,makepasses this as an argument to the compiler. - LDFLAGS
- like
CFLAGS, but these options are for the linker when creating an executable from object code. - LDLIBS
- if you use
LDFLAGSbut don’t want to combine the library name options with the search path, put the library name options in this file. - CC
- the C compiler. The default is
cc. - CPPFLAGS
- C preprocessor options. When make runs the C preprocessor in some way, it passes this macro’s expansion on as an argument.
- CXXFLAGS
- GNU make uses this for C++ compiler flags.
- $@
- when inside a rule, this variable expands to the current target.
- $<
- when inside a rule, this variable expands to the first dependency of the target.
- $*
- this variable expands to the basename or stem of the current target. For example, if you’re building
blah.o, this expands toblah.
- clean
- the
cleantarget is ubiquitous; amake cleanusually instructsmaketo remove all of the object files and executables so that you can make a fresh start or pack up the software. - distclean
- a Makefile created by way of the GNU autotools system always has a
distcleantarget to remove everything that wasn’t part of the original distribution, including the Makefile. - install
- this target copies files and compiled programs to what the Makefile thinks is the proper place on the system.
- test or check
- some developers provide
testorchecktargets to make sure that everything works after performing a build. - depend
- this target creates dependencies by calling the compiler with
-Mto examine the source code. - all
- this is commonly the first target in the Makefile. You’ll often see references to this target instead of an actual executable.
The configure program is a shell script that is supplied with the source tree. Its job is to analyze the build environment. Most source code is designed to be portable. That is, it is designed to build on more than one kind of Unix-like system. But to do that, the source code may need to undergo slight adjustments during the build to accommodate differences between systems. configure also checks to see that necessary external tools and components are installed.
The makefile is a configuration file that instructs the make program exactly how to build the program. Without it, make will refuse to run.
The make program takes as input a makefile (which is normally named Makefile), which describes the relationships and dependencies among the components that comprise the finished program.
- Unpack the source code archive.
- Configure the package (
./configure). - Run
makeor another build command to build the programs. - Run
make installor a distribution-specific install command to install the package.
GNU autoconf is a popular system for automatic Makefile generation. Packages using this system come with files named configure, Makefile.in, and config.h.in. The .in files are templates; the idea is to run the configure script in order to discover the characteristics of your system, and then make substitutions in the .in files to create the real build files.
The default prefix in GNU autoconf and many other packages is /usr/local, the traditional directory for locally installed software. Operating system upgrades ignore /usr/local, so you won’t lose anything installed there during an operating system upgrade, and for small local software installations, /usr/local is fine. The only problem is that if you have a lot of custom software installed, this can turn into a terrible mess. Thousands of odd little files can make their way into the /usr/local hierarchy, and you may have no idea where the files came from.
If things really start to get unruly, you should create your own packages with # checkinstall make install.
Autoconf targets:
- make clean
- removes all object files, executables, and libraries.
- make distclean
- this is similar to
make cleanexcept it removes all automatically generated files, includingMakefiles,config.h,config.log, and so on. The idea is that the source tree should look like a newly unpacked distribution after runningmake distclean. - make check
- some packages come with a battery of tests to verify that the compiled programs work properly; the command
make checkruns those tests. - make install-strip
- this is like
make installexcept it strips the symbol table and other debugging information from executables and libraries when installing. Stripped binaries require much less space.
- We’ll install the GNU coreutils package in the own home directory.
- Get the package from http://ftp.gnu.org/gnu/coreutils/, unpack it, change to its directory.
- Configure it like this:
./configure --prefix=$HOME/mycoreutils. - Now run
make. - Try to run one of the executables you just created, such as
./src/ls, and try runningmake checkto run a series of tests on the package. - Do a dry run with
make -n installfirst to see whatmake installdoes without actually doing the install. - Do the install
make install.
Advantages:
- You can customize package defaults.
- When installing a package, you often get a clearer picture of how to use it. You control the release that you run.
- It’s easier to back up a custom package.
- It’s easier to distribute self-installed packages across a network.
Disadvantages:
- If the package you want to install is already installed on your system, you might overwrite important files, causing problems. Avoid this by using the
/usr/localinstall prefix. - It takes time.
- Custom packages do not automatically upgrade themselves.
- There is a potential for misconfiguring packages.
If you must use LD_LIBRARY_PATH to run some crummy program for which you don’t have the source (or an application that you’d rather not recompile, like Firefox or some other beast), use a wrapper script. Let’s say your executable is /opt/crummy/bin/crummy.bin and needs some shared libraries in /opt/crummy/lib. Write a wrapper script called crummy that looks like this:
#!/bin/sh
LD_LIBRARY_PATH=/opt/crummy/lib
export LD_LIBRARY_PATH
exec /opt/crummy/bin/crummy.bin $@
Avoiding LD_LIBRARY_PATH prevents most shared library problems. But one other significant problem that occasionally comes up with developers is that alibrary’s API may change slightly from one minor version to another, breaking installed software. The best solutions here are preventive: either use a consistent methodology to install shared libraries with -Wl, -rpath to create a runtime link path, or simply use the static versions of obscure libraries.
In its rawest form, main memory is just a big storage area for a bunch of 0s and 1s. This is where the running kernel and processes reside—they’re just big collections of bits. All input and output from peripheral devices flows through main memory, also as a bunch of bits.
Is just an operator on memory; it reads its instructions and data from the memory and writes data back out to the memory.
Displays the current time. This is followed by the system uptime, which tells us the time for which the system has been running.
The load average section represents the average “load” over one, five and fifteen minutes. “Load” is a measure of the amount of computational work a system performs. On Linux, the load is the number of processes in the R (Running) and D (Uninterruptible sleep) states at any given moment. The “load average” value gives you a relative measure of how long you must wait for things to get done.
On a single core system, a load average of 0.4 means the system is doing only 40% of work it can do. A load average of 1 means that the system is exactly at capacity — the system will be overloaded by adding even a little bit of additional work. A system with a load average of 2.12 means that it is overloaded by 112% more work than it can handle.
Note: Examining Load Average The Intuitive Interpretation
The three load-average values in the first line of top output are the 1-minute, 5-minute and 15-minute average. (These values also are displayed by other commands, such as uptime, not only top.) That means, reading from left to right, one can examine the aging trend and/or duration of the particular system state. The state in question is CPU load—not to be confused with CPU percentage. In fact, it is precisely the CPU load that is measured, because load averages do not include any processes or threads waiting on I/O, networking, databases or anything else not demanding the CPU. It narrowly focuses on what is actively demanding CPU time. This differs greatly from the CPU percentage. The CPU percentage is the amount of a time interval (that is, the sampling interval) that the system’s processes were found to be active on the CPU. If top reports that your program is taking 45% CPU, 45% of the samples taken by top found your process active on the CPU. The rest of the time your application was in a wait. (It is important to remember that a CPU is a discrete state machine. It really can be at only 100%, executing an instruction, or at 0%, waiting for something to do. There is no such thing as using 45% of a CPU. The CPU percentage is a function of time.) However, it is likely that your application’s rest periods include waiting to be dispatched on a CPU and not on external devices. That part of the wait percentage is then very relevant to understanding your overall CPU usage pattern.
The load averages differ from CPU percentage in two significant ways: 1) load averages measure the trend in CPU utilization not only an instantaneous snapshot, as does percentage, and 2) load averages include all demand for the CPU not only how much was active at the time of measurement.
Authors tend to overuse analogies and sometimes run the risk of either insulting the reader’s intelligence or oversimplifying the topic to the point of losing important details. However, freeway traffic patterns are a perfect analogy for this topic, because this model encapsulates the essence of resource contention and is also the chosen metaphor by many authors of queuing theory books. Not surprisingly, CPU contention is a queuing theory problem, and the concepts of arrival rates, Poisson theory and service rates all apply. A four-processor machine can be visualized as a four-lane freeway. Each lane provides the path on which instructions can execute. A vehicle can represent those instructions. Additionally, there are vehicles on the entrance lanes ready to travel down the freeway, and the four lanes either are ready to accommodate that demand or they’re not. If all freeway lanes are jammed, the cars entering have to wait for an opening. If we now apply the CPU percentage and CPU load-average measurements to this situation, percentage examines the relative amount of time each vehicle was found occupying a freeway lane, which inherently ignores the pent-up demand for the freeway—that is, the cars lined up on the entrances. So, for example, vehicle license XYZ 123 was found on the freeway 30% of the sampling time. Vehicle license ABC 987 was found on the freeway 14% of the time. That gives a picture of how each vehicle is utilizing the freeway, but it does not indicate demand for the freeway.
Moreover, the percentage of time these vehicles are found on the freeway tells us nothing about the overall traffic pattern except, perhaps, that they are taking longer to get to their destination than they would like. Thus, we probably would suspect some sort of a jam, but the CPU percentage would not tell us for sure. The load averages, on the other hand, would.
This brings us to the point. It is the overall traffic pattern of the freeway itself that gives us the best picture of the traffic situation, not merely how often cars are found occupying lanes. The load average gives us that view because it includes the cars that are queuing up to get on the freeway. It could be the case that it is a nonrush-hour time of day, and there is little demand for the freeway, but there just happens to be a lot of cars on the road. The CPU percentage shows us how much the cars are using the freeway, but the load averages show us the whole picture, including pent-up demand. Even more interesting, the more recent that pent-up demand is, the more the load-average value reflects it.
Taking the discussion back to the machinery at hand, the load averages tell us by increasing duration whether our physical CPUs are over- or under-utilized. The point of perfect utilization, meaning that the CPUs are always busy and, yet, no process ever waits for one, is the average matching the number of CPUs. If there are four CPUs on a machine and the reported one-minute load average is 4.00, the machine has been utilizing its processors perfectly for the last 60 seconds. This understanding can be extrapolated to the 5- and 15-minute averages.
In general, the intuitive idea of load averages is the higher they rise above the number of processors, the more demand there is for the CPUs, and the lower they fall below the number of processors, the more untapped CPU capacity there is. But all is not as it appears. The Wizard behind the Curtain
The load-average calculation is best thought of as a moving average of processes in Linux’s run queue marked running or uninterruptible. The words “thought of” were chosen for a reason: that is how the measurements are meant to be interpreted, but not exactly what happens behind the curtain. It is at this juncture in our journey when the reality of it all, like quantum mechanics, seems not to fit the intuitive way as it presents itself.
State of a process:
- Runnable (R)
- A process in this state is either executing on the CPU, or it is present on the run queue, ready to be executed.
- Interruptible sleep (S)
- Processes in this state are waiting for an event to complete.
- Uninterruptible sleep (D)
- In this case, a process is waiting for an I/O operation to complete.
- Stopped (T)
- These processes have been stopped by a job control signal (such as by pressing Ctrl+Z) or because they are being traced.
- Zombie (Z)
- The kernel maintains various data structures in memory to keep track of processes. A process may create a number of child processes, and they may exit while the parent is still around. However, these data structures must be kept around until the parent obtains the status of the child processes. Such terminated processes whose data structures are still around are called zombies.
Processes in the D and S states are shown in “sleeping”, and those in the T state are shown in “stopped”. The number of zombies are shown as the “zombie” value.
The CPU usage section shows the percentage of CPU time spent on various tasks. The us value is the time the CPU spends executing processes in userspace. Similarly, the sy value is the time spent on running kernelspace processes.
Linux uses a “nice” value to determine the priority of a process. A process with a high “nice” value is “nicer” to other processes, and gets a low priority. Similarly, processes with a lower “nice” gets higher priority. The default “nice” value can be changed. The time spent on executing processes with a manually set “nice” appear as the ni value.
This is followed by id, which is the time the CPU remains idle. Most operating systems put the CPU on a power saving mode when it is idle. Next comes the wa value, which is the time the CPU spends waiting for I/O to complete.
Interrupts are signals to the processor about an event that requires immediate attention. Hardware interrupts are typically used by peripherals to tell the system about events, such as a keypress on a keyboard. On the other hand, software interrupts are generated due to specific instructions executed on the processor. In either case, the OS handles them, and the time spent on handling hardware and software interrupts are given by hi and si respectively.
In a virtualized environment, a part of the CPU resources are given to each virtual machine (VM). The OS detects when it has work to do, but it cannot perform them because the CPU is busy on some other VM. The amount of time lost in this way is the “steal” time, shown as st.
The avail Mem value is the amount of memory that can be allocated to processes without causing more swapping.
The Linux kernel also tries to reduce disk access times in various ways. It maintains a “disk cache” in RAM, where frequently used regions of the disk are stored. In addition, disk writes are stored to a “disk buffer”, and the kernel eventually writes them out to the disk. The total memory consumed by them is the buff/cache value.
The NI field shows the “nice” value of a process. The PR field shows the scheduling priority of the process from the perspective of the kernel. The nice value affects the priority of a process.
The PR (priority) column lists the kernel’s current schedule priority for the process. The higher the number, the less likely the kernel is to schedule the process if others need CPU time. The schedule priority alone doesn’t determine the kernel’s decision to give CPU time to a process, however, and the kernel may also change the priority during program execution according to the amount of CPU time the process consumes.
The priority column is the NI (nice value) column, which gives a hint to the kernel’s scheduler. This is what you care about when trying to influence the kernel’s decision. The kernel adds the nice value to the current priority to determine the next time slot for the process. When you set the nice value higher, you’re being “nicer” to other processes because the kernel prioritizes them.
By default, the nice value is 0. Now, say you’re running a big computation in the background that you don’t want to bog down your interactive session. To make that process take a back seat to other processes and run only when the other tasks have nothing to do, you can change the nice value to 20 with the renice command (where pid is the process ID of the process that you want to change)
These three fields are related with to memory consumption of the processes. VIRT (Virtual memory used) is the total amount of memory consumed by a process. This includes the program’s code, the data stored by the process in memory, as well as any regions of memory that have been swapped to the disk. RES (Resident memory) (that’s more or less how much memory a program needs) is the memory consumed by the process in RAM, and %MEM (The share of physical memory) expresses this value as a percentage of the total RAM available. Finally, SHR (Shared Memory) is the amount of memory shared with other processes.
As we have seen before, a process may be in various states. This field shows the process state in the single-letter form.
This is the total CPU time used by the process since it started, precise to the hundredths of a second.
If you want to kill a process, simply press k when top is running. This will bring up a prompt, which will ask for the process ID of the process and press enter.
Next, enter the signal using which the process should be killed. If you leave this blank, top uses a SIGTERM, which allows processes to terminate gracefully. If you want to kill a process forcefully, you can type in SIGKILL here. You can also type in the signal number here. For example, the number for SIGTERM is 15 and SIGKILL is 9.
One of the most frequent reasons to use a tool like top is to find out which process is consuming the most resources. You can press the following keys to sort the list:
Mto sort by memory usagePto sort by CPU usageNto sort by process IDTto sort by the running time
- 14:59:20
- the current time of day.
- up 6:30
- uptime. It is the amount of time since the machine was last booted.
- load average:
- refers to the number of processes that are waiting to run, that is, the number of processes that are in a runnable state and are sharing the cpu. Values less than 1.0 indicate that the machine is not busy.
- Cpu(s)
- this row describes the character of the activities that the CPU is performing.
- 0.7%us
- 0.7 percent of the CPU is being used for user processes. This means processes outside the kernel.
- 1.0%sy
- 1.0 percent of the CPU is being used for system (kernel) processes.
- 0.0%ni
- 0.0 percent of the CPU is being used by “nice” (low-priority) processes.
- 98.3%id
- 98.3 percent of the CPU is idle.
- 0.0%wa
- 0.0 percent of the CPU is waiting for I/O.
top - 14:59:20 up 6:30, 2 users, load average: 0.07, 0.02, 0.00
Tasks: 109 total, 1 running, 106 sleeping, 0 stopped, 2 zombie
Cpu(s): 0.7%us, 1.0%sy, 0.0%ni, 98.3%id, 0.0%wa, 0.0%hi, 0.0%si
Mem: 319496k total, 314860k used, 4636k free, 19392k buff
Swap: 875500k total, 149128k used, 726372k free, 114676k cach
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6244 me 39 19 31752 3124 2188 S 6.3 1.0 16:24.42 trackerd
11071 me 20 0 2304 1092 840 R 1.3 0.3 00:00.14 top
6180 me 20 0 2700 1100 772 S 0.7 0.3 00:03.66 dbus-dae
6321 me 20 0 20944 7248 6560 S 0.7 2.3 02:51.38 multiloa
4955 root 20 0 104m 9668 5776 S 0.3 3.0 02:19.39 Xorg
1 root 20 0 2976 528 476 S 0 0 0.2 00:03.14 init
2 root 15 -5 0 0 0 S 0.0 0.0 00:00.00 kthreadd
3 root RT -5 0 0 0 S 0.0 0.0 00:00.00 migratio
4 root 15 -5 0 0 0 S 0.0 0.0 00:00.72 ksoftirq
5 root RT -5 0 0 0 S 0.0 0.0 00:00.04 watchdog
6 root 15 -5 0 0 0 S 0.0 0.0 00:00.42 events/0
7 root 15 -5 0 0 0 S 0.0 0.0 00:00.06 khelper
41 root 15 -5 0 0 0 S 0.0 0.0 00:01.08 kblockd/
67 root 15 -5 0 0 0 S 0.0 0.0 00:00.00 kseriod
114 root 20 0 0 0 0 S 0.0 0.0 00:01.62 pdflush
116 root 15 -5 0 0 0 S 0.0 0.0 00:02.44 kswapd0
The STAT column is short for “state” and reveals the current status of the process.
- R (running)
- this means that the process is running or ready to run.
- S (sleeping)
- the process is not running; rather, it is waiting for an event, such as a keystroke or network packet.
- D (uninterruptible sleep)
- the process is waiting for I/O such as a disk drive.
- T (stopped)
- the process has been instructed to stop. More on this later in the chapter.
- Z (a defunct or “zombie” process)
- this is a child process that has terminated but has not been cleaned up by its parent.
- < (a high-priority process)
- it’s possible to grant more importance to a process, giving it more time on the CPU. This property of a process is called niceness. A process with high priority is said to be less nice because it’s taking more of the CPU’s time, which leaves less for everybody else.
- N (a low-priority process)
- a process with low priority (a “nice” process) will get processor time only after other processes with higher priority have been serviced.
GNU find searches the directory tree rooted at each given starting-point by evaluating the given expression from left to right, according to the rules of precedence, until the outcome is known (the left hand side is false for and operations, true for or), at which point find moves on to the next file name. If no starting-point is specified, ( . ) is assumed.
Informal syntax: find dir [ [atttributes logical-relationship] ... ] action
find allows actions to be performed based on the search results. There are a set of predefined actions and several ways to apply user-defined actions.
By default, find evaluates from left to right. That is, the logical relationship between the tests and actions determines which of them are performed, so the order of the tests and actions is important.
In the find command, there is, by default, an implied -and relationship between each test and action (that is, when no operator is present, -and is implied by default).
The operators -and, -or and -not can be abbreviated by -a, -o and !.
# These commands are the same.
find . -type f -name '*.bak' -print
find . -type f -a -name '*.bak' -a -print
find . -type f -and -name '*.bak' -and -print
Warning: use extreme caution when using the -delete action. Always test the command first by substituting the -print action for -delete to confirm the search results.
Dealing with Funny Filenames (-print0):
Unix-like systems allow embedded spaces (and even newlines!) in filenames. This causes problems for programs like xargs that construct argument lists for other programs. An embedded space will be treated as a delimiter, and the resulting command will interpret each space-separated word as a separate argument. To overcome this, find and xargs allow the optional use of a null character as an argument separator. The find command provides the action -print0, which produces null-separated output, and the xargs command has the --null (or -0) option, which accepts null separated input.
find . -iname '*.jpg' -print0 | xargs --null ls -l
# Using this technique, we can ensure that all files, even those
# containing embedded spaces in their names, are handled correctly.
Note: find does not produce results in sorted order. Its order is determined by the layout of the storage device.
Tests return a true or false value, usually on the basis of some property of a file we are considering. Adding the test -type limited the search to directories.
# List only directories.
find . -type d
| Type | Description |
|---|---|
| b | Block special device file |
| c | Character special device file |
| d | Directory |
| f | Regular file |
| l | Symbolic link |
| s | Socket |
We can also search by file size.
We add the -size test followed by the string +1M. The leading plus sign ( + ) indicates that we are looking for files larger than the specified number. A leading minus ( - ) sign would change the meaning of the string to be smaller than the specified number. Using no sign means, “match the value exactly.” The trailing letter M indicates that the unit of measurement is megabytes.
# Match the wildcard *.jpg and are larger than one megabyte.
find . -type f -name "*.jpg" -size +1M
| Char | Unit |
|---|---|
| b | 512-byte blocks (default) |
| c | Bytes |
| w | 2-byte words |
| k | Kilobytes (units of 1024 bytes) |
| M | Megabytes (units of 1048576 bytes) |
| G | Gigabytes (units of 1073741824 bytes) |
The cases where a numeric argument is required, the ( + ) mean “files larger than the specified number” and ( - ) “files smaller than the specified number” and no sign means, “match the value exactly”.
| Test | Match |
|---|---|
| -empty | empty fs and ds |
| -type c | fs of type c |
| -size n | fs of size n |
| -name pattern | fs a ds with the specified wildcard pattern |
| -iname pattern | like the -name test but case-insensitive |
| -perm mode | fs o ds that have permissions set to specified mode |
| -cmin n | fs o ds whose conts o atts last mod exactly n min |
| -cnewer file | fs o ds whose conts o atts last mod more recently |
| -ctime n | fs o ds whose conts o atts last mod n*24 hours |
| -mmin n | fs o ds whose conts were last mod n minutes ago |
| -mtime n | fs o ds whose conts were last mod n*24 hours ago |
| -newer file | fs a ds whose conts were mod more recently than f * |
| -inum n | f with inode number n (find hardlinks to inode) |
| -samefile name | similar -inum. fs share the same inode as file name |
| -user name | fs o ds belonging to user name |
| -nouser | f a ds that do not belong to a valid user * |
| -group name | f o ds belonging to group. |
| -nogroup | fs a ds that do not belong to a valid group |
| Abbreviation | Name |
| fs | files |
| fs o ds | files or directories |
| f o ds | file or directories |
| fs a ds | files and directories |
| f a ds | file and directories |
| conts | contents |
| conts o atts | contents or attributes |
| mod | modified |
| last mod | were last modified |
Notes: *
- This
-newer fileis useful when writing shell scripts that perform file backups. Each time you make a backup, update a file (such as a log) and then use find to determine which files have changed since the last update. - This
-nousercan be used to find files belonging to deleted accounts or to detect activity by attackers.
find provides a way to combine tests using logical operators to create more complex logical relationships.
# Looking for all the files with permissions that are not 0600 and the
# directories with permissions that are not 0700.
find . \( -type f -not -perm 0600 \) -o \( -type d -not -perm 0700 \)
| Opt | Match |
|---|---|
| -and | if the tests on both sides of the operator are true |
| -or | if a test on either side of the operator is true |
| -not | if the test following the operator is false |
| ( ) | Groups tests and operators to form larger expression |
( ) This is used to control the precedence of the logical evaluations. By default, find evaluates from left to right. It is often necessary to override the default evaluation order to obtain the desired result. Even if not needed, it is helpful sometimes to include the grouping characters to improve the readability of the command. Note that since the parentheses have special meaning to the shell, they must be quoted when using them on the command line to allow them to be passed as arguments to find. Usually the backslash character is used to escape them.
find allows actions to be performed based on the search results. There are a set of predefined actions and several ways to apply user-defined actions.
# Every file in dir and its subdirectories is searched and when they are
# found, they are deleted.
find . -type f -name '*.bak' -delete
# Could express this way to make the logical relationships easier to see.
find . -type f -a -name '*.bak' -a -print
Warning: Extreme caution when using the -delete action. Always test the command first by substituting the -print action for -delete to confirm the search results.
| Action | Description |
|---|---|
| -delete | delete the currently matching file |
| -ls | perform the equivalent of ls -dils on match file |
| output the full path of the matc file to stdout | |
| -quit | quit once a match has been made |
We can also invoke arbitrary commands. The traditional way of doing this is with the -exec action. This action works like this:
-exec command {} ;
Here command is the name of a command, {} is a symbolic representation of the current pathname, and the semicolon ( ; ) is a required delimiter indicating the end of the command. Note that the brace and semicolon characters have special meaning to the shell, so they must be quoted or escaped.
# In this example -exec act like the -delete action.
-exec rm '{}' ';'
# It’s possible to execute a user-defined action interactively. By using
# the -ok action in place of -exec
find . -type f -name 'foo*' -ok ls -l '{}' ';'
# < ls ... /home/me/bin/foo > ? y
# -rwxr-xr-x 1 me me 224 2007-10-29 18:44 /home/me/bin/foo
# < ls ... /home/me/foo.txt > ? y
# -rw-r--r-- 1 me me 0 2016-09-19 12:53 /home/me/foo.txt
Improving Efficiency:
When the -exec action is used, it launches a new instance of the specified command each time a matching file is found. There are times when we might prefer to combine all of the search results and launch a single instance of the command.
By changing the trailing semicolon character to a plus sign, we activate the ability of find to combine the results of the search into an argument list for a single execution of the desired command.
# This will execute ls each time a matching file is found.
find . -type f -name 'foo*' -exec ls -l '{}' ';'
# Lower performance since it launches a new instance of the specified
# command each time a matching file is found.
# Same results, but the system has to execute the ls command only once.
find . -type f -name 'foo*' -exec ls -l '{}' +
# Better performace since combine the results of the search into an
# argument list for a single execution of the command.
# Another approach with xargs command.
find . -type f -name 'foo*' -print | xargs ls -l
# The output of the find command piped into xargs, which, in turn,
# constructs an argument list for the ls command and then executes it.
Note: While the number of arguments that can be placed into a command line is quite large, it’s not unlimited. It is possible to create commands that are too long for the shell to accept. When a command line exceeds the maximum length supported by the system, xargs executes the specified command with the maximum number of arguments possible and then repeats this process until standard input is exhausted. To see the maximum size of the command line, execute xargs with the --show-limits option.
There is an important consideration to keep in mind when using regular expressions in find versus grep. Whereas grep will print a line when the line contains a string that matches an expression, find requires that the pathname exactly match the regular expression.
# Find every pathname that contains any character that is not a member
# of the set: [-_./0-9a-zA-Z]
# Such a scan would reveal pathnames that contain embedded spaces and
# other potentially offensive characters.
find . -regex '.*[^-_./0-9a-zA-Z].*'
We have the options. The options are used to control the scope of a find search. They may be included with other tests and actions when constructing find expressions.
| Option | Description |
|---|---|
| -depth | Direct find to process a directory’s files before |
| dir itself (default when the -delete is specified) | |
| -maxdepth | Set the max of levels that find will descend into |
| a directory tree when performing tests and actions | |
| -mindepth | Set the min of levels that find will descend into |
| a directory tree before applying tests and actions | |
| -mount | Direct find not to traverse directories that are |
| mounted on other file systems |
On computers, dates are usually formatted in YYYY-MM-DD order to make chronological sorting easy, but ours are in the American format of MM/DD/YYYY. How can we sort this list in chronological order?
Fortunately, sort provides a way. The key option allows specification of offsets within fields, so we can define keys within fields.
sort -k 3.7nbr -k 3.1nbr -k 3.4nbr file_dates
# Fedora 10 11/25/2008
# Ubuntu 8.10 10/30/2008
# SUSE 11.0 06/19/2008
# By specifying -k 3.7, we instruct sort to use a sort key that begins
# at the seventh character within the third field, which corresponds
# to the start of the year. Likewise, we specify -k 3.1 and -k 3.4 to
# isolate the month and day portions of the date. We also add the n
# and r options to achieve a reverse numeric sort. The b option is
# included to suppress the leading spaces (whose numbers vary from
# line to line, thereby affecting the outcome of the sort) in the date
# field.
sort --key=1,1 --key=2n file_dates
# Fedora 5 03/20/2006
# Fedora 6 10/24/2006
# Fedora 7 05/31/2007
# Though we used the long form of the option for clarity, -k 1,1 -k 2n
# would be exactly equivalent. In the first instance of the key
# option, we specified a range of fields to include in the first
# key. Since we wanted to limit the sort to just the first field, we
# specified 1,1 which means “start at field 1 and end at field 1.” In
# the second instance, we specified 2n, which means field 2 is the
# sort key and that the sort should be numeric. An option letter may
# be included at the end of a key specifier to indicate the type of
# sort to be performed.
sort -t ':' -k 7 /etc/passwd
# me:x:1001:1001:Myself,,,:/home/me:/bin/bash
# root:x:0:0:root:/root:/bin/bash
# dhcp:x:101:102::/nonexistent:/bin/false
# By specifying the colon character as the field separator, we can sort
# on the seventh field.
xargs reads items from the standard input, delimited by blanks (which can be protected with double or single quotes or a backslash) or newlines, and executes the command (default is /bin/echo) one or more times with any initial-arguments followed by items read from standard input. Blank lines on the standard input are ignored.
That is, xargs accepts input from standard input and converts it into an argument list for a specified command.
# These command are the same.
find . -type f -name 'foo*' -print | xargs ls -l
find . -type f -name 'foo*' -exec ls -l '{}' ';'
# Lower performance since it launches a new instance of the specified
# command each time a matching file is found.
find . -type f -name 'foo*' -exec ls -l '{}' +
# Better performace since combine the results of the search into an
# argument list for a single execution of the command.
Dealing with Funny Filenames (-print0):
Unix-like systems allow embedded spaces (and even newlines!) in filenames. This causes problems for programs like xargs that construct argument lists for other programs. An embedded space will be treated as a delimiter, and the resulting command will interpret each space-separated word as a separate argument. To overcome this, find and xargs allow the optional use of a null character as an argument separator. The find command provides the action -print0, which produces null-separated output, and the xargs command has the --null (or -0) option, which accepts null separated input.
find . -iname '*.jpg' -print0 | xargs --null ls -l
# Using this technique, we can ensure that all files, even those
# containing embedded spaces in their names, are handled correctly.
Note: While the number of arguments that can be placed into a command line is quite large, it’s not unlimited. It is possible to create commands that are too long for the shell to accept. When a command line exceeds the maximum length supported by the system, xargs executes the specified command with the maximum number of arguments possible and then repeats this process until standard input is exhausted. To see the maximum size of the command line, execute xargs with the --show-limits option.
In the default format, each group of changes is preceded by a change command in the form of range operation range to describe the positions and types of changes required to convert the first file to the second file.
In this format, a range is a comma-separated list of the starting line and the ending line.
| Change | Description |
|---|---|
| r1ar2 | Append the lines at the position r2 in the second |
| file to the position r1 in the first file | |
| r1cr2 | Change (replace) the lines at position r1 with the |
| lines at the position r2 in the second file | |
| r1dr2 | Delete the lines in the first file at position r1, which |
| would have appeared at range r2 in the second file |
cat file1 cat file2
# a b
# b d
# c d
# d e
diff file1 file2
# 1d0
# < a # < Means delete that line.
# 4a4
# > e # > Means add that line.
The unified format is specified with the -c option.
| Indic | Meaning |
|---|---|
| blank | A line shown for context. It doesn’t indicate |
| a difference between the two files | |
| - | A line deleted. This line will appear in the |
| first file but not in the second file | |
| + | A line added. This line will appear in the |
| second file but not in the first file | |
| ! | A line changed.The 2 version of line will be displayed |
| , each in its respective section of change group |
cat file1 cat file2
# a b
# b d
# c d
# d e
diff -c file1 file2
# *** file1.txt 2008-12-23 06:40:13.000000000 -0500
# --- file2.txt 2008-12-23 06:40:34.000000000 -0500
# ***************
# *** 1,4 **** # lines 1 through 4 in the first file
# - a
# b
# c
# d
# --- 1,4 ---- # lines 1 through 4 in the second file
# b
# c
# d
# + e
The unified format is specified with the -u option.
@@ -1,4 +1,4 @@ This indicates the lines in the first file and the lines in the second file described in the change group. Following this are the lines themselves, with the default three lines of context. Each line starts with one of three possible characters listed below.
| Char | Meaning |
|---|---|
| blank | this line is shared by both files |
| - | this line was removed from the first file |
| + | this line was added to the first file |
cat file1 cat file2
# a b
# b d
# c d
# d e
diff -c file1 file2
# --- file1.txt 2008-12-23 06:40:13.000000000 -0500
# +++ file2.txt 2008-12-23 06:40:34.000000000 -0500
# @@ -1,4 +1,4 @@
# -a
# b
# c
# d
# +e
cat a.txt cat b.txt
one one
two two
three THREE
four FOUR
five FIVE
six six
7 10
8 9
9 8
10 7
eleven eleven
twelve twelve
13 0
14 13
15 14
16 15
16
# diff show how to change the first file to match the second.
diff -u a.txt b.txt
--- a.txt 2023-09-28 12:32:42.998667401 -0400 # --- first file.
+++ b.txt 2023-09-28 12:32:41.958654590 -0400 # +++ second file.
@@ -1,15 +1,16 @@
# -1,15 show line 1 and the following 15 lines.
# +1,16 show line 1 and the following 16 lines.
# Show 3 lines before and after the change.
# - deleted lines + add lines.
# whitespace at the begining indicates the line doesn't change.
one
two
-three
-four
-five
+THREE
+FOUR
+FIVE
six
-7
-8
-9
10
+9
+8
+7
eleven
twelve
+0
13
14
15
In general, sed takes an address and an operation as one argument. The address is a set of lines (every line by default), and the command determines what to do with the lines. With no file arguments, sed reads from the standard input.
| Address | Description |
|---|---|
| n | A line number where n is a positive integer |
| $ | The last line |
| regexp | Lines matching a POSIX basic regexp |
| addr1,addr2 | A range of lines from addr1 to addr2, inclusive |
| addr1,+n | Match addr1 and the following n lines |
| addr! | Match all lines except addr |
| first~step | Match the line represented by the number first, |
| then each subsequent line at step intervals | |
| Command | Description |
| = | Output the current line number |
| a | Append text after the current line |
| i | Insert text in front of the current line |
| d | Delete the current line |
| p | Print the current line |
| q | Exit sed without processing any more lines |
| Q | Exit sed without processing any more lines |
| s/regex/rep/ | Sub cont of replacement wherever regexp is found |
| y/set1/set2 | Perform transliteration by converting characters |
| from set1 to the corresponding charactrs in set2 |
- p By default,
sedprints every line and only edits lines that match a specified address within the file. The default behavior can be overridden by specifying the-noption. - replacement may include the special character
&, which is equivalent to the text matched by regexp. In addition, replacement may include the sequences\1through\9, which are the contents of the corresponding subexpressions in regexp. After the trailing slash following replacement, an optional flag may be specified to modify the s command’s behavior.
Notes:
- when any command has extra slashes in its regular expression, this would confuse sed when it tries to interpret the command, so it must be applied the backslashes to escape the offending characters.
sed 's/([0-9]{2})/([0-9]{4})$\2-\1/' file
# Application of backslashes to escape the offending characters.
sed 's/\([0-9]\{2\}\)\/\([0-9]\{4\}\)$/\2-\1/' file
seddoes not support character ranges (for example,[a-z]), nor does it support POSIX character classes.
Some tips on how to exclude patterns:
- You can have as many
--excludeparameters as you like. - If you use the same patterns repeatedly, place them in a plaintext file (one pattern per line) and use
--exclude-from=file. - To exclude directories named item but include files with this name, use a trailing slash:
--exclude=item/. - The exclude pattern is based on a full file or directory name component and may contain simple globs (wildcards). For example,
t*smatchesthis, but it does not matchethers. - If you exclude a directory or filename but find that your pattern is too restrictive, use
--includeto specifically include another file or directory.
curl is a tool to transfer data from or to a server, using one of the supported protocols (DICT, FILE, FTP, FTPS, GO‐ PHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP). The command is designed to work without user interaction.
curl --trace-ascii trace_file http://www.example.org/
# curl attempts to establish the TCP connection to the server (transport
# layer orbelow).
== Info: Trying 93.184.216.34...
== Info: TCP_NODELAY set
== Info: Connected to www.example.org (93.184.216.34) port 80 (#0)
# If connection succeeds, curl then tries to send the request (header);
# this is where the application layer starts.
=> Send header, 79 bytes (0x4f) # curl debugging output.
# Next lines show what curl sends to the server.
# The hexadecimal numbers adds help to keep track of how much data was
# sent or received.
0000: GET / HTTP/1.1 # Requests the target resource.
0010: Host: www.example.org
0027: User-Agent: curl/7.58.0
0040: Accept: */*
004d: # empty line
# Next, the server sends a reply, first with its own header.
<= Recv header, 17 bytes (0x11) # debugging output.
# In curl , the header won’t count toward the offset; that’s why all of
# these lines begin with 0000.
0000: HTTP/1.1 200 OK
<= Recv header, 22 bytes (0x16)
0000: Accept-Ranges: bytes
<= Recv header, 12 bytes (0xc)
0000: Age: 17629
# Headers and the actual requested document.
<= Recv header, 22 bytes (0x16)
0000: Content-Length: 1256
<= Recv header, 2 bytes (0x2)
0000: # blank line means end of headers HTTP
<= Recv data, 1256 bytes (0x4e8)
0000: <!doctype html>.<html>.<head>. <title>Example Domain</title>.
0040: . <meta charset="utf-8" />. <meta http-equiv="Content-type
Administration tool for IPv4/IPv6 packet filtering and NAT.
Each firewall chain has a default policy that specifies what to do with a packet if no rule matches the packet.
ACCEPT -> kernel allows the packet to pass through the packet-filtering system.
DROP -> tells the kernel to discard the packet.
The kernel reads the chain from top to bottom, using the first rule that matches. When a rule matches, the kernel carries out the action and looks no further down in the chain.
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
# iptables -A INPUT -s 192.168.34.63 -j DROP
Drop packets from 192.168.34.63 host.
# If someone at 192.168.34.63 is annoying you this command prevent them
# from talking to machine.
iptables -A INPUT -s 192.168.34.63 -j DROP
# Chain INPUT (policy ACCEPT)
# target prot opt source destination
# DROP all -- 192.168.34.63 anywhere
# The 192.168.34.63 has told everyone on his subnet to open connections
# to your SMTP port (TCP port 25). To get rid of that traffic, run:
iptables -A INPUT -s 192.168.34.0/24 -p tcp --destination-port 25 -j DROP
# Chain INPUT (policy ACCEPT)
# target prot opt source destination
# DROP all -- 192.168.34.63 anywhere
# DROP tcp -- 192.168.34.0/24 anywhere tcp dpt:smtp
# Someone at 192.168.34.37 saying that she can’t send you email because
# you blockedher machine. Insert that rule at the top of the chain:
iptables -I INPUT -s 192.168.34.37 -j ACCEPT
# Insert a rule in 4 position:
iptables -I INPUT 4 -s 192.168.34.37 -j ACCEPT
The general format of a tcp protocol line is:
src > dst: flags data-seqno ack window urgent options
Src and dst are the source and destination IP addresses and ports. Flags are some combination of S (SYN), F (FIN), P (PUSH) or R (RST) or a single `.’ (no flags). Data-seqno describes the portion of sequence space covered by the data in this packet (see example below). Ack is sequence number of the next data expected the other direction on this connection. Window is the number of bytes of receive buffer space available the other direction on this connection. Urg indicates there is `urgent’ data in the packet. Options are tcp options enclosed in angle brackets (e.g., <mss 1024>).
Src, dst and flags are always present. The other fields depend on the contents of the packet’s tcp protocol header and are output only if appropriate.
Here is the opening portion of an rlogin from host rtsg to host csam.
rtsg.1023 > csam.login: S 768512:768512(0) win 4096 <mss 1024>
csam.login > rtsg.1023: S 947648:947648(0) ack 768513 win 4096 <mss 1024>
rtsg.1023 > csam.login: . ack 1 win 4096
rtsg.1023 > csam.login: P 1:2(1) ack 1 win 4096
csam.login > rtsg.1023: . ack 2 win 4096
rtsg.1023 > csam.login: P 2:21(19) ack 1 win 4096
csam.login > rtsg.1023: P 1:2(1) ack 21 win 4077
csam.login > rtsg.1023: P 2:3(1) ack 21 win 4077 urg 1
csam.login > rtsg.1023: P 3:4(1) ack 21 win 4077 urg 1
The first line says that tcp port 1023 on rtsg sent a packet to port login on csam. The S indicates that the SYN flag was set. The packet sequence number was 768512 and it contained no data. (The notation is `first:last(nbytes)’ which means `sequence numbers first up to but not including last which is nbytes bytes of user data’.) There was no piggy-backed ack, the available receive window was 4096 bytes and there was a max-segment-size option requesting an mss of 1024 bytes.
Csam replies with a similar packet except it includes a piggy-backed ack for rtsg’s SYN. Rtsg then acks csam’s SYN. The `.’ means no flags were set. The packet contained no data so there is no data sequence number. Note that the ack sequence number is a small integer (1). The first time tcpdump sees a tcp `conversation’, it prints the sequence number from the packet. On subsequent packets of the conversation, the difference between the current packet’s sequence number and this initial sequence number is printed. This means that sequence numbers after the first can be interpreted as relative byte positions in the conversation’s data stream (with the first data byte each direction being `1’). `-S’ will override this feature, causing the original sequence numbers to be output.
On the 6th line, rtsg sends csam 19 bytes of data (bytes 2 through 20 in the rtsg -> csam side of the conversation). The PUSH flag is set in the packet. On the 7th line, csam says it’s received data sent by rtsg up to but not including byte 21. Most of this data is apparently sitting in the socket buffer since csam’s receive window has gotten 19 bytes smaller. Csam also sends one byte of data to rtsg in this packet. On the 8th and 9th lines, csam sends two bytes of urgent, pushed data to rtsg.
UDP format is illustrated by this rwho packet:
actinide.who > broadcast.who: udp 84
This says that port who on host actinide sent a udp datagram to port who on host broadcast, the Internet broadcast address. The packet contained 84 bytes of user data.
Some UDP services are recognized (from the source or destination port number) and the higher level protocol information printed. In particular, Domain Name service requests (RFC-1034/1035) and Sun RPC calls (RFC-1050) to NFS.
“Traceroute” is a network debugging utility that attempts to trace the path a packet takes through the network - its route. A key word here is “attempts” - by no means does traceroute work in all cases.
If you’ve been paying attention, you already know that the only facilities TCP/IP provide for tracing packet routes are IP packet options (record route and its variants) that are poorly specified, rarely implemented in a useful way, and often disabled for security reasons. Traceroute does not depend on any of these facilities. Traceroute, to put it simply, is a hack.
How Traceroute Works: Traceroute transmits packets with small TTL values. Recall that the TTL (Time To Live) is an IP header field that is designed to prevent packets from running in loops. Every router that handles a packet subtracts one from the packet’s TTL. If the TTL reaches zero, the packet has expired and is discarded. Traceroute depends on the common router practice of sending an ICMP Time Exceeded message, documented in RFC 792, back to the sender when this occurs. By using small TTL values which quickly expire, traceroute causes routers along a packet’s normal delivery path to generate these ICMP messages which identify the router. A TTL value of one should produce a message from the first router; a TTL value of two generates a message from the second; etc.
+--------+ +--------+
| SENDER | | TARGET |
+--------+ +--------+
| ^|
[============( Router )=====( Router )=====( Router )==|====]
^ ^ ^ |
| TTL=1 | TTL=2 | TTL=3 | TTL=4
Traceroute | | | |
shows these -----+--------------+--------------+------------/
IP addresses
In a typical traceroute session, a group of packets with TTL=1 are sent. A single router should respond, using the IP address of the interface it transmits the ICMP Timeout messages on, which should be the same as the interface it received the original packets on. The user is told this IP address, and DNS is used to convert this into a symbolic domain address. Also, round trip times are reported for each packet in the group. Traceroute reports any additional ICMP messages (such as destination unreachables) using a rather cryptic syntax - !N means network unreachable, !H means host unreachable, etc. Once this first group of packets has been processed (this can take 10 seconds or no time at all), the second group (TTL=2) begins transmitting, and the whole process repeats.
- On the bar “Apply a display filter..”
- Protocol name (for example: dns)
- On the bar “Apply a display filter..”
- ip.addr == 192.168.1.1
- On the bar “Apply a display filter..”
- ip.src == 192.168.1.1
- On the bar “Apply a display filter..”
- ip.dst == 192.168.1.1
- Clicking on the plus button to add a new display filter (+ sign at the end of the bar “Apply a display filter..”)
- ip.addr == 192.168.1.1
- Clicking on the plus button to add a new display filter (+ sign at the end of the bar “Apply a display filter..”)
- ip.src == 192.168.1.1
- Clicking on the plus button to add a new display filter (+ sign at the end of the bar “Apply a display filter..”)
- ip.dst == 192.168.1.1
https://www.wireshark.org/docs/wsug_html_chunked/ChWorkBuildDisplayFilterSection.html
- How Linux Works by Brian Ward.
- The Linux Command Line by William Shotts.
- Debian Reference.