-
Bertrand NÉRON authoredBertrand NÉRON authored
Linux administration
Introduction
What is Unix?
- Unix, Linux, *nix, …. : What is it?
- Operating System
- Created in 1969 in the Bell Labs
- Multi-user
- Multi-task
- Protected memory
- Unix family: BSD, Linux, Mac OS X (Darwin), Android, etc.
Why use Unix on the computing servers?
- Fares well in the scientific domain
- Multi-user
- Versatile
- Powerful
What is the Unix shell?
- a shell is a user interface to access an operating system's services.
- a shell can be a command line interface or a graphical user interface
- in a command line interface, the user issues commands to the program in the form of successive lines of text (command lines).
- what is commonly referred as "Unix shell" is a set of commonly-used CLI programs.
Commands
- A command is an instruction you send to the shell to ask it to execute an action.
Syntax: command, options, arguments
-
the command: what action should be executed
-
the arguments: the object of the action
-
the options: flags that specify or modify the behaviour of the command
-
options are usually preceded by
-
-
if its a one letter option -
--
if its a word option
command <options> <aguments> ls -a ls --all
-
-
On line Help
Get help
- man command or program displays the on-line manual pages.
- info command or program Read documentation.
- help command or program help retrieve information about bash commands only.
Man vs info vs help
- man pages are the UNIX traditional way of distributing documentation about programs.
- In the early '90s, the GNU project decided that "man" documentation system was outdated, and wrote the info command to replace it.
- help is a bash command. It uses internal bash structures to store and retrieve information about bash commands.
Set the date and time
Setting the time correctly is important, a lot of tools use time to know if they have to do something ... We will set the time of the machine to the UTC time. Then the correction to the localtime will be perform by Linux.
To set the time:
- restart your PC and press escape key during boot phase
- select enter set up
- set the time
- save and exit
Sudoers
In most distros, there is a special user named root which have the rights to do anything. This role is dangerous so to avoid mistakes it is reserved to special actions and we should take care of what we do when we are logged as root.
Debian/Ubuntu have no root user anymore, but some users are granted for some special operations that need root privileges. The first user you created during the installation phase is sudoer and is granted for all root privilege for all operation. To be granted and execute a command with the root privilege, you need to precede your command by the keyword sudo (super user do) The first time you execute a command as sudo you will ask to confirm your password. Your password is available for one shell, so if you execute a second command in an other terminal you will ask again for your password.
sudo useradd ...
Allow one user to be sudoers just add it in group sudo
.
sudo usermod -a -G sudo <username>
This command a <username> in the secondary group sudo
.
To fine tune the sudoers permission you have to edit the sudoers configuration file with the command.
sudo visudo
User and Group management
Create a new user
The command is useradd and the syntax is useradd [options] username
sudo useradd -u <UID> -g <primary group> -G <group1,group2,...> -m --shell /bin/bash <username>
will create a user with
- a specific uid -u
- a specific primary group -g
- belonging to the secondary groups -G (groups are separated by comma without space surrounding comma)
- create a home directory -m (/home/<username>)
- with a specific shell --shell
But if you can let the system choose the uid, primary group, ... So the command line could be simpler.
sudo useradd -m --shell /bin/bash <username>
Will create a user with a home and bash as shell
Only root can perform this command.
Delete a user
userdel [OPTIONS] USERNAME
userdel username
The command above remove the user <username> from the system. But does not remove it's home directory To do this use the -r option
userdel -r username
Warning
Before to remove a user from the system ensure that he has no job running. Kill all jobs run by user
sudo killall -u username
Also ensure that he is not logged or force to logout
sudo userdel -f username
of course you can use -r and -f on same command.
sudo userdel -rf username
Create new group
sudo groupadd <name of group>
Only root can perform this command.
To take into account new user and group you need to logout and login again.
Delete a Group
sudo groupdel groupname
Add an Existing User to a Group
sudo usermod -a -G <group> <login>
for instance to add the user zeus in group viro
sudo usermod -a -G viro zeus
Always use the -a (append) option when adding a user to a new group. If you omit the -a option, the user will be removed from any groups not listed after the -G option.
On success, the usermod command does not display any output. It warns you only if the user or group doesn’t exist.
sudo usermod -a -G group1,group2 username
Add username to group1 and group2 in one command
Remove a User From a Group
sudo gpasswd -d username groupname
Check you identity
whoami
To know your login
id
to know your
- login/uid
- primary group/gid
- all secondaries groups the user belongs to
$ id
uid=1000(cronos) gid=10010(cronos) groups=1001(cronos),4(adm),24(cdrom),27(sudo),48(docker),100(users), ...
$ id cronos
uid=1000(cronos) gid=10010(cronos) groups=1001(cronos),4(adm),24(cdrom),27(sudo),48(docker),100(users), ...
Change password
To change your password
The command to use is passwd
passwd
Changing password for cronos.
Current password:
New password:
Retype new password:
passwd: password updated successfully
To change the password of another user
Only root/sudoer can change the password for another user.
sudo passwd <login>
For instance cronos which is sudoer modify the password of user zeus (which have a local login)
sudo passwd zeus
[sudo] password for cronos:
New password:
Retype new password:
passwd: password updated successfully
Note
When sudo/root change the password for another user, the system does not ask for the user password.
Access to external hard drive
All files accessible in a Unix system are arranged in one big tree, the file hierarchy, rooted at /. These files can be spread out over several devices.
Note
A quick note on the filesystem: it’s a hierarchy of directories that systems use to organize file(s) and folder(s) on a storage media. Filesystem exists in every single storage solution: USB flash drive, CD-ROM, HDD, SSD, and even floppy disks! In the case of UNIX/Linux and similar systems, the filesystem starts with the root directory (noted as “/”). Under “root”, all the other child filesystems exist.
The mount command serves to attach the file system found on some device to the big file tree.
When you insert usbkey in your computer, xubuntu detects it and trigger a mount command. Unfortunately it does not do the same for filesystem on a network. It will be the case to connect the PC to the mk1c sequencer.
mk1c connection schema
So we will have to mount manually the filesystem of the sequencer on our machine to allow rampart
to access
to the data in realtime.
We saw that mount include a device on the our local file system. So we need to know:
- which device
- a location where to mount the device, we call it the mounting point. The mounting point is just a regular directory.
- there are several filesystem formats ext4, vfat, ... . So we need to specify which protocol.
- only root can mount a new device.
sudo mount -t <format> <device> <mounting point> <option>
In our case the mk1c exports its filesystem over network with the cifs protocol.
We create the mounting point (to do only once)
sudo mkdir -p /mnt/mk1c
To do each time you need to connect the Pc to the mk1c
sudo mount -t cifs //mk1c ip address/data -ousername=<the user on mk1c>,uid=<your uid on the Pc>,gid=<your primary gid on the Pc> /mnt/mk1c
$ sudo mount -t cifs //192.168.0.10/data -ousername=minit,uid=1000,gid=1000 /mnt/mk1c
passowrd ******
ls -l /mnt/mk1c
Note
If you use WSL (Windows Subsystem for Linux) the fylesystem type -t option to use is drvfs So the command to use to mount the mk1c on the Pc is
$ sudo mount -t drvfs //192.168.0.10/data -ousername=minit,uid=1000,gid=1000 /mnt/mk1c
passowrd ******
Note
In the 2 examples above the you need to know the IP of you mk1c and replace the IP example 192.168.0.10 by the mk1c IP
Conversely, the umount command will detach it again.
To disconnect the PC from the mk1c
sudo umount /mnt/mk1c
The storage system is not infinite
Any hard drive have a size.
Sequencing generates data.
You have to monitor the size used and free on the mk1c, otherwise you risk to have an error
No space left on device
and you will lose the run.
df
The df command stands for “disk filesystem“, it is used to get a full summary of available and used disk space usage of the file system on the Linux system.
df without option or argument display space usage for all mounted devices
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 1559800 2016 1557784 1% /run
/dev/nvme0n1p5 284239328 20531532 249196232 8% /
tmpfs 7798984 0 7798984 0% /dev/shm
tmpfs 5120 4 5116 1% /run/lock
/dev/nvme0n1p1 262144 33432 228712 13% /boot/efi
tmpfs 1559796 92 1559704 1% /run/user/1000
The -h stand for humam readable displays the same information but the size is converted in Ko/Mo/Go.
$ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 1.5G 2.1M 1.5G 1% /run
/dev/nvme0n1p5 272G 20G 238G 8% /
tmpfs 7.5G 0 7.5G 0% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
/dev/nvme0n1p1 256M 33M 224M 13% /boot/efi
tmpfs 1.5G 92K 1.5G 1% /run/user/1000
You can restrain the display to one device here / is for the root filesystem
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p5 272G 20G 238G 8% /
du
Conversely du shows you the space used by some files / directories.
One very useful command with du is with options:
- -h for human readable
- -s summary instead to show you the size of all files and recursively files in directories, it displays only the size of the level asked and displays the size used by the directory (not each files inside directories)
$ cd
$ du -sh *
30M artic-ncov2019
30M Desktop
4.0K Documents
508K Downloads
5.3G miniconda3
59M Miniconda3-py39_4.9.2-Linux-x86_64.sh
4.0K Music
4.0K Pictures
4.0K Public
350M snap
20K Templates
4.0K Videos
Installing package
Linux distinguishes from other operating system by packaging a lot of software and a package management system. Each linux distros have it's own packaging system:
- centos/redhat have yum/dnf package management system and .rpm packages
- debian/ubuntu have apt package management system with .deb packages
- gentoo have portage package management system with .ebuild packages
The advantages of official packages is that the installation of these software is easy and well tested. The maintenance of these package is ensure by the packages manager. Lot of bioinformatics software are packages in these distros.
The inconvenient of these packages is that often the latest bleeding edge version of the software is not available. The package manager can handle only one version of each package at one time (for instance you cannot use blast+2.11 for one project and 2.0 for an other one). the packages are upgraded when you upgrade the system, so the version can change during a project lifetime. We will see alternatives.
search
To search if a software is packaged for your ubuntu use the apt command
apt search <soft name>
the low level command, to use if it's inside a script (dockerfile, ...)
apt-cache search <soft name>
install
To install a package, once you have the exact name of the package you want to install
sudo apt install <package name>
sudo apt-get install <package name>
.. note::
notice that anybody can search if a package exists, but only users granted can install a new package.
Update system
To update the whole system (all packages), it can be perform via a widget "software updater", but I will describe below the command line way.
You need to update the list of package, then perform the upgrade of the packages
$ sudo apt update
$ apt list --upgradable
Listing... Done
apt/focal-updates 2.0.9 amd64 [upgradable from: 2.0.2ubuntu0.1]
base-files/focal-updates 11ubuntu5.6 amd64 [upgradable from: 11ubuntu5.2]
bash/focal-updates,focal-security 5.0-6ubuntu1.2 amd64 [upgradable from: 5.0-6ubuntu1.1]
bsdutils/focal-updates,focal-security 1:2.34-0.1ubuntu9.3 amd64 [upgradable from: 1:2.34-0.1ubuntu9.1]
dpkg/focal-updates,focal-security 1.19.7ubuntu3.2 amd64 [upgradable from: 1.19.7ubuntu3]
e2fsprogs/focal-updates,focal-security 1.45.5-2ubuntu1.1 amd64 [upgradable from: 1.45.5-2ubuntu1]
fdisk/focal-updates,focal-security 2.34-0.1ubuntu9.3 amd64 [upgradable from: 2.34-0.1ubuntu9.1]
gcc-10-base/focal-updates,focal-security 10.3.0-1ubuntu1~20.04 amd64 [upgradable from: 10.2.0-5ubuntu1~20.04]
gpgv/focal-updates,focal-security 2.2.19-3ubuntu2.2 amd64 [upgradable from: 2.2.19-3ubuntu2]
gzip/focal-updates,focal-security 1.10-0ubuntu4.1 amd64 [upgradable from: 1.10-0ubuntu4]
libapt-pkg6.0/focal-updates 2.0.9 amd64 [upgradable from: 2.0.2ubuntu0.1]
libblkid1/focal-updates,focal-security 2.34-0.1ubuntu9.3 amd64 [upgradable from: 2.34-0.1ubuntu9.1]
libc-bin/focal-updates 2.31-0ubuntu9.9 amd64 [upgradable from: 2.31-0ubuntu9.1]
libc6/focal-updates 2.31-0ubuntu9.9 amd64 [upgradable from: 2.31-0ubuntu9.1]
libcom-err2/focal-updates,focal-security 1.45.5-2ubuntu1.1 amd64 [upgradable from: 1.45.5-2ubuntu1]
libext2fs2/focal-updates,focal-security 1.45.5-2ubuntu1.1 amd64 [upgradable from: 1.45.5-2ubuntu1]
libfdisk1/focal-updates,focal-security 2.34-0.1ubuntu9.3 amd64 [upgradable from: 2.34-0.1ubuntu9.1]
libgcc-s1/focal-updates,focal-security 10.3.0-1ubuntu1~20.04 amd64 [upgradable from: 10.2.0-5ubuntu1~20.04]
...
$ sudo apt upgrade
Debian/Ubuntu is a distro based on releases, sometimes you need to switch to a new release. For this, you need to perform a dist-upgrade but only when a new release is available. You will be warned for new releases by the package manager after a "classical" update operation.
sudo apt dist-upgrade