Skip to content
Snippets Groups Projects
linux_admin.rst 16.62 KiB

Linux administration

Introduction

What is Unix?

  • Unix, Linux, *nix, …. : What is it?
  • Operating System
  • Created in 1969 in the Bell Labs
  • Multi-user
  • Multi-task
  • Protected memory
  • Unix family: BSD, Linux, Mac OS X (Darwin), Android, etc.

Why use Unix on the computing servers?

  • Fares well in the scientific domain
  • Multi-user
  • Versatile
  • Powerful

What is the Unix shell?

  • a shell is a user interface to access an operating system's services.
  • a shell can be a command line interface or a graphical user interface
  • in a command line interface, the user issues commands to the program in the form of successive lines of text (command lines).
  • what is commonly referred as "Unix shell" is a set of commonly-used CLI programs.

Commands

  • A command is an instruction you send to the shell to ask it to execute an action.

Syntax: command, options, arguments

  • the command: what action should be executed

  • the arguments: the object of the action

  • the options: flags that specify or modify the behaviour of the command

    • options are usually preceded by

      • - if its a one letter option
      • -- if its a word option
      command <options> <aguments>
      
      ls -a
      ls --all

On line Help

Get help

  • man command or program displays the on-line manual pages.
  • info command or program Read documentation.
  • help command or program help retrieve information about bash commands only.

Man vs info vs help

  • man pages are the UNIX traditional way of distributing documentation about programs.
  • In the early '90s, the GNU project decided that "man" documentation system was outdated, and wrote the info command to replace it.
  • help is a bash command. It uses internal bash structures to store and retrieve information about bash commands.

Set the date and time

Setting the time correctly is important, a lot of tools use time to know if they have to do something ... We will set the time of the machine to the UTC time. Then the correction to the localtime will be perform by Linux.

To set the time:

  1. restart your PC and press escape key during boot phase
  2. select enter set up
  3. set the time
  4. save and exit

Sudoers

In most distros, there is a special user named root which have the rights to do anything. This role is dangerous so to avoid mistakes it is reserved to special actions and we should take care of what we do when we are logged as root.

Debian/Ubuntu have no root user anymore, but some users are granted for some special operations that need root privileges. The first user you created during the installation phase is sudoer and is granted for all root privilege for all operation. To be granted and execute a command with the root privilege, you need to precede your command by the keyword sudo (super user do) The first time you execute a command as sudo you will ask to confirm your password. Your password is available for one shell, so if you execute a second command in an other terminal you will ask again for your password.

sudo useradd ...

Allow one user to be sudoers just add it in group sudo.

sudo usermod -a -G sudo <username>

This command a <username> in the secondary group sudo.

To fine tune the sudoers permission you have to edit the sudoers configuration file with the command.

sudo visudo

User and Group management

Create a new user

The command is useradd and the syntax is useradd [options] username

sudo useradd -u <UID> -g <primary group>  -G <group1,group2,...>  -m   --shell /bin/bash <username>

will create a user with

  • a specific uid -u
  • a specific primary group -g
  • belonging to the secondary groups -G (groups are separated by comma without space surrounding comma)
  • create a home directory -m (/home/<username>)
  • with a specific shell --shell

But if you can let the system choose the uid, primary group, ... So the command line could be simpler.

sudo useradd -m --shell /bin/bash <username>

Will create a user with a home and bash as shell

Only root can perform this command.

Delete a user

userdel [OPTIONS] USERNAME
userdel username

The command above remove the user <username> from the system. But does not remove it's home directory To do this use the -r option

userdel -r username

Warning

Before to remove a user from the system ensure that he has no job running. Kill all jobs run by user

sudo killall -u username

Also ensure that he is not logged or force to logout

sudo userdel -f username

of course you can use -r and -f on same command.

sudo userdel -rf username

Create new group

sudo groupadd <name of group>

Only root can perform this command.

To take into account new user and group you need to logout and login again.

Delete a Group

sudo groupdel groupname

Add an Existing User to a Group

sudo usermod -a -G <group> <login>

for instance to add the user zeus in group viro

sudo usermod -a -G viro zeus

Always use the -a (append) option when adding a user to a new group. If you omit the -a option, the user will be removed from any groups not listed after the -G option.

On success, the usermod command does not display any output. It warns you only if the user or group doesn’t exist.

sudo usermod -a -G group1,group2 username

Add username to group1 and group2 in one command

Remove a User From a Group

sudo gpasswd -d username groupname

Check you identity

whoami

To know your login

id

to know your

  • login/uid
  • primary group/gid
  • all secondaries groups the user belongs to
$ id
uid=1000(cronos) gid=10010(cronos) groups=1001(cronos),4(adm),24(cdrom),27(sudo),48(docker),100(users), ...
$ id cronos
uid=1000(cronos) gid=10010(cronos) groups=1001(cronos),4(adm),24(cdrom),27(sudo),48(docker),100(users), ...

Change password

To change your password

The command to use is passwd

passwd
Changing password for cronos.
Current password:
New password:
Retype new password:
passwd: password updated successfully

To change the password of another user

Only root/sudoer can change the password for another user.

sudo passwd <login>

For instance cronos which is sudoer modify the password of user zeus (which have a local login)

sudo passwd zeus
[sudo] password for cronos:
New password:
Retype new password:
passwd: password updated successfully

Note

When sudo/root change the password for another user, the system does not ask for the user password.

Access to external hard drive

All files accessible in a Unix system are arranged in one big tree, the file hierarchy, rooted at /. These files can be spread out over several devices.

Note

A quick note on the filesystem: it’s a hierarchy of directories that systems use to organize file(s) and folder(s) on a storage media. Filesystem exists in every single storage solution: USB flash drive, CD-ROM, HDD, SSD, and even floppy disks! In the case of UNIX/Linux and similar systems, the filesystem starts with the root directory (noted as “/”). Under “root”, all the other child filesystems exist.

The mount command serves to attach the file system found on some device to the big file tree.

When you insert usbkey in your computer, xubuntu detects it and trigger a mount command. Unfortunately it does not do the same for filesystem on a network. It will be the case to connect the PC to the mk1c sequencer.

../_static/install/mk1c_connection.*

mk1c connection schema

mk1c is connected to network as well your PC

So we will have to mount manually the filesystem of the sequencer on our machine to allow rampart to access to the data in realtime.

We saw that mount include a device on the our local file system. So we need to know:

  • which device
  • a location where to mount the device, we call it the mounting point. The mounting point is just a regular directory.
  • there are several filesystem formats ext4, vfat, ... . So we need to specify which protocol.
  • only root can mount a new device.
sudo mount -t <format> <device> <mounting point> <option>

In our case the mk1c exports its filesystem over network with the cifs protocol.

We create the mounting point (to do only once)

sudo mkdir -p /mnt/mk1c

To do each time you need to connect the Pc to the mk1c

sudo mount -t cifs //mk1c ip address/data  -ousername=<the user on mk1c>,uid=<your uid on the Pc>,gid=<your primary gid on the Pc> /mnt/mk1c

$ sudo mount -t cifs //192.168.0.10/data -ousername=minit,uid=1000,gid=1000 /mnt/mk1c
passowrd ******

ls -l /mnt/mk1c

Note

If you use WSL (Windows Subsystem for Linux) the fylesystem type -t option to use is drvfs So the command to use to mount the mk1c on the Pc is

$ sudo mount -t drvfs //192.168.0.10/data -ousername=minit,uid=1000,gid=1000 /mnt/mk1c
passowrd ******

Note

In the 2 examples above the you need to know the IP of you mk1c and replace the IP example 192.168.0.10 by the mk1c IP

Conversely, the umount command will detach it again.

To disconnect the PC from the mk1c

sudo umount /mnt/mk1c

The storage system is not infinite

Any hard drive have a size. Sequencing generates data. You have to monitor the size used and free on the mk1c, otherwise you risk to have an error No space left on device and you will lose the run.

df

The df command stands for “disk filesystem“, it is used to get a full summary of available and used disk space usage of the file system on the Linux system.

df without option or argument display space usage for all mounted devices

$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
tmpfs            1559800     2016   1557784   1% /run
/dev/nvme0n1p5 284239328 20531532 249196232   8% /
tmpfs            7798984        0   7798984   0% /dev/shm
tmpfs               5120        4      5116   1% /run/lock
/dev/nvme0n1p1    262144    33432    228712  13% /boot/efi
tmpfs            1559796       92   1559704   1% /run/user/1000

The -h stand for humam readable displays the same information but the size is converted in Ko/Mo/Go.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.5G  2.1M  1.5G   1% /run
/dev/nvme0n1p5  272G   20G  238G   8% /
tmpfs           7.5G     0  7.5G   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
/dev/nvme0n1p1  256M   33M  224M  13% /boot/efi
tmpfs           1.5G   92K  1.5G   1% /run/user/1000

You can restrain the display to one device here / is for the root filesystem

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p5  272G   20G  238G   8% /

du

Conversely du shows you the space used by some files / directories.

One very useful command with du is with options:

  • -h for human readable
  • -s summary instead to show you the size of all files and recursively files in directories, it displays only the size of the level asked and displays the size used by the directory (not each files inside directories)
$ cd
$ du -sh *
30M     artic-ncov2019
30M     Desktop
4.0K    Documents
508K    Downloads
5.3G    miniconda3
59M     Miniconda3-py39_4.9.2-Linux-x86_64.sh
4.0K    Music
4.0K    Pictures
4.0K    Public
350M    snap
20K     Templates
4.0K    Videos

Installing package

Linux distinguishes from other operating system by packaging a lot of software and a package management system. Each linux distros have it's own packaging system:

  • centos/redhat have yum/dnf package management system and .rpm packages
  • debian/ubuntu have apt package management system with .deb packages
  • gentoo have portage package management system with .ebuild packages

The advantages of official packages is that the installation of these software is easy and well tested. The maintenance of these package is ensure by the packages manager. Lot of bioinformatics software are packages in these distros.

The inconvenient of these packages is that often the latest bleeding edge version of the software is not available. The package manager can handle only one version of each package at one time (for instance you cannot use blast+2.11 for one project and 2.0 for an other one). the packages are upgraded when you upgrade the system, so the version can change during a project lifetime. We will see alternatives.

search

To search if a software is packaged for your ubuntu use the apt command

apt search <soft name>

the low level command, to use if it's inside a script (dockerfile, ...)

apt-cache search <soft name>

install

To install a package, once you have the exact name of the package you want to install

sudo apt install <package name>
sudo apt-get install <package name>


.. note::

    notice that anybody can search if a package exists, but only users granted can install a new package.

Update system

To update the whole system (all packages), it can be perform via a widget "software updater", but I will describe below the command line way.

You need to update the list of package, then perform the upgrade of the packages

$ sudo apt update
$ apt list --upgradable
Listing... Done
apt/focal-updates 2.0.9 amd64 [upgradable from: 2.0.2ubuntu0.1]
base-files/focal-updates 11ubuntu5.6 amd64 [upgradable from: 11ubuntu5.2]
bash/focal-updates,focal-security 5.0-6ubuntu1.2 amd64 [upgradable from: 5.0-6ubuntu1.1]
bsdutils/focal-updates,focal-security 1:2.34-0.1ubuntu9.3 amd64 [upgradable from: 1:2.34-0.1ubuntu9.1]
dpkg/focal-updates,focal-security 1.19.7ubuntu3.2 amd64 [upgradable from: 1.19.7ubuntu3]
e2fsprogs/focal-updates,focal-security 1.45.5-2ubuntu1.1 amd64 [upgradable from: 1.45.5-2ubuntu1]
fdisk/focal-updates,focal-security 2.34-0.1ubuntu9.3 amd64 [upgradable from: 2.34-0.1ubuntu9.1]
gcc-10-base/focal-updates,focal-security 10.3.0-1ubuntu1~20.04 amd64 [upgradable from: 10.2.0-5ubuntu1~20.04]
gpgv/focal-updates,focal-security 2.2.19-3ubuntu2.2 amd64 [upgradable from: 2.2.19-3ubuntu2]
gzip/focal-updates,focal-security 1.10-0ubuntu4.1 amd64 [upgradable from: 1.10-0ubuntu4]
libapt-pkg6.0/focal-updates 2.0.9 amd64 [upgradable from: 2.0.2ubuntu0.1]
libblkid1/focal-updates,focal-security 2.34-0.1ubuntu9.3 amd64 [upgradable from: 2.34-0.1ubuntu9.1]
libc-bin/focal-updates 2.31-0ubuntu9.9 amd64 [upgradable from: 2.31-0ubuntu9.1]
libc6/focal-updates 2.31-0ubuntu9.9 amd64 [upgradable from: 2.31-0ubuntu9.1]
libcom-err2/focal-updates,focal-security 1.45.5-2ubuntu1.1 amd64 [upgradable from: 1.45.5-2ubuntu1]
libext2fs2/focal-updates,focal-security 1.45.5-2ubuntu1.1 amd64 [upgradable from: 1.45.5-2ubuntu1]
libfdisk1/focal-updates,focal-security 2.34-0.1ubuntu9.3 amd64 [upgradable from: 2.34-0.1ubuntu9.1]
libgcc-s1/focal-updates,focal-security 10.3.0-1ubuntu1~20.04 amd64 [upgradable from: 10.2.0-5ubuntu1~20.04]
...

$ sudo apt upgrade

Debian/Ubuntu is a distro based on releases, sometimes you need to switch to a new release. For this, you need to perform a dist-upgrade but only when a new release is available. You will be warned for new releases by the package manager after a "classical" update operation.

sudo apt dist-upgrade