{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# UNIX and Linux"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{epigraph}\n",
"The time has come,' the Walrus said,\n",
" To talk of many things:\n",
"Of shoes — and ships — and sealing-wax —\n",
" Of cabbages — and kings —\n",
"And why the sea is boiling hot —\n",
" And whether pigs have wings.'\n",
"\n",
"-- Lewis Carroll\n",
"\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Introduction \n",
"\n",
"Here we are going to learn about the UNIX operating system, and in particular, it's most popular user-friendly avatar, Linux (and even more specifically, Debian based Linux distributions, with focus on Ubuntu). If you are not a Unix/Linux user, you still need to learn the UNIX environment because under the hood, all other operating systems that need to do *serious* computing have to use Unix. Indeed, MacOS is built on UNIX, and Windows now a days has a UNIX sub-system to enable users to things that can only really be done in UNIX (like Genomics!). "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### What is UNIX?\n",
"\n",
"UNIX is a machine-independent operating system (OS) developed in the 1970s by AT&T programmers (notably Brian Kernighan and Dennis Ritchie, fathers of `C`) for programmers (you!). It is multi-user and network-oriented by design, uses plain text files for storing data (no proprietary file formats), and has a strictly hierarchical directory structure (more on this below). This makes it an ideal environment for developing your code and storing your data.\n",
"\n",
"Linux and Mac OS are Unix-like (or UN\\*X or *nix) operating systems that have evolved from UNIX. Ubuntu is a Linux distribution."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Why UNIX?\n",
"\n",
"* It was designed for developing code and storing data – an ideal native habitat for programming languages like Python and R\n",
"\n",
"* Robust, stable, secure (very few UNIX viruses and malware – I am yet to encounter one after almost 15 years!)\n",
"\n",
"* Its free and open-source\n",
"\n",
"* Scores of small programs available to perform simple tasks – can be combined easily\n",
"\n",
"* Easy to automate tasks (e.g., using shell scripts)\n",
"\n",
"* Multi-user (multiple users can log in and concurrently use computer)\n",
"\n",
"* Multi-tasking (can perform many tasks at the same time)\n",
"\n",
"* Network-ready (easy to communicate between computers)\n",
"\n",
"* Amazing support – a large body of tutorials and support web sites are readily available online\n",
"\n",
"* Basically all resources for High-Performance Computing (computer clusters, large workstations, etc.) run a UNIX or Linux operating system\n",
"\n",
"* **UNIX has been around since the early 1970's and will likely be around at the end of your career (the hard work you are putting into learning UNIX will pay off over a lifetime!)**"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"source": [
"```{note}\n",
"**32-bit vs 64-bit:** The terms 32-bit and 64-bit refer to the way a computer's processor (also called a CPU), handles information. A 64-bit OS handles large amounts of random access memory (RAM) more effectively than a 32-bit system. Specifically, while 32 bits of information can only access 4 GB of RAM, a 64-bit machine can access essentially unlimited system memory (though this is not yet physically possible)! The combination of your operating system, and processor bit count is often called the \"Platform\".\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## UNIX directory structure"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"The key UNIX directories are:\n",
"\n",
"| Directory | Description | \n",
"| :- | :- | \n",
"| `/` | Is the \"root\" directory | \n",
"| `/bin` | Contains basic programs | \n",
"| `/etc` | Contains configuration files | \n",
"| `/dev` | Contains files connecting to devices (keyboard, mouse, screen, etc.)| \n",
"| `/home` |Your home directory; this is where all your documents are and where you will usually work| \n",
"| `/tmp` | Contains temporary files| \n",
"\n",
"This hierarchical directory structure makes navigating your computer from the terminal/shell (coming up next!) or encoding this navigation in your computer programs easier."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"---\n",
":::{figure-md} UNIX-directory-structure\n",
"\n",
"\n",
"\n",
"**UNIX directory structure.** The UNIX directory (same as \"Folder\") structure is hierarchical, with a single tree starting from the \"root\" `/`. This is quite unlike Windows or MS-DOS, where there are separate trees for disk partitions, removable media, network, etc.
(Source: [https://pathanruet.files.wordpress.com/2012/05/unix-tree.png](https://pathanruet.files.wordpress.com/2012/05/unix-tree.png))\n",
":::\n",
"\n",
"---\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Meet the UNIX shell\n",
"\n",
"The shell (or terminal) is a text command processor that can run commands to interface with the Operating System's \"kernel\" – the core software that \"talks\" to the harware, including the CPU, RAM and hard drive. We will use the popular (yes, it's *very* popular in the new universe you are entering!) `bash` shell.\n",
"\n",
"```{note}\n",
"`bash` stand for the \"`B`ourne `A`gain `Sh`ell\", a pun (\"born again\" oh, us geeks!) on the name of the original Bourne shell (`sh`) that it replaces. \n",
"```\n",
"\n",
"$\\star$ To launch a `bash` shell, do `Ctrl + Alt + t` (or use ` Meta` key) – try it now. (On Mac: `Cmd+Space` to open spotlight search, and type \"terminal\" and hit return).\n",
"\n",
"```{warning}\n",
"In Mac OS, the default shell will likely be `zsh` shell, assuming you are using one of the newer versions of the operating system. Most of the commands will be the same between these shells, but some may differ (look out for tips about this, and also [read this](https://apple.stackexchange.com/questions/361870/what-are-the-practical-differences-between-bash-and-zsh)).\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"OK, so you have met the shell. Isn't it *lovely*? \n",
"\n",
"Let's look at key features of the bash shell/terminal: \n",
"\n",
"* The shell, by default, automatically starts in your home directory `/home/yourname/` \n",
"\n",
"* The start of the command prompt, where you will type your commands is indicated by a `:~$`)\n",
"\n",
"* The text before the `:~$` tells you what the name of the current user is (the part before the `@`), and the name of the computer (the part after the `@` upto the `:~$`).\n",
"\n",
"```{tip} \n",
"The path `/home/yourname/` can be replaced with shorthand `~` (important to remember!). So, for example, the command `cd /home/yourname/Documents` is the same as the command `cd ~/Documents`\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Some useful keyboard shortcuts that work in the bash terminal are:\n",
"\n",
"|Command | What it means |\n",
"| :- |:- | \n",
"| `Ctrl + A` | Go to the beginning of the line|\n",
"| `Ctrl + E` | Go to the end of the line|\n",
"| `Ctrl + L` | Clear the screen|\n",
"| `Ctrl + U` | Clear the line before cursor position|\n",
"| `Ctrl + K` | Clear the line after the cursor|\n",
"| `Ctrl + C` | Kill whatever you are running|\n",
"| `Ctrl + D` | Exit (close) the current shell|\n",
"| `Ctrl + right arrow`| Move cursor forward one whole word|\n",
"| `Ctrl + left arrow` | Move cursor backward one whole word|\n",
"\n",
"Also, note that:\n",
"\n",
"* Use the `Tab` key – very handy (try `ls` with a double `Tab` `Tab`).\n",
"\n",
"* You can navigate commands you previously typed using the up/down arrows"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Special Characters\n",
"\n",
"Certain \"special\" characters have a particular meaning (and use) to the UNIX shell. The forward slash `/` that separates parts of file/ directory paths, and the tilde `~` that serves as a shorthand for your `home` directory path are two examples that you have already seen.\n",
"\n",
"The following table gives the meanings of important (not all) special characters in shell command lines that you should know about (we will see more or different special characters as we go through the Shell scripting and Python chapters): \n",
"\n",
"| Special character | Meaning |\n",
"| :------------- |:-------------| \n",
"| `~` | Home directory path|\n",
"| `/` | Path names separator\t|\n",
"| `#` | Comment (anything after this will be ignored by the shell interpreter; in fact, this serves the same purpose across shell scripts, `R` and `Python`)|\n",
"| `$` | Variable, expression or command to follow this|\n",
"|`{` and `}`| Start and end of an expression or command block|\n",
"| `*` | Wildcard for strings; for example, `ls nam*` lists all files and and directories starting with `nam`|\n",
"| `?` | Single-character wildcard|\n",
"| `\\` | Escape character; any character following this will be quoted; for example, `#` indicates the start of a comment, but `\\#` indicates a literal `#`|\n",
"| `;` | Command separator; commands can be entered sequentially in a single command line separated by this|\n",
"| `|` | Pipe.|\n",
"| `>` | Redirect output to (any valid) target following this|\n",
"| `!` | Pipeline logical NOT|"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Basic UNIX commands\n",
"\n",
"Here are some basic commands, many of which you will use frequently. \n",
"\n",
"| Command | What it does |\n",
"| :------------- |:-------------| \n",
"| `man COMMAND` | Show help page of a command.|\n",
"| `whoami` | Display your user-name.| \n",
"| `pwd`| Show the current directory.| \n",
"| `ls`| List the files in the directory.| \n",
"| `cd DIRNAME`| Change directory.| \n",
"| `cd ..` | Move one directory up.| \n",
"| `cd /`| Go to the root directory.| \n",
"| `cd ~`or just `cd ` | Go to the home directory.| \n",
"| `cp FROM TO`| Copy a file, or a directory non-recursively (what's this? Google it!).| \n",
"| `cp -r FROM TO`| Copy a directory recursively (what's this? Google it!).| \n",
"| `mv FROM TO` | Move or rename a file or directory.| \n",
"| `touch FILENAME`| Create an empty file.| \n",
"| `echo \"My string\"`| Print a string (in this example, \"My string\").| \n",
"| `rm FILEorDIRNAME` | Remove a file or directory non-recursively.| \n",
"| `rm -r DIRNAME` | Remove a directory recursively.| \n",
"| `wc FILENAME`| Count the number of lines and words in a file.|\n",
"| `sort FILENAME` | Sort the lines of a file and print result.|\n",
"| `uniq ` | Shows only unique elements of a list.|\n",
"| `cat FILENAME` | Print the file on the screen.|\n",
"| `less FILENAME` | Progressively print a file on the screen (\"q\" to exit).|\n",
"| `head FILENAME` | Print the first few lines of a file.|\n",
"| `tail FILENAME` | Print the last few lines of a file.|\n",
"| `history` | Show the last commands you typed.|\n",
"| `date` | Print current date.|\n",
"| `file FILENAME` | Determine the type of a file.|\n",
"| `passwd` | Change user password.|\n",
"| `chmod FILENAME`| Change file permissions.|"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exploring files and directories \n",
"\n",
"A fundamental command that you will probably use the most frequently of all is `ls` (this stands for *list* files and directories in current directory).\n",
"\n",
"$\\star$ Type the `ls` command in at the command prompt (which will have opened in your `home` directory) and hit Enter. \n",
"\n",
"You will see a number of directories and maybe some files in the output (this is currently what you have in your user account's `home`).\n",
"\n",
"$\\star$ Next try `ls /`\n",
"\n",
"This will list what you have in your root, including the directories listed in the table [above](#UNIX-directory-structure) . \n",
"\n",
"\n",
"$\\star$ The `ls` command by default does not list hidden directories and files. To see these, use\n",
"\n",
"```bash\n",
"ls -a\n",
"```\n",
"This will list all files and directories that start with a `.`, such as `.config` (this contains sub-directories and files containing the configuration settings for applications).\n",
"\n",
"$\\star$ To see more info on the files and directories, use \n",
"\n",
"```bash\n",
"ls -l\n",
"```\n",
"\n",
"And if you want to see detailed info on ALL files and directories including hidden ones, combine the `-a` and `-l` flags:\n",
"\n",
"```bash\n",
"ls -al\n",
"```\n",
"\n",
"An example line in the output of `ls -l` is:\n",
"\n",
"`drwxr-xr-x 9 mhasoba mhasoba 4096 Oct 11 10:45 Desktop`\n",
"\n",
"This is what all that text preceding the folder name (in his case, `Desktop`) means (you will get similar info for files):\n",
"\n",
"* Item type and permissions (`drwxr-xr-x`)\n",
" * The first letter is `d` because this is a directory. This will most often be `d` (directory), `-` (file), or `l` (a symbolic link – a file that contains a reference to another file or directory, similar to a \"shortcut\" in Windows OS).\n",
" * Following this, there are three triplets of letters (`rwx`,`r-x`,`r-x`), with each letter in each of the three indicating read, write and execute permissions for the owner, the group, and anybody else. \n",
" * The first, `rwx`, indicates that the owner (`mhasoba`, see below) has read (`r`), write (`w`) and execute (`x`) permissions for the Desktop directory\n",
" * The second and third `r-x` mean that the group and everybody else have read and execute but not write permission (they can view the Desktop, but not modify anything in it) \n",
"\n",
"* Number of links or directories inside this directory (`9`) (for files this value would be 1)\n",
"\n",
"* Who owns it (user `mhasoba`),\n",
"\n",
"* The owner's group (also `mhasoba`; on this machine the root is the (single) user `mhasoba`),\n",
"\n",
"* File size in bytes (`4096`),\n",
"\n",
"* Time of last modification (`Oct 11 10:45`), and\n",
"\n",
"* File/directory name (`Desktop`)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Building your coursework directory structure\n",
"\n",
"It is time to start building your CMEE coursework directory structure. Please follow these rules:\n",
"\n",
"* Do all your work in a single directory with the abbreviation of your course followed by `CourseWork` (no spaces please) located in a suitable place in your `/home`. So for example, for CMEE students, the directory would be `CMEECourseWork`.\n",
"\n",
"* Each week's coursework should be in its respective directory; e.g., `CMEECourseWork/week1`, `CMEECourseWork/week2`, etc.\n",
"\n",
"* Each week's directory should contain directories called `code`, `data`, `results`, and `sandbox` (can be all small(like in these example names), sentence (first letter capitalized), or all large case, but *be consistent*)\n",
"\n",
"You will bring your `CMEECourseWork` and all it's contents under version control using Git (coming up in the [Version Control Chapter](03-Git.ipynb).\n",
"\n",
"$\\star$ OK, make your coursework directory now. First, `cd` to an appropriate place (e.g., `Documents` on your `home`):"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"cd ~/Documents # same as /home/YourUserName/Documents"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Check what you have in there with:"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"```bash\n",
"ls\n",
"```\n",
"\n",
"(You should see the contents of your `Documents` directory)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Now make the directory and `cd` to it"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"mkdir CMEECourseWork"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You will get an error if a directory by that name already exists. "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"cd CMEECourseWork"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can also combine commands using `&&`. For example,\n",
"\n",
"```bash \n",
"cd CMEECourseWork && cd CMEECourseWork\n",
"```\n",
"\n",
"Next, create your week1 sub-directory:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"mkdir week1"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"cd week1"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"mkdir sandbox code data results"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then, try to cd to it:\n",
"\n",
"```bash\n",
"cd Sandbox\n",
"```\n",
"You will get `bash: cd: Sandbox: No such file or directory` error, because in the UNIX/Linux file system, commands are case-sensitive. So \"`Sandbox`\" is not the same as \"`sandbox`\"."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{warning}\n",
"\n",
"**Mac OS has case insensitive commands**: Unlike UNIX/Linux, MacOS allows execution of commands and file/directory names in a case-insensitive way. Keep this in mind from the perspective of platform independence - if you write your code to call a directory called, say, `Sandbox`, but the directory is actually called `sandbox`, then you will not get an error, but your collaborator (me!) will, if she/he uses Linux!\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, try and remove `sandbox`:\n",
"\n",
"``` bash\n",
"rm sandbox\n",
"```\n",
"\n",
"You will get a `rm: cannot remove 'sandbox': Is a directory` error because UNIX is conservative about allowing users to delete directories (this is one of the reasons why it is so stable). To delete it, you will need to use the `-r` (for *recursive*) flag:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"rm -r sandbox"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Careful with the `-r` option, because starting with the target directory (`sandbox` in this case), it `r`ecursively removes the entire directory tree inside the target directory, files and all!\n",
"\n",
"\n",
"Now recreate your `sandbox`:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"mkdir sandbox"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's try renaming it:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"mv sandbox Sandbox"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[0m\u001b[01;34mcode\u001b[0m \u001b[01;34mdata\u001b[0m \u001b[01;34mresults\u001b[0m \u001b[01;34mSandbox\u001b[0m\n"
]
}
],
"source": [
"ls"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's move back to all small caps `sandbox` for naming consistency:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[0m\u001b[01;34mcode\u001b[0m \u001b[01;34mdata\u001b[0m \u001b[01;34mresults\u001b[0m \u001b[01;34msandbox\u001b[0m\n"
]
}
],
"source": [
"mv Sandbox sandbox\n",
"ls"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next try some more directory and file manipulations:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"cd sandbox"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Try `pwd` (`p`rint `w`orking `d`irectory) next to see the full path of the current directory. \n",
"\n",
"Let's continue:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"ls # nothing here yet"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note the hash mark (`#`) after the command above – anything after a `#` is ignored (so you can use it for commenting)."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"touch TestFile # OR, \"touch TestFile.txt\""
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TestFile\n"
]
}
],
"source": [
"ls"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TestFile2\n"
]
}
],
"source": [
"mv TestFile TestFile2; ls # commands can also be sequentially executed with a `;`"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"rm TestFile2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You could have made your project directories and subdirectories in one swoop by using the `-p` option of `mkdir` (`cd` back up to `CMEECourseWork` first) with the `-p` flag for `mkdir`:\n",
"\n",
"```bash\n",
"\n",
"mkdir -p Week1/{Data,Code,Results,Sandbox}\n",
"\n",
"```\n",
"\n",
"*How would you `cd` to `CMEECourseWork`?*\n",
"\n",
"There are two reasonable options:\n",
"\n",
"```bash\n",
"cd ../../../CMEECourseWork\n",
"```\n",
"\n",
"Or, \n",
"\n",
"```bash\n",
"cd /home/YourUserName/Documents/CMEECourseWork\n",
"```\n",
"Try them both. \n",
"\n",
"Which is the same as `cd ~/Documents/CMEECourseWork`\n",
"\n",
"*Which of these two would you choose and why?* "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Command arguments\n",
"\n",
"Most UNIX commands accept arguments that modify their behavior. E.g., `ls -l` (ls “minus”l) lists the files in longer format. Some useful arguments:\n",
"\n",
"| Command/Argument | What it does |\n",
"| :------------- |:-------------| \n",
"|`cp -r DIR1 DIR2` |Copy a directory recursively (i.e., including all the sub-directories and files).|\n",
"|`rm -i FILENAME` | Removes a file, but asks first (for safety).|\n",
"|`rm -r DIR` | Remove a directory recursively (i.e., including all the sub-directories and files).|\n",
"|`ls -a` | List all files, including hidden ones.|\n",
"| `ls -h` | List all files, with human-readable sizes (Mb, Gb).|\n",
"| `ls -l` | List all files, long format.|\n",
"| `ls -S` | List all files, order by size.|\n",
"| `ls -t` | List all files, order by modification time.|\n",
"| `ls -1` | List all files, one file per line.|\n",
"| `mkdir -p Dir1/Dir2/Dir3` | Create the directory Dir3 and Dir1 and Dir2 if they do not already exist.|\n",
"| `sort -n` | Sort all the lines, but use numeric values instead of dictionary (i.e., 11 follows 2).|\n",
"\n",
"You can also combine command arguments / flags. Try:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"cd ../../ # cd two directory levels higher, assuming you are currently in CMEECourseWork/Week1/sandbox"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
".:\n",
"\u001b[0m\u001b[01;34mweek1\u001b[0m\n",
"\n",
"./week1:\n",
"\u001b[01;34mcode\u001b[0m \u001b[01;34mdata\u001b[0m \u001b[01;34mresults\u001b[0m \u001b[01;34msandbox\u001b[0m\n",
"\n",
"./week1/code:\n",
"\n",
"./week1/data:\n",
"\n",
"./week1/results:\n",
"\n",
"./week1/sandbox:\n"
]
}
],
"source": [
"ls -R"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`-R` flag lists directory tree recursively (through all subdirectories)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
".:\n",
"total 4\n",
"drwxrwxr-x 6 mhasoba mhasoba 4096 Oct 5 12:00 \u001b[0m\u001b[01;34mweek1\u001b[0m\n",
"\n",
"./week1:\n",
"total 16\n",
"drwxrwxr-x 2 mhasoba mhasoba 4096 Oct 5 12:00 \u001b[01;34mcode\u001b[0m\n",
"drwxrwxr-x 2 mhasoba mhasoba 4096 Oct 5 12:00 \u001b[01;34mdata\u001b[0m\n",
"drwxrwxr-x 2 mhasoba mhasoba 4096 Oct 5 12:00 \u001b[01;34mresults\u001b[0m\n",
"drwxrwxr-x 2 mhasoba mhasoba 4096 Oct 5 12:00 \u001b[01;34msandbox\u001b[0m\n",
"\n",
"./week1/code:\n",
"total 0\n",
"\n",
"./week1/data:\n",
"total 0\n",
"\n",
"./week1/results:\n",
"total 0\n",
"\n",
"./week1/sandbox:\n",
"total 0\n"
]
}
],
"source": [
"ls -lR # same as ls -l -R"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The combines `-l` and `-R` to give you both the directory permissions info and when it was modfied while listing the directories and files recursively."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Redirection and pipes\n",
"\n",
"\n",
"Output of programs can also be \"redirected\" to a file in two ways:\n",
"\n",
"* `>`: Redirect output from a command to a file on disk. If the file already exists, it will be overwritten.\n",
"\n",
"* `>>`: Append the output from a command to a file on disk. If the file does not exist, it will be created.\n",
"\n",
"Let's try some examples.\n",
"\n",
"First make sure you are back in `sandbox`:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"cd week1/sandbox"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"My first line.\n"
]
}
],
"source": [
"echo \"My first line.\" > test.txt\n",
"cat test.txt"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"My first line.\n",
"My second line\n"
]
}
],
"source": [
"echo \"My second line\" >> test.txt\n",
"cat test.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now try \n",
"\n",
"```bash\n",
"ls / >> ListRootDir.txt\n",
"cat ListRootDir.txt\n",
"```\n",
"*What do you see?* Cool!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also concatenate commands using \"pipes\" with \"`|`\" e.g., to count how many files are in root (`/`) directory:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```bash \n",
"ls / | wc -l\n",
"```\n",
"\n",
"*How many is that?* "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Look up `man wc` in the terminal, or search \"unix wc\" online."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or try:"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 4\n",
"-rw-rw-r-- 1 mhasoba mhasoba 30 Oct 5 12:01 test.txt\n"
]
}
],
"source": [
"ls -lt | head -5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"*What does this do?*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Wildcards\n",
"\n",
"We can use wildcards to find files based on their names (still in `Week1/Sandbox`):"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Anotherfile.csv File1.csv File2.csv File3.csv File4.csv\n",
"Anotherfile.txt File1.txt File2.txt File3.txt File4.txt\n",
"10\n"
]
}
],
"source": [
"mkdir TestWild\n",
"cd TestWild\n",
"touch File1.txt\n",
"touch File2.txt\n",
"touch File3.txt\n",
"touch File4.txt\n",
"touch File1.csv\n",
"touch File2.csv\n",
"touch File3.csv\n",
"touch File4.csv\n",
"touch Anotherfile.csv\n",
"touch Anotherfile.txt\n",
"ls \n",
"ls | wc -l"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will use the following wildcards:\n",
"\n",
"| Wildcard | Meaning |\n",
"| :------------- |:-------------| \n",
"|`?`|Any single character, except a leading dot (hidden files).|\n",
"|`*`|Zero or more characters, except a leading dot (hidden files).|\n",
"|`[A-Z]`|Define a class of characters (e.g., upper-case letters).| \n",
"\n",
"Now let's try to find the files using wildcards:"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Anotherfile.csv File1.csv File2.csv File3.csv File4.csv\n",
"Anotherfile.txt File1.txt File2.txt File3.txt File4.txt\n",
"File1.csv File2.csv File3.csv File4.csv\n",
"File1.txt File2.txt File3.txt File4.txt\n",
"Anotherfile.txt File1.txt File2.txt File3.txt File4.txt\n",
"File1.txt File2.txt File3.txt File4.txt\n",
"File1.txt File2.txt\n",
"File1.csv File1.txt File2.csv File2.txt File4.csv File4.txt\n"
]
}
],
"source": [
"ls *\n",
"ls File*\n",
"ls *.txt\n",
"ls File?.txt\n",
"ls File[1-2].txt\n",
"ls File[!3].*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(Using-grep)=\n",
"## Using `grep`\n",
"\n",
"`grep` is a command that matches strings in a file (why is this useful?). It is based on regular expressions (more on this [later](Python_II:python-regex)). Let's explore some basic usage of `grep`. For a test file let's download a list of protected species from the web (to `sandbox`):"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2021-10-05 12:01:21-- https://raw.githubusercontent.com/mhasoba/TheMulQuaBio/master/content/data/spawannxs.txt\n",
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.109.133, ...\n",
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 13467 (13K) [text/plain]\n",
"Saving to: ‘spawannxs.txt’\n",
"\n",
"spawannxs.txt 100%[===================>] 13.15K --.-KB/s in 0.003s \n",
"\n",
"2021-10-05 12:01:21 (5.03 MB/s) - ‘spawannxs.txt’ saved [13467/13467]\n",
"\n"
]
}
],
"source": [
"cd ../ # move one level up from TestWild, back to sandbox\n",
"wget https://raw.githubusercontent.com/mhasoba/TheMulQuaBio/master/content/data/spawannxs.txt #Cool!"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"spawannxs.txt test.txt \u001b[0m\u001b[01;34mTestWild\u001b[0m\n"
]
}
],
"source": [
"ls"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [],
"source": [
"mv spawannxs.txt ../data # Move the file to the appropriate location (note the ../ - relative path)"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [],
"source": [
"cd ../data # Move to the data directory"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ANNEX I\n",
"List of Species of Marine and Coastal Flora \n",
"Protected Under Article 11(1)(a)\n",
"\n",
"TRACHAEOPHYTA (Vascular Plants)\n",
"\n",
"Family\t\tGenus\t\tSpecies\n",
"\n",
"Aquifoliaceae\tIlex\t\tcookii\n",
"Bignoniaceae\tCrescentia\tmirabilis\n",
"Bignoniaceae\tCrescentia\tportoricensis\n",
"Boraginaceae\tCordia\t\twagnerorum\n",
"Buxaceae\tBuxus\t\tvahlii\n",
"Cactaceae\tEchinocereus \treichenbachii var. albertii\n",
"Cactaceae\tHarrisia\tfragrans\n",
"Cactaceae\tHarrisia\tportoricensis\n",
"Cactaceae\tLeptocereus\tgrantianus\n",
"Cactaceae\tLeptocereus\twrightii\n",
"Cactaceae\tMelocactus\tguitartii\n",
"Cactaceae\tMelocactus \tharlowii sensu lato\n",
"Cactaceae\tPilosocereus\tdeeringii\n",
"Cactaceae\tPilosocereus\trobinii\n",
"Convolvulaceae\tBonamia\t\tgrandiflora\n",
"Convolvulaceae\tIpomoea\t\tflavopurpurea\n",
"Convolvulaceae\tIpomoea\t\twalpersiana\n",
"Cyatheaceae\tCyathea\t\tdryopteroides\n",
"Cyperaceae\tRhynchospora\tbucherorum\n",
"Dioscoreaceae\tRajania\t\ttheresensis\n",
"Ericaceae\tRhododendron\tchapmanii\n",
"Euphorbiaceae\tAndrachne\tbrittonii\n",
"Euphorbiaceae\tBernardia\tvenosa\n",
"Euphorbiaceae\tCnidoscolus\tfragrans\n",
"Euphorbiaceae\tDrypetes\ttriplinervia\n",
"Flacourtiaceae\tBanaras\t\tvanderbiltii\n",
"Flacourtiaceae\tSamyda\t\tmicrophylla\n",
"Hydrophyllaceae\tHydrolea\ttorroei\n",
"Icacinaceae\tOttoschulzia \trhodoxylon\n",
"Leguminosae\tAcacia\t\tcupeyensis\n",
"(Fabaceae)\n",
"Leguminosae\tAcacia\t\troigii\n",
"Leguminosae\tStahlia\t\tmonosperma\n",
"Liliaceae\tHarperocallis\tflava\n",
"Loranthaceae\tDendropemon\tacutifolius\n",
"Malvaceae\tAbutilon\tvirginianum\n",
"Meliaceae\tTrichilia\ttriacantha\n",
"Olacaceae\tXimenia\t\troigii\n",
"Orchidaceae\tBrachionidium\tciliolatum\n",
"Orchidaceae\tCranichis\tricartii\n",
"Orchidaceae\tLapanthes\teltoroensis\n",
"Orchidaceae\tOncidium\tjacquinianum\n"
]
}
],
"source": [
"head -n 50 spawannxs.txt #You will see \"head\" in R and Python as well"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"OK, let's look for falcons:"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[01;31m\u001b[KFalco\u001b[m\u001b[Knidae\t\u001b[01;31m\u001b[KFalco\u001b[m\u001b[K\t\tfemoralis septentrionalis\n",
"\u001b[01;31m\u001b[KFalco\u001b[m\u001b[Knidae\t\u001b[01;31m\u001b[KFalco\u001b[m\u001b[K\t\tperegrinus\n",
"\u001b[01;31m\u001b[KFalco\u001b[m\u001b[Knidae\tPolyborus\tplancus\n",
"\u001b[01;31m\u001b[KFalco\u001b[m\u001b[Knidae\t\u001b[01;31m\u001b[KFalco\u001b[m\u001b[K\t\tcolumbarius\n"
]
}
],
"source": [
"grep Falco spawannxs.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Nothing there? Let's use `-i` to make the matching case-insensitive:"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Order:\t\u001b[01;31m\u001b[KFALCO\u001b[m\u001b[KNIFORMES\n",
"\u001b[01;31m\u001b[KFalco\u001b[m\u001b[Knidae\t\u001b[01;31m\u001b[KFalco\u001b[m\u001b[K\t\tfemoralis septentrionalis\n",
"\u001b[01;31m\u001b[KFalco\u001b[m\u001b[Knidae\t\u001b[01;31m\u001b[KFalco\u001b[m\u001b[K\t\tperegrinus\n",
"\u001b[01;31m\u001b[KFalco\u001b[m\u001b[Knidae\tPolyborus\tplancus\n",
"Order:\t\u001b[01;31m\u001b[KFALCO\u001b[m\u001b[KNIFORMES\n",
"Order:\t\u001b[01;31m\u001b[KFALCO\u001b[m\u001b[KNIFORMES\n",
"Order:\t\u001b[01;31m\u001b[KFALCO\u001b[m\u001b[KNIFORMES\n",
"\u001b[01;31m\u001b[KFalco\u001b[m\u001b[Knidae\t\u001b[01;31m\u001b[KFalco\u001b[m\u001b[K\t\tcolumbarius\n"
]
}
],
"source": [
"grep -i Falco spawannxs.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's find the beautiful [\"Ara\" macaws](https://en.wikipedia.org/wiki/Ara_(genus)):"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Flacourtiaceae\tBan\u001b[01;31m\u001b[Kara\u001b[m\u001b[Ks\t\tvanderbiltii\n",
"Order:\tCH\u001b[01;31m\u001b[KARA\u001b[m\u001b[KDRIIFORMES\n",
"Ch\u001b[01;31m\u001b[Kara\u001b[m\u001b[Kdriidae\tCh\u001b[01;31m\u001b[Kara\u001b[m\u001b[Kdrius\tmelodus\n",
"Psittacidae\tAmazona\t\t\u001b[01;31m\u001b[Kara\u001b[m\u001b[Kusica\n",
"Psittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\tmacao\n",
"Dasyproctidae\tDasyprocta\tguam\u001b[01;31m\u001b[Kara\u001b[m\u001b[K\n",
"Palmae\t\tSyagrus (= Rhyticocos)\tam\u001b[01;31m\u001b[Kara\u001b[m\u001b[K\n",
"Psittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\t\u001b[01;31m\u001b[Kara\u001b[m\u001b[Krauna\n",
"Psittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\tchloroptera\n",
"Psittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[Ko\t\tmanilata\n",
"Mustelidae\tEira\t\tbarb\u001b[01;31m\u001b[Kara\u001b[m\u001b[K\n",
"Order:\tCH\u001b[01;31m\u001b[KARA\u001b[m\u001b[KDRIIFORMES\n"
]
}
],
"source": [
"grep -i ara spawannxs.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"But this poses a problem (*What is the problem?*). \n",
"\n",
"We can solve this by specifying `-w` to match only full words:"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Psittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\tmacao\n",
"Psittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\tararauna\n",
"Psittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\tchloroptera\n"
]
}
],
"source": [
"grep -i -w ara spawannxs.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And also show line(s) after the one that was matched, we can use `-A x`, where `x` is number of lines to use:"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Psittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\tmacao\n",
"\n",
"\u001b[36m\u001b[K--\u001b[m\u001b[K\n",
"Psittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\tararauna\n",
"Psittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\tchloroptera\n",
"Psittacidae\tArao\t\tmanilata\n"
]
}
],
"source": [
"grep -i -w -A 1 ara spawannxs.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similarly, -B shows the lines before:"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Psittacidae\tAmazona\t\tvittata\n",
"Psittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\tmacao\n",
"\u001b[36m\u001b[K--\u001b[m\u001b[K\n",
"Psittacidae\tAmazona\t\tochrocephala\n",
"Psittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\tararauna\n",
"Psittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\tchloroptera\n"
]
}
],
"source": [
"grep -i -w -B 1 ara spawannxs.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Use -n to show the line number of the match:"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[32m\u001b[K216\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[KPsittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\tmacao\n",
"\u001b[32m\u001b[K461\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[KPsittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\tararauna\n",
"\u001b[32m\u001b[K462\u001b[m\u001b[K\u001b[36m\u001b[K:\u001b[m\u001b[KPsittacidae\t\u001b[01;31m\u001b[KAra\u001b[m\u001b[K\t\tchloroptera\n"
]
}
],
"source": [
"grep -i -w -n ara spawannxs.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To print all the lines that do not match a pattern, use -v:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Also try `grep -i -w -v ara spawannxs.txt`\n",
"\n",
"This will produce a lot of output (which I will not show)! *What does this command do?*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To match one of several strings, use `grep\"string1\\|string2” file`. `grep` can be used on multiple files, all files, using wildcards for filenames, etc – explore as and when you need.\n",
"\n",
"## Finding files\n",
"\n",
"It's easy to find files in UNIX using, well, the `find` command! Let's test it (make sure you are in `Sandbox`, not `Data`!)"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [],
"source": [
"cd ../sandbox # again, note the relative path!"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"mkdir TestFind\n",
"cd TestFind\n",
"mkdir -p Dir1/Dir11/Dir111 #what does -p do?\n",
"mkdir Dir2\n",
"mkdir Dir3\n",
"touch Dir1/File1.txt\n",
"touch Dir1/File1.csv\n",
"touch Dir1/File1.tex\n",
"touch Dir2/File2.txt\n",
"touch Dir2/file2.csv\n",
"touch Dir2/File2.tex\n",
"touch Dir1/Dir11/Dir111/File111.txt\n",
"touch Dir3/File3.txt"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[0m\u001b[01;34mDir1\u001b[0m \u001b[01;34mDir2\u001b[0m \u001b[01;34mDir3\u001b[0m\n"
]
}
],
"source": [
"ls"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[0m\u001b[01;34mDir11\u001b[0m File1.csv File1.tex File1.txt\n"
]
}
],
"source": [
"ls Dir1"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Dir1:\n",
"\u001b[0m\u001b[01;34mDir11\u001b[0m File1.csv File1.tex File1.txt\n",
"\n",
"Dir2:\n",
"file2.csv File2.tex File2.txt\n",
"\n",
"Dir3:\n",
"File3.txt\n"
]
}
],
"source": [
"ls Dir* # the asterisk wildcards names so you can find all dirs with a particular start to their name"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Don't forget to use `man ` to check what `mkdir` and `touch` do. \n",
"\n",
"Now find particular files:"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"./Dir1/File1.txt\n"
]
}
],
"source": [
"find . -name \"File1.txt\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using `-iname` ignores case, and you can use wildcards:"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"./Dir1/File1.txt\n",
"./Dir1/Dir11/Dir111/File111.txt\n",
"./Dir3/File3.txt\n",
"./Dir2/File2.txt\n"
]
}
],
"source": [
"find . -iname \"fi*.txt\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can limit the search to exclude sub-directories:"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"./Dir1/File1.txt\n",
"./Dir3/File3.txt\n",
"./Dir2/File2.txt\n"
]
}
],
"source": [
"find . -maxdepth 2 -name \"*.txt\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can exclude certain files:"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
".\n",
"./Dir1\n",
"./Dir1/File1.tex\n",
"./Dir1/File1.csv\n",
"./Dir1/Dir11\n",
"./Dir3\n",
"./Dir2\n",
"./Dir2/File2.tex\n",
"./Dir2/file2.csv\n"
]
}
],
"source": [
"find . -maxdepth 2 -not -name \"*.txt\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To find only directories:"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"./Dir1\n",
"./Dir1/Dir11\n",
"./Dir1/Dir11/Dir111\n",
"./Dir3\n",
"./Dir2\n"
]
}
],
"source": [
"find . -type d -iname *dir*"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Installing or removing software in Ubuntu\n",
"\n",
"You can install software in your `/home` directory. In UNIX you originally had to login as `root` (administrator). But in Ubuntu, it is sufficient to add `sudo` (`s`uper `u`ser `do`) in front of a command. To install packages in all Debian (including Ubuntu) distributions you need to use the `apt` group of commands which allow you to deal with various installation (and un-installation) tasks.\n",
"\n",
"---\n",
"\n",
":::{figure-md} UNIX-sudo\n",
"\n",
"\n",
"\n",
"**`sudo` rules.** (Source: [XKCD](http://xkcd.com/149/))\n",
"\n",
":::\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Installing from the main repository\n",
"\n",
"You can install anything that is in the main Ubuntu package/software repository.\n",
"\n",
"★ Try installing R, which is pre-packaged the main Ubuntu repository:\n",
"\n",
"```bash\n",
"\n",
"sudo apt install r-base\n",
"\n",
"```\n",
"\n",
"This will prompt you for your password. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"```{tip}\n",
"**Using the `tab` key in terminal**: Don't forget to use the `tab` key in the terminal. It aut-ocompletes directory/file names for you (same in `R`and `Python`, and/or provides a list of files in the current directory (hit `tab` twice after certain commands. For example, try double tab after typing `ls`at the bash prompt.)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Enabling additional repositories\n",
"\n",
"You can enable additonal repositories that are already present (but not enabled by default) by using the Ubuntu \"Software Centre\" GUI. Launch Ubuntu Software Centre, and then Edit > Software Sources > Other Software (the exact approach will vary depnding on your Debian or Ubuntu distribution)."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Installing from other repositories\n",
"\n",
"In addition there are other repositories you can access, but which first need to be auhtenticated locally for your system's security and stability. There are typically available in the form of *Personal Package Archives* (PPAs). The general approach is: \n",
"\n",
"* Find the repository's web page (it's Launchpad page)\n",
"* Look for \"Adding this PPA to your system\" on the page. Make a note of the PPA's location, which has the format ppa:user/ppa-name.\n",
"* Open a terminal and enter: `sudo add-apt-repository ppa:user/ppa-name`. (replace 'ppa:user/ppa-name' with the PPA's location that you noted above)\n",
"\n",
"Your system will now fetch the PPA's key. This enables your system to verify that the packages in the PPA have not been interfered with since they were built."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Finally, download the latest list of software from each repository archive on your system (all of them: main, additional, and extra ones you have added manually such as the PPA you just added. To do this, use:\n",
"\n",
"```bash\n",
"sudo apt-get update\n",
"```\n",
"You can now start installing software from the new repository. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"★ Try installing something that isn't in the main repository – a weather indicator. \n",
"\n",
"For this, you first have to add the repository:\n",
"\n",
"```bash\n",
"sudo add-apt-repository ppa:atareao/atareao\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"This will ask you to first approve the addition of this repository. Then, `sudo apt-get update`, and then, finally,\n",
"\n",
"```bash\n",
"sudo apt install my-weather-indicator\n",
"\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"```{tip}\n",
"\n",
"**Ubuntu repositories:** Official Repositories are defined in `/etc/apt/sources.list`. You can enable or disable individual repositories by editing that file. \n",
"\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Installing from downloaded `*.deb` files\n",
"\n",
"You can also install software from third-party sources by downloading an installable package archive. Such archives come with a `*.deb` extension. \n",
"\n",
"For example, you can install [Visual Studio Code](https://code.visualstudio.com/) this way. Let's try it.\n",
"\n",
"★ First download and install the .deb package (64-bit) from their website (google it!). This will save it to your `Downloads` directory.Then `cd` to its location:\n",
"\n",
"```bash\n",
"cd ~/Downloads\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"This is same as typing `cd /home//Downloads`. Then, you can install using:\n",
"\n",
"```bash\n",
"sudo dpkg -i code_1.71.2-1663191218_amd64.deb # install, including dependencies\n",
"```\n",
"\n",
"Installing the `.deb` package will automatically also install the softwarre's apt repository and authorize it. That means henceforth, the program `code` will automatically auto-update during the regular system updates. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Removing software\n",
"\n",
"You can also easily remove software by, well, using the `remove` command! \n",
"\n",
"You will find that the list of UNIX commands are quite exhaustive, and their names quite intuitive, as they should be. If you think a certain command with a certain name should exist, it very often does!\n",
"\n",
"★ Try removing the totem video player and installing the more versatile VLC player instead:\n",
"\n",
"```bash \n",
"sudo apt remove totem\n",
"```\n",
"and then, \n",
"\n",
"```bash \n",
"sudo apt install vlc\n",
"\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Particularly useful UNIX tools\n",
"\n",
"That's it for our introduction to UNIX. We have just scratched the surface, but hopefully you have seen the potential for using the the UNIX/Linux terminal in particular for performing important tasks efficiently. UNIX has an incredibly wide array of commandline tools that can make your life easier. Here is a list of particularly useful commands (that often belong to a bigger toolkit; some of these will have to be installed using `apt` as you have learned above):\n",
"\n",
"| Tool | What it does |\n",
"| :------------- |:-------------| \n",
"| `top` | Find which processes are using your memory and cpu |\n",
"| `killall` | Kill processes (use with caution) |\n",
"| `grep` | Find patterns in directory and file names, as well as file contents; really useful for combining with other tools to perform operations on specific subsets of files and directories |\n",
"| `find` | Find files and directories|\n",
"| `pandoc` | Convert between document formats (e.g., \\LaTeX and Word or Markdown and PDF)|\n",
"| `convert` | Convert between file |\n",
"| `pdftk` | Manipulate PDF files, like merging multple ones or splitting them |\n",
"| `wget` | Download files from the web |\n",
"| `rsync` | Sync folders on your computer, including on any atatched devices (like an external hard drive) |\n",
"| `cron` | Set up automated tasks (like run a backup script based on `rsync`)|\n",
"| `unison` | Keep multiple computers synced |\n",
"\n",
"You will next learn to automate the use of such tools using [shell scripts](02-ShellScripting.ipynb).\n",
"\n",
"To find out more about these commands and tools, look at their `man` pages. You can also use `whatis` for a brief description, or`--help` *after* the command for usage syntax. \n",
"\n",
"An of course, you can use the internet to get help on practically everything about UNIX – there is huge (and fast growing) community out there."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Practicals\n",
"\n",
"### Instructions\n",
"\n",
"* Review (especially if you got lost along the way) and make sure you can run and understand all the commands and get the expected outputs we have covered above.\n",
"\n",
"* Make sure you have your directory organized with `code`, `data` etc and the necessary files, under your coursework directory (e.g., `CMEECourseWork/week1`).\n",
"\n",
"* Along with the completeness of the practicals/exercises themselves, you will be marked on the basis of how complete and well-organized your directory structure and content is – in all coming weeks as well.\n",
"\n",
"* *Please put (judicious) comments in all of your script files.*\n",
"\n",
"### Bash command challenge\n",
"\n",
"Here is a somewhat complicated bash command using two pipes (*you are not expected to include the answer to this one as part of your weekly submission*):\n",
"\n",
"```bash\n",
"find . -type f -exec ls -s {} \\; | sort -n | head -10\n",
"\n",
"```\n",
"\n",
"What does this command do? (Hint: try it on the test directories and files we created in `sandbox`)\n",
"\n",
"\n",
"### FASTA exercise\n",
"\n",
"In the directory `/data/fasta` you will find some [FASTA files](https://en.wikipedia.org/wiki/FASTA_format). These files have an header starting with `>` followed by the name of the sequence and other metadata. The sequence data start from the second line. Write a file called `UnixPrac1.txt` with UNIX shell commands that do the following (number each command with a hashed comment like so – `#1`, `#2`, etc):\n",
" * Count how many lines there are in each file\n",
" * Print everything starting from the second line for the `E. coli` genome\n",
" * Count the sequence length of this genome\n",
" * Count the matches of a particular sequence, \"ATGC\" in the genome of *E. coli* (hint: Start by removing the first line and removing newline characters)\n",
" * Compute the AT/GC ratio. That is, the (A+T)/(G+C) ratio (as a single number). This is a summary measure of base composition of double-stranded DNA. DNA from different organisms and lineages has different ratios of the A-to-T and G-to-C base pairs (google \"Chargaff's rule\"). For example, DNA from organisms that live in hot springs have a higher GC content, the GC base pair is more thermally stable.\n",
"\n",
"Note that there may be some `Y` and `R`'s in DNA sequences, denoting bases where there is uncertainty about identity, with R denoting purine (A or G) and Y pyrimidine (C or T) ribonucleotides. No agreed symbols exist for the other possible combinations, but these too may arbitrarily be reprsented by some non ATGC letters. To handle these, the calculation should be done by keeping these extra letters in place. That is, *do not delete* them before calculating the ratio, as this will shift the sequences. \n",
"\n",
"* Save `UnixPrac1.txt` in the `code` directory. \n",
"* Please make sure that each command calls the data from the `data` directory! \n",
"\n",
"*Do not write any of the above as shell scripts* (that's not been covered yet; see [Shell Scripting Chapter](02-ShellScripting.ipynb) – each answer should be a single line solution made of (potentially piped together) UNIX commands."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Readings & Resources\n",
"\n",
"Your University library likely gives you access to several e- and paper books on UNIX, some specific to Ubuntu. Browse or search and find a intro book that suits you.\n",
"\n",
"### General \n",
"* If you like history, [see this]( https://www.howtogeek.com/182649/htg-explains-what-is-unix).\n",
"\n",
"* Lots of UNIX tutorials out there. Try [http://software-carpentry.org/lessons.html](http://software-carpentry.org/lessons.html) (Chapter “shell”).\n",
"\n",
"* The bash reference manual: [https://www.gnu.org/software/bash/manual/bash.html](https://www.gnu.org/software/bash/manual/bash.html)\n",
"\n",
"### Repositories and packages in Ubuntu\n",
"\n",
"* Read about [Apt authentication](https://help.ubuntu.com/community/SecureApt) in Debian Linux distributions (including Ubuntu).\n",
"\n",
"* [What are Repositories in Ubuntu?](https://help.ubuntu.com/community/Repositories/Ubuntu)\n",
"\n",
"* [How to use repositories](https://help.ubuntu.com/community/Repositories/Ubuntu)\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Bash",
"language": "bash",
"name": "bash"
},
"language_info": {
"codemirror_mode": "shell",
"file_extension": ".sh",
"mimetype": "text/x-sh",
"name": "bash"
},
"latex_envs": {
"LaTeX_envs_menu_present": true,
"autoclose": false,
"autocomplete": false,
"bibliofile": "biblio.bib",
"cite_by": "apalike",
"current_citInitial": 1,
"eqLabelWithNumbers": true,
"eqNumInitial": 1,
"hotkeys": {
"equation": "Ctrl-E",
"itemize": "Ctrl-I"
},
"labels_anchors": false,
"latex_user_defs": false,
"report_style_numbering": false,
"user_envs_cfg": false
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": false,
"skip_h1_title": false,
"title_cell": "Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 4
}