Week 2: Linux and CLI Commands Lab Guide

Most of the time we interact with remote servers exclusively through command-line interface (CLIs) instead of using Graphic User Interface (GUI). Before computers had graphical displays, though, people typed instructions into a program called a command-line interface (CLIs). A CLI is a text only interface through which users interact with computers by typing text instructions in a console (or terminal), using specific syntax.

Here are a few reasons why it is important to learn the CLI:

• It is a very popular technology and a very common tool in DevOps and Computer Science.

• It is one of the best ways to control and interact with cloud services.

• You type faster than you click.

• It allows you to automate repetitive tasks.

To start, please:

• Open the VirtualBox, start a Linux environment (Ubuntu/Fedora) we set up in the previous lab.

• You can either SSH to your VM. But if your SSH has some issues then you can just type commands straight in the terminal of your VM for this lab.

• Ready to play around with CLI commands in the challenges below.

• Check your current date:

  • Note: the time and date is shown is the default EDT ("Eastern Daylight Time.") It is the daylight saving time observed in the Eastern Time Zone of the United States and parts of Canada so it might not exactly reflect the time format in Vietnam without the conversation.

date

• Print out the current login user:

whoami

• Print out the group which current login user belongs to:

groups $(whoami)

• Print "Hello World" to the screen:

echo Hello World

• Check current working directory (where you are):

pwd

Question: Can you guess what “pwd” stands for?

• List all files and directories in current working directory (where you are):

ls

Question: Do you see any files or directories? Also, can you guess what “ls” stands for?

• Create a new directory so we can work on it:

mkdir my_new_folder

Question: How to check if that my_new_folder is created after your previous command ?

• List all files and folders in current working directory (where you are) again:

ls

Question: Do you see any files or folders now?

• Change current working directory to that my_new_folder directory:

cd my_new_folder

Question: How to confirm if you are already inside my_new_folder ?

• Let's create 2 folders: folder1, folder2:

mkdir folder1

mkdir folder2

Or in one line:

mkdir folder1 folder2

• Let's create two empty files: file1, file2:

touch file1.txt

touch file2.txt

Question: Is there any quicker way to create these two files in one go?

• List all files and directories (including hidden ones -a and in long format -l):

Can you try each of these commands and see the differences?

ls

ls -l

ls -a

ls -la

Question: What is the . and .. (only show with -a) ?

• Let's add some text to the files:

echo Hello What are you doing > file1.txt

echo Bonjour How are you doing > file1.txt

echo I am fine thank you >> file2.txt

echo I am coding bash commands now >> file2.txt

• Let's check the content of file1 and file2:

cat file1.txt

cat file2.txt

Question: After checking the content of file1.txt and file2.txt, what is the difference between > and >> ?

• Let's move those files to the different directories:

mv file1.txt folder1

mv file2.txt folder2

Question: How can you check if your files are moved after you have run the commands?

• Let's check what inside folder1

cd folder1

ls

• Let's make a copy of file1.txt to folder2

cp file1.txt ../folder2

Question: Why do we use .. here?

• Let's change current directory to folder2:

cd ..

cd folder2

Question: How can you confirm if you are already in the folder2? Also, how can you check what files are in folder2?

Question: Can you quickly do two steps of changing directories from folder 1 to folder 2 in one go?

• Let's remove the copied file1 in folder2:

rm file1.txt

Question: How can you confirm before deleting, that file1.txt is in folder1 or in folder2?

• Let's try to remove the whole folder2:

cd ..

ls

rm folder2

Question: Why does this last command give you an error?

• Now, let's try to force it to delete the directory folder2:

rm -rf folder2

Question: Can you guess what are the r and f arguments of the rm command?

Here is the meme for the most destructive command in the history of computers:

Please do not try the command in this meme :)) (or you might need a new VM for that)

• Let's rename file1.txt to file3.txt:

cd folder1

mv file1.txt file3.txt

Question: Is there any quicker way to rename file1.txt to file3.txt without cd folder1 first?

• Let's create more files (file4.txt and file5.txt) in folder1:

touch file4.txt file5.txt

• Let's add more content in file4.txt and file5.txt:

echo Hello world >> file4.txt

echo Hello good friends to DevOps >> file4.txt

echo Learning new DevOps materials >> file5.txt

echo Good bye new friends >> file5.txt

• Let's see what is in file4.txt:

cat file4.txt

• Let's search the word "Hello" in file4.txt:

grep Hello file4.txt

• Let's search the word "DevOps" in the whole folder1:

grep DevOps -r .

Question: Can you guess what is -r and what is . ? Also, for this task, is there any scenario, we use "folder1" instead of . ?

• Let's save the results of the search in a new file "result.txt":

grep DevOps -r . >> ../search_result.txt

Question: So where is the location of search_result.txt? How can we check the content of search_result.txt? Can you guess what exact location of .. and . in this context?

• grep is for searching word content. However, to find file names then you need to use “find” command:

find ~ -name "*.txt"

Question: what is different between that command and “ls *.txt”

• Let's create more empty files with different extensions in the folder1:

touch image.jpg music.mp3 dataset.zip

Question: How can you list those newly created files in the folder1?

• We have a bunch of files in folder1 but we want to move all file3.txt, file4.txt and file5.txt outside of folder1:

mv *.txt ..

or

mv file* ..

Question: What is the * here? Can you explain the difference between the * in the first command and the * in the second command?

• Let's change the directory outside of folder1 (back to my_new_folder):

cd ..

• Let's count how many words in file4.txt:

wc -w file4.txt

Question: Can you guess what are the parameters w of the wc command?

• Let's count how many lines in file4.txt:

wc -l file4.txt

Question: Can you guess what are the parameters l of the wc command?

Question: What is the -w and -l ?

• Let's see how many times the word "world" appear all files in my_new_folder

grep world -r .

• Let's count how many times the word "friends" appear all files in my_new_folder

grep friends -r . | wc -l

Question: What is the purpose of |? Could you break this command into two parts and explain it?

• To check all the past commands I have used so far, you can use “history” command:

history

• Let’s edit a file using the nano command. But first, let’s install nano command:

sudo yum install nano

Enter “y” to confirm that installation!

Note: you don’t need to install nano command on Ubuntu, but only on CentOS as Ubuntu got the nano command already installed!

Question: what does “sudo” do? When do we need to use it to run commands?

• To edit “file5.txt”, we need to run:

nano file5.txt

• You should see this screen:

You use arrow keys to move the cursor around to edit the file, and just type like normal to add characters, use backspace to remove characters, and when you are ready to save and quit nano, press “Ctrl+X”.

• Add two new lines in the “file5.txt”:

• Press “Ctrl+X” to save the file and quit Nano:

Press “Y” to confirm to save the file with new changes or “N” to not save the changes.

Then nano will ask you if you want to rename the file: you can either change the file name to save as a new file or just press Enter to overwrite the current file! That’s it! That’s how you learn how to use the

nano editor!

• You can confirm the changes by using the cat command for file5.txt:

cat file5.txt

Hopefully, you will see:

• Sometimes you do not want to display all content of the file, you just want to show the beginning or the end of the file:

head -2 file5.txt

tail -2 file5.txt

Question: can you figure out what head and tail do? and what are the number parameters of each command?

Alright, that is fun so far! To automate stuff, we need to run these commands at once! Then we need to learn how to create a bash script which contains many commands and when we execute that bash script file, all the commands inside that file will be executed!

• First, let’s keep things organized by create a new directory to learn it:

mkdir learn_bash_script

cd learn_bash_script

pwd

• Then, let’s create an empty bash script file (which is just a normal file with the extension sh)

touch my_bash_script.sh

• Using nano command, we can start to edit this file:

nano my_bash_script.sh

• We would like to create these directory structures like in the picture:

• Using nano to edit the file like so and save the file:

Question: Can you make sense of these order of commands?

• Now, you can execute this bash script file:

bash my_bash_script.sh

Now, explore a new created src directory and see if what inside matches what we want!

Question: Can you run the bash script by simply running it “./my_bash_script.sh” without the command bash? If not, then what is the issue and how do we solve it?

Other interesting commands:

• Check the current Linux Distro information by looking at os-release file:

cat /etc/os-release

• To get a neat summary of the current logged-in user, we can install the “finger” package for that:

  • Replace “yum” with “apt” if you use Ubuntu VM instead of Fedora.

sudo yum install finger

• Once you have finger installed, you can use it to display some information about any user:

finger $(whoami)

• Here is my example screenshot:

That's it, you have finished the basics of the command line, please go through those challenges below!

• Create a directory named bash_learning, and change your current working directory to that directory.

• Create three folders named dir1, dir2, dir3 with one command.

• Create three empty files in each folder named file1.txt in dir1, file2.txt in dir2, file3.txt in dir3 with one command.

• Append 'Hello' to file1.txt, 'Bonjour' to file2.txt, 'Hallo' to file3.txt using echo.

• Copy file1.txt to dir2, file2.txt to dir3, file3.txt to dir1.

• From bash_learning/ find all files that contain 'Hallo' and save the result in hallo.txt

That Project Gutenberg text file includes many of Shakespeare’s best-known works, such as the tragedies Hamlet, Macbeth, Othello, King Lear, and Romeo and Juliet; the comedies A Midsummer Night’s Dream, Twelfth Night, and Much Ado About Nothing; the histories Henry V and Richard III; plus the Sonnets and other poems.

• Download the file containing the works of Shakespeare with this command:

curl https://www.gutenberg.org/files/100/100-0.txt > shakespeare.txt

• Use Linux commands you have learnt so far (like grep) to find out:

  • Count how many words are there in the file.

  • Count how many lines of this file.

  • Count how many times the word Romeo occurs, and compare with the word Juliet.

  • These are the top five most popular characters in Shakespeare’s book. Can you figure out

which character names are mentioned the most:

• Romeo

  • Juliet

    • Macbeth

    • Hamlet

    • Duncan

• Also, can you find the total amount of mentions of these five characters?

Write a bash script to generate this project structure:

|-- data/
|   |-- raw/
|   |   |-- cat/
|   |   `-- dog/
|   |-- train/
|   |-- test/
|   `-- validation/
|-- app/
|   |-- blueprints/
|   |   `-- home_page/
|   |       `-- blueprint.py
|   |-- static/
|   |   |-- styles.css
|   |   `-- index.js
|   |-- templates/
|   |   |-- base.html
|   |   `-- home.html
|   |-- app.py
|   `-- .gitignore
|-- scripts/
|   |-- preprocess.py
|   |-- model.py
|   `-- train.py
`-- notebooks/

Create a bash script file named "my_script.sh" and when you execute that script:

bash my_script.sh

This will generate all of these folders and files.

[Extra Information] Run the following command to check the file permissions of my_script.sh:

ls -l my_script.sh

You should see output similar to this, with your own username instead of tom:

The permissions -rw-rw-r-- mean that the file is readable and writable by the owner and group, and readable by others. However, it is not executable by anyone.

[Question] So why can we still run the script using this command?

Answer: You can run a script without execute permission by explicitly calling an interpreter, such as bash.

When you run bash my_script.sh, you are telling the Bash program to read and execute the commands inside my_script.sh. In this case, the script itself does not need execute permission because it is being passed as an argument to bash.

This works because:

• the bash program has execute permission

• you have read permission for my_script.sh

This method is useful for quick tests or when you do not want to make a script executable. However, if you plan to run the script often, you can add execute permission for the owner:

chmod u+x my_script.sh

After that, you can run the script directly:

./my_script.sh

• Download a small dataset of cat and dog on my github:

curl -L -O https://raw.githubusercontent.com/TomHuynhSG/public_datasets/master/small_cat_dog_dataset.zip

• Check the file which you just downloaded

ls

• Realise the file name is pretty ugly "small_cat_dog_dataset.zip_raw=true"

• Rename that file name to "small_cat_dog.zip"

mv small_cat_dog_dataset.zip\?raw\=true small_cat_dog.zip

• To unzip that file, we need to install unzip command:

  • Remove “apt” with “yum” if you use Fedora VM instead of Ubuntu VM

sudo apt install unzip

• Unzip that file using the command

unzip small_cat_dog_dataset.zip

• Now, you should see there is a new folder "DATA" contains 10 images of dogs and 10 images of cats

• Create a new folder “SORTED_DATA”.

• Create two new folders "cat" and "dog" inside the “SORTED_DATA”.

• Figure out how to quickly move all cat images to cat folder and move all dogs images to dog folder

Note: This is a very common first step you need to do during your machine learning project. You download a dataset which is messy, and you need to sort them and put them in the right folders to be ready for preprocessing.

This challenge is straightforward! Write a bash script to combine all the steps of Challenge 3 and Challenge 4 so when you run your bash script:

bash my_ultimate_script.sh

All of those folders and files are generated. Also the dataset file is downloaded, unzipped and sorted inside the folder data/raw/cat and data/raw/dog. All of those steps in one go!

|-- data/
|   `-- raw/
|       |-- cat/  <-- Put all the cat images in this folder
|       `-- dog/  <-- Put all the dog images in this folder
|-- app/
|-- scripts/
`-- notebooks/

• Navigate to Home Directory :

  • Start by going to your home directory. This is usually the default directory when you open a terminal, but you can always navigate to it using:

cd ~

• Create a File:

  • Use the touch command to create a blank file named filename.txt:

touch filename.txt

• Verify Creation:

  • To make sure the file was created successfully, list the contents of your directory:

ls -l

• You should see filename.txt in the list, along with its default permissions.

• Add some content:

  • For demonstration purposes, let's add some content to filename.txt. You can use the echo command in conjunction with the > redirection operator:

echo "This is a sample text for filename.txt" > filename.txt

• Verify Content:

  • Use the cat command to display the contents of the file:

cat filename.txt

• You should see the text "This is a sample text for filename.txt" printed to the terminal.

  • Now that filename.txt is set up and has some content, you can proceed with the steps related to file permissions and ownership.

• Modifying Permissions with chmod

  • The chmod command allows you to change permissions on a file or directory.

    • Using Symbolic Mode:

      • u stands for user, g for group, o for others, and a for all.

      • + is used to add permissions, - to remove, and = to set exact permissions.

      • Give execute permission to the owner:

chmod u+x filename.txt

• Remove write permission for the group:

chmod g-w filename.txt

• Give read and execute permissions to everyone:

chmod a+wx filename.txt

• Using Octal Mode:

  • Permissions can be represented as octal numbers: r is 4, w is 2, and x is 1. The sum of these numbers gives the permission set.

    • To give read, write, execute permissions to the owner, and only read and execute to group and others:

chmod 755 filename.txt

• This is calculated as:

  • Owner: rwx = 4 + 2 + 1 = 7

    • Group: r-x = 4 + 0 + 1 = 5

      • Others: r-x = 4 + 0 + 1 = 5

      • [Question] For server security, what is usually the default permission of the production project on the server? Is 755 good enough or is it very bad?
• Changing File Ownership with chown and chgrp

  • Using chown to Change Owner:

    • To change the owner of a file or directory:

      • “sudo chown newuser thefile”

      • For this example, we can set the new owner to be the root user

sudo chown root filename.txt

• Using chgrp to Change Group:

  • To change the group of a file or directory:

    • “sudo chgrp newgroup thefile”

      • Let’s change the group to root as well (so the file will belong to both user root and group root)

sudo chgrp root filename.txt

• To change both the owner and the group at the same time:

  • “sudo chown newowner:newgroup thefile”

    • In this example, we set the owner is tom and the group is wheel:

sudo chown tom:wheel filename.txt

Objective: Familiarize yourself with the installation and usage of popular Linux packages on Ubuntu or Centos.

As a DevOps engineer, one of your most common tasks is to troubleshoot issues by analyzing log files. This challenge simulates a real-world scenario where you must parse a web server's access log to identify errors, find performance bottlenecks, and generate a summary report for your team. This will test your mastery of command-line pipelines and text-processing tools.

The main website at 'RMIT Web Solutions' has been experiencing serious errors and slowdowns. The senior engineer has handed you the access.log file from the main web server and tasked you with finding the root causes. Your mission is to become a command-line detective and extract actionable intelligence from this raw data.

"The Command Line Murders" is a fun, detective-themed game that teaches the basics of navigating and manipulating files in a Linux-style command line by solving a fictional murder mystery. The game is designed for those new to the command line, so it's a great way to learn commands like grep, cat, head, tail, find, and so on.

Let’s go, detective!

There's been a murder in Terminal City, and the City Police Department needs your help!

To figure out who did it, you need access to a bash command line.

• Once you're ready, download it as a zip file using curl.

curl -L -O https://github.com/TomHuynhSG/clmystery/archive/master.zip

• It is a zip file so you need to unzip it:

unzip master.zip

• Change your current directory inside it:

cd clmystery-master

• Let’s list all files inside this special folder:

ls

• You should see these files then you are set to ready for your detective investigation:

• Begin by opening the 'instructions' file using a command using cat command so like:

cat instructions

(cat is a command that will print the contents of the file called instructions for you to read.)

Note: Be the true bash terminal detective! Do the right thing by only using the bash commands to solve this great mystery! Great Power comes with great responsibilities!

Ok, Detective! Let's start!

Everything you need inside the mystery directory so make sure you change your current working directory there!

You'll want to start by collecting all the clues at the crime scene (the 'crimescene' file).

The officers on the scene are pretty meticulous, so they've written down EVERYTHING in their officer reports. You'll have a file that represents the crime scene, which might contain numbers corresponding to different people, places, and things at the crime scene. Use cat to read the contents of the file.

You'll find files that contain people's names. You may need to look inside these to check their alibis. Again, use cat or grep to find the necessary information.

Similar to verifying alibis, you might need to find out which weapon was used in the murder. This could involve looking for files with names of different weapons and using command-line tools to determine which one was used.

Eventually, you'll collect enough evidence to identify the killer. The solution will come from piecing together the right combination of clues.

Fortunately the sergeant went through and marked the real clues with the word "CLUE" in all caps.

So your first hint is to search for the word “CLUE" for all files in the crimescene file! Thus, you will see three clues, and continue to use these clues to search for stuff you want to investigate in other files.

I hope you find the name of the person who committed the crime!

Remember, the idea is to simulate a real investigation using the command line, so avoid using text editors or other tools to view the files except for the instructions and cheatsheets provided.

Good luck and have fun, DevOps Detective!