In this section, we will review more commands to get you comfortable doing basic things on Hyak. Some of the data and exercises for this tutorial were sampled from The Unix Shell by Software Carpentry, but have been tailored to fit most Hyak users. Sampled materials are under the Copyright of Software Carpentry and are made available under the Creative Commons Attribution license (CC BY 4.0).
We'll use some commands you already know like pwd, cd, and ls, but we'll cover these commands and concepts for the first time:
wget- web get../- relative pathmkdir- make directorynano- for editing filescat- concatenatehead,tail,more,less- viewing optionscp- copymv- moverm- removetouch- create a file
You should have already completed these steps earlier in the tutorial, but before we continue, let’s make sure everyone is starting from the same place.
For the remainder of this tutorial, your working directory will be:
/gscratch/scrubbed/$USER/linux-fundamentalsNavigate to your working directory before moving on:
cd /gscratch/scrubbed/$USER/linux-fundamentalsIf this you don't have you working directory yet, return to the Filesystem Scrubbed section to set it up.
Download example data from the Software Carpentry github repository for the next exercises.
wget https://swcarpentry.github.io/shell-novice/data/shell-lesson-data.zipList your working directory to show the zipped version of the lesson directory shell-lesson-data.zip.
lsUnzip the lesson directory with the unzip command.
unzip shell-lesson-data.zipls should show a unzipped version of the shell-lesson-data directory as well as the zipped version.
Change directory to go into shell-lesson-data/exercise-data/writing/
cd shell-lesson-data/exercise-data/writing/Print your working directory to understand where you are.
pwdList the writing directory
ls The expected result is:
haiku.txt LittleWomen.txtAnother way to move around the directories on the filesystem is with relative paths. Use ../ to move backwards (toward the root) one directory at a time.
Print your working directory to understand where you are.
cd ../Print working directory to see the result of cd ../
pwdUse cd and ../ twice to go backward 2 directories
cd ../../Print working directory again to see the result
pwdYou can list the contents of the directory "above" where we are now without changing directory.
ls ../
Let's go back to shell-lesson-data/exercise-data/writing and practice making a directory.
cd shell-lesson-data/exercise-data/writingMake a directory called "thesis"
mkdir thesisNote that mkdir is not limited to creating single directories one at a time. The -p option for "path" allows mkdir to create a directory with nested subdirectories in a single operation:
# make a directory called project with subdirectories data and results
# make these one directory "above" where we are now
mkdir -p ../project/data ../project/results-F option with ls puts a / after directories to differentiate them from other objects.
ls -FThe -R option to the ls command will list all nested subdirectories within a directory. Let’s use ls -FR to recursively list the new directory hierarchy we just created in the project directory:
ls -FR ../project
⚠️ WARNING: Avoid complicated names for files and directories
- Avoid spaces - Spaces separate arguments on the command line, so they often cause unexpected behavior. Use
-or_instead (for example,north-pacific-gyre/rather thannorth pacific gyre/).- Don’t start names with
-(dash) - Anything beginning with-is interpreted as a command option, not a filename.- Stick to safe characters - Use letters, numbers,
.,-, and_. Many other characters have special meanings in the shell and can cause commands to fail or behave in unexpected ways.
To edit files on Klone we need to go back to basic text editors. You will not have access to a word processor, and formatting and syntax doesn't always translate from Microsoft Word or similar software to executable commands on Klone. Next, we will create a file called draft.txt and open it in the text editor nano.
First change directory to thesis:
cd /gscratch/scrubbed/$USER/linux-fundamentals/shell-lesson-data/exercise-data/writing/thesisThe create and open a file called draft.txt with the command:
nano draft.txtNOTE: There are other text editors you could choose from.
vimis a popular choice.nanois beginner friendly, so that is what we will use here.
Let's type a few lines of text.
Screenshot of nano text editor showing draft.txt and the example text.
Once we’re happy with our text, we can press Ctrl+O (press the Ctrl or Control key and, while holding it down, press the O key) to write our data to disk. We will be asked to provide a name for the file that will contain our text. Press Return to accept the suggested default of draft.txt.
Once our file is saved, we can use Ctrl+X to quit the editor and return to the shell.
Let's view our work.
cat draft.txtImportant concept: Paths and Access
On the command line, your location determines what you have access to. To demonstrate this important concept, consider the following example:
There is a dataset of sequences from a mythical creature called a Basilisk (from the Harry Potter Universe). If you wanted to execute anything on
basilisk.datyou would not be able to unless you are inside of the directory where it is stored or using its absolute path. Let's go find it and print its contents to the shell withcat.cd /gscratch/scrubbed/$USER/linux-fundamentals/shell-lesson-data/exercise-data/creaturesList the directory.
ls
catthe Basilisk datacat basilisk.datTo illustrate the concept of paths and access. Let's go back to our previous directory (likely
/gscratch/scrubbed/$USER/linux-fundamentals/shell-lesson-data/exercise-data/writing/thesisif you have been following along) and try tocatthe Basilisk data.cd - cat basilisk.datThe expected output:
cat: basilisk.dat: No such file or directoryWe get an error because we can't access
basilisk.datfrom our location without more information. However, we can provide the relative or absolute path to accessbasilisk.dat:cat ../../creatures/basilisk.dator with the absolute path:
cat /gscratch/scrubbed/$USER/linux-fundamentals/shell-lesson-data/exercise-data/creatures/basilisk.datIn this example, we have provided more information about the location of
basilisk.datrelative to our location (i.e., using../to indicate the position relative to our current directory location) and are able to access it and execute thecatcommand upon it.We can also use absolute paths to access a file from ANY location on the Klone filesystem. This is a error-proof method of making sure you can execute commands on files no matter where on the filesystem the command is issued. Let's use an absolute path to see a file from elsewhere on the filesystem. There is a dataset called
animals.csvunder/mmfs1/sw/hyak101/basics/data/. Let's view it withcatfrom our current directory.# first print your working directory to see your location pwdNow use the absolute path to
animals.csvtocatitcat /mmfs1/sw/hyak101/basics/data/animals.csvTo summarize, to access a file or directory (i.e., item) you must be:
- inside of the directory where the item is
- provide a relative path to the item from your current directory
- provide an absolute path to the item
These four commands are other ways to view file contents, great for viewing smaller chunks of longer files.
# change directory to creatures data
cd /gscratch/scrubbed/$USER/linux-fundamentals/shell-lesson-data/exercise-data/creatures/
head basilisk.dat # displays the first part of a file
tail basilisk.dat # displays the last part of a file
more basilisk.dat # displays file contents one page at a time
# exit the more command with Q
less basilisk.dat # displays file contents one page at a time
# exit the less command with QBy default, head and tail commands display the first and last 10 lines of a file. To view a specific number of lines use -n followed by the desired number of lines.
head -n 20 basilisk.datMove to top of working directory.
cd /gscratch/scrubbed/$USER
pwdCopy animals.csv to your current directory using the shorthand . to mean "here"
cp /mmfs1/sw/hyak101/basics/data/animals.csv .Copy a directory with all its contents using recursive copy
cp -r /mmfs1/sw/hyak101/basics/data/ .mv animals.csv dataset.csv
⚠️ WARNING:rmpermanently deletes a file. This action is irreversible.
rm dataset.csvRemove a directory with recursive rm.
cd shell-lesson-data/exercise-data/writing/
ls
# Use extreme caution with this command
rm -r thesiscd /gscratch/scrubbed/$USER
pwd
touch file1.txt file2.txt file3.txt
ls # make sure the .txt files were createdContinue to 07-filtering.md to learn more command line tools for working with files and text.