Revision as of 16:34, 4 June 2013

Help! My Dreamhack has a command line!

If you've never used a Unix-based operating system before, the command line can be intimidating. Fear not! Working with files and data on the command line is actually fairly easy, and once you get used to it, you'll wonder how you ever lived without it.

The key thing to remember in unix commands is that they date from the days when vowels were an endangered species they are often shortened, contracted, or otherwise non-obvious at first glance. This is to make them easier to type -- the general theory is that it's nicer to type 'ls' sixteen thousand times a day than it is to type 'list'. They're generally designed to be typed quickly on a QWERTY keyboard.

To execute a command, you type the command and hit enter/return. The command won't execute until you hit enter.

Connecting

The window where you connect to your Dreamhack and type commands is also known as a 'shell window', for reasons that don't need explaining at this juncture in time. In order to connect to your Dreamhack, you'll need a program capable of doing SSH (Secure Shell) connections. For Windows, the best program is PuTTY. If you're on a Mac you won't need a separate program; the Terminal (Applications/Utilities/Terminal) will let you connect.

Details on connecting to your Dreamhack can be found in Dreamhack getting started.

Directories

Everything in Unix is either a process (a running program), a file, or a directory. Files live in directories, which can themselves contain directories, and so on.

When you first log into your Dreamhack, you'll be in your home directory. This is a special directory that stores all of your files and folders, and serves as the root for a lot of commands if you don't otherwise specify an argument. Your home directory is also known as "~" for short.

On the Dreamhack system, your home directory is located in /dreamhack/home/8XXX-yourusername. For instance, if my Dreamhack username is 'dh-rahaeli', and my assigned number is 8083, my home directory is located at /dreamhack/home/8083-rahaeli.

Directory paths are given with slashes separating the directory tree -- each slash represents another directory down the tree. If the directory path starts with a slash, it is an absolute path: it's counted from the root of the machine, the directory that holds all other directories. If a directory path doesn't have a slash before it, it's a relative path, and is usually counted from the directory you're in at the moment. (There are some exceptions to this, but I'll get into that in a bit.)

Command prompt

The command prompt is where you type in what you want the system to do. On a Dreamhack, it looks like this:

dh-rahaeli@hack:~$ 

Breaking that down into component pieces, it's: your username @ the machine you're on : (your present working directory). So, if you're in the ~/dw/cvs/dw-free directory (and remember, ~ means your home directory, so in this case it's shorthand for '/dreamhack/home/8083-rahaeli'), your prompt will look like:

dh-rahaeli@hack:~/dw/cvs/dw-free$ 

It's useful to have a long command line like that, because it'll remind you where you are, who you're logged in as, and what directory you're in. (Those of us who often have five or six shell windows open all the time, to various machines, sometimes get lost.)

Command flags

There are generally other options to a command -- things that make the command behave differently -- and those are known as 'flags'. To use a command with a flag, you type the flag after the command name, with a dash in front of it.

As we go through the tutorial, you'll see commands with flags for some other options. Flags are generally single letters, and remembering what single letter stands for what can often be annoying, but as time goes on you'll start to remember them.

Some commands have both flags and "long options" that are aliases of each other. Long options are full-word options, and they're called with two dashes instead of a single dash. (You generally won't need to know this, but you might see references to this style of option in tutorials elsewhere.)

Tab complete

If you have a file called thisisareallylongandridiculousfilenametobetyping.txt, or a directory called thisisaridiculousdirectoryname, the good news is that you don't need to type it out every time you need to refer to it. The Dreamhack shell offers tab completion. Type a few characters and hit tab, and the machine will complete the name for you.

If you get a beep and nothing happens, it means that there are multiple matches that don't have any text in common past the point you typed. For instance, if you have the files "update.bml", "update.bml.txt", and "uploadpic.bml.text", and you type "up" and hit tab, it'll beep and do nothing: it doesn't know which one you mean.

If you get a beep and part of the filename fills in, but not all, it means there are multiple matches that have some text in common past the point you typed, but not all. It will fill in as much as it can and then wait for you to either hit return or type more of the filename. For instance, if you have the files "update.bml", "update.bml.txt", and "uploadpic.bml.text", and you type "upd" and hit tab, it'll beep and fill in "update.bml", then wait to see if you mean "update.bml" or "update.bml.txt".

If you hit tab twice, it'll show you all of the possible completions. This can be useful if you forgot what the heck you called the file and don't feel like deleting the command to go look.

Tab complete only works for files and directories, not for commands.

Don't use spaces

You can't use spaces in Unix directory or filenames. If you try to, the system will think you mean two separate files or directories.

(This is actually a lie, but the process of dealing with files and directories with spaces is complicated. None of the existing DW files or directories have spaces, though, so you shouldn't need to know how to handle them, and you shouldn't make files or directories with spaces yourself.)

Listing files and directories

To list files, type:

ls

This returns all the files in a directory, formatted like this:

dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw 

On Dreamhacks, the output is color-coded, so directories are blue, executable files are green, and plain files are white. (More about permissions and the difference between executable and non-executable files in a bit.)

If you would like the information in a format that's accessible without color-coding, you can use the -F flag (note that the F is upper-case):

dh-daily@newhack:~$ ls -F
README  apache/  bin/  crontab-import  dw/

In this mode, directories will have a slash ("/") added, executable files will have an asterisk ("*") added, and plain files won't have anything added.

By default, 'ls' doesn't show you all the files in a directory, though. Any file that starts with a period (sometimes known as a dot-file) is a "hidden", system file. They're usually used for autogenerated preferences files. You generally shouldn't need to mess with them, but there are times when you explicitly want to set your preferences for something, or want to see what the hidden files in a directory are.

To show all files in a directory, use the -a flag (for "all"):

dh-rahaeli@hack:~$ ls -a
.              .bash_logout   .joe_state     .vim      apache         
..             .bash_profile  .nano_history  .viminfo  bin
.bash_aliases  .bashrc        .plan          .vimrc    crontab-import
.bash_history  .forward       .subversion    README    dw

You can also get more information about a file, in a "long form" directory listing. This includes information like who owns the file, what the permissions on the file are, and when it was last changed. (Permissions and ownership are something you shouldn't have to worry much about, but 'last changed' can be useful.) To get the long form, use the -l flag:

dh-rahaeli@hack:~$ ls -l
total 32
-rw-r--r--  1 dh-rahaeli dreamhack  3542 Feb 13 11:53 README
drwxr-xr-x  5 dh-rahaeli dreamhack  4096 Jul 28  2008 apache
drwxr-xr-x  2 dh-rahaeli dreamhack  4096 Jul 20 04:40 bin
-rw-r--r--  1 dh-rahaeli dreamhack  1617 Feb 20 13:42 crontab-import
drwxr-xr-x 16 dh-rahaeli dreamhack  4096 Jul 20 04:40 dw

The output of the -l flag is, in order: permissions, block size (ignore this), owner, group owner, file size (in bytes), date last modified, and the file name.

You can combine those two flags to get "long form listing of all files" by using "ls -al".

If you find the raw byte file size to be hard to read, you can use the flags -lh instead of just -l. This shows you the size in human-readable form:

dh-rahaeli@hack:~$ ls -lh
total 32K
-rw-r--r--  1 dh-rahaeli dreamhack 3.5K Feb 13 11:53 README
drwxr-xr-x  5 dh-rahaeli dreamhack 4.0K Jul 28  2008 apache
drwxr-xr-x  2 dh-rahaeli dreamhack 4.0K Jul 20 04:40 bin
-rw-r--r--  1 dh-rahaeli dreamhack 1.6K Feb 20 13:42 crontab-import
drwxr-xr-x 16 dh-rahaeli dreamhack 4.0K Jul 20 04:40 dw

Moving from directory to directory

Moving from directory to directory is done with the 'cd' command, for 'change directory'. It takes the argument of the directory you want to move into. For instance, if I'm in my home directory and I want to move into the 'dw' directory, I type:

dh-rahaeli@hack:~$ cd dw
dh-rahaeli@hack:~/dw$ 

(Notice that the prompt changes to reflect what directory I'm in right now.)

There's an easy shortcut for "directory above the directory I'm in right now" so you don't have to type out the absolute path, which is '../'. If you type that, it brings you to the directory above the directory you're in:

dh-rahaeli@hack:~/dw$ cd ../
dh-rahaeli@hack:~$ 

You can combine directories to go through multiple levels at once, too. If I wanted to cd from my home directory into the ~/dw/cvs/dw-free directory, I don't have to do it one level at a time; I can just type:

dh-rahaeli@hack:~$ cd dw/cvs/dw-free
dh-rahaeli@hack:~/dw/cvs/dw-free$ 

And you can go up and then over, using the ../ reference multiple times to mean "directory above the directory above", etc. If I'm in the ~/dw/cvs/dw-free/htdocs/manage/settings directory, and I want to go to the ~/dw/htdocs/manage/settings directory, that would look like:

dh-rahaeli@hack:~/dw/cvs/dw-free/htdocs/manage/settings$ cd ../../../../../htdocs/manage/settings
dh-rahaeli@hack:~/dw/htdocs/manage/settings$ 

If you just type 'cd' without any arguments, it brings you to your home directory.

You can combine using relative directory paths with listing files, copying files, removing files, etc. For instance, if you were in your home directory, and you want to list the contents of the ~/dw/cvs/ directory in long mode, you could type:

ls -l dw/cvs

If you were in the ~/dw/cvs/dw-free directory, and wanted to list the contents of the ~/dw/cvs/dw-nonfree directory, that would look like:

dh-rahaeli@hack:~/dw/cvs/dw-free$ ls ../dw-nonfree
bin  cgi-bin  cvs  etc  htdocs

Copying files

To copy one file to a new file name, the command is 'cp'. If I have a file named "file.txt", and I want to make a copy to "file.txt.backup", it would look like this:

dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw  file.txt 
dh-rahaeli@hack:~$ cp file.txt file.txt.backup
dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw  file.txt  file.txt.backup 

This will overwrite anything that already exists in file.txt.backup. If you're can't remember whether or not you already have a "file.txt.backup" file, and you want it to check before doing the moving, use the -i flag, which will ask you if you want to overwrite it:

dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw  file.txt  file.txt.backup 
dh-rahaeli@hack:~$ cp -i file.txt file.txt.backup
cp: overwrite `file.txt.backup'? n
dh-rahaeli@hack:~$ 

You can combine using directory paths with copying files. If you're in your home directory, and you want to copy "file.txt" to ~/dw/cvs/dw-free, you can do that like this:

cp file.txt dw/cvs/dw-free

(If you specify a directory for the destination, it will copy it into the directory. If you specify a file for the destination, it will overwrite that file. It's a good idea to use the -i flag if you're copying things around, in case you get the directory structure wrong.)

If the directory path you specify doesn't exist, you'll get an error message:

dh-rahaeli@hack:~$ cp file.txt dw/cvs/dw-missing/foo/bar/blah
cp: cannot create regular file `dw/cvs/dw-missing/foo/bar/blah': No such file or directory

This just means that you got your directory structure wrong.

Moving files

The command to move or rename files is 'mv'. It works exactly like 'cp' (Including overwriting the destination file if it already exists, and the -i flag to warn you first), only instead of copying the source file to the destination file, it renames it. It might seem strange to use a command whose name is obviously derived from the word "move" for renaming, but it made sense to the founding fathers of unix at the time with the way unix file systems work, it's exactly the same operation to both move and rename a file.

You can use it to rename files:

dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw  file.txt 
dh-rahaeli@hack:~$ mv file.txt newfilename.txt
dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw  newfilename.txt 
dh-rahaeli@hack:~$ 

You can also use it to move files from one directory to another:

dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw  newfilename.txt 
dh-rahaeli@hack:~$ ls dw
LICENSE  bin      cvs  etc     locks  src      t     test
README   cgi-bin  doc  htdocs  logs   ssldocs  temp  var
dh-rahaeli@hack:~$ mv newfilename.txt dw
dh-rahaeli@hack:~$ ls dw
LICENSE  bin      cvs  etc     locks  newfilename.txt  ssldocs  temp  var
README   cgi-bin  doc  htdocs  logs   src              t        test
dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw 

If you want to combine moving a file from one directory to another and changing the file name, you need to specify the new file name and the directory location all at once when you move it:

dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw  filename.txt 
dh-rahaeli@hack:~$ mv filename.txt dw/newfilename.txt
dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw 
dh-rahaeli@hack:~$ ls dw
LICENSE  bin      cvs  etc     locks  newfilename.txt  ssldocs  temp  var
README   cgi-bin  doc  htdocs  logs   src              t        test
dh-rahaeli@hack:~$

Deleting files

The first thing you need to remember is that on a Unix machine, there is no undo for deleting files. You're probably used to "delete" on your home machine meaning "move to Trash" or "move to the Recycle Bin", so if you screw up and delete the wrong thing, you can fish it back out of the Trash. You can't do that on your Dreamhack. Once you hit 'return' on the delete command, the file is gone and you can't get it back.

Now that I've scared you...

The command to delete a file is 'rm', for 'remove':

dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw  newfilename.txt 
dh-rahaeli@hack:~$ rm newfilename.txt
dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw 
dh-rahaeli@hack:~$ 

If you want the system to ask you whether or not you're really sure, use the -i flag:

dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw  newfilename.txt 
dh-rahaeli@hack:~$ rm -i newfilename.txt
rm: remove regular file `newfilename.txt'? Y
dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw 
dh-rahaeli@hack:~$ 

Like everything else, you can use relative directory paths to delete files, so you don't need to be in the directory that the file you want to delete is in.

Making new directories

To make a new directory, the command is "mkdir":

dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw 
dh-rahaeli@hack:~$ mkdir new-directory
dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw  new-directory 
dh-rahaeli@hack:~$ 

Deleting directories

To delete a directory, the command is 'rmdir':

dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw  new-directory 
dh-rahaeli@hack:~$ rmdir new-directory
dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw 
dh-rahaeli@hack:~$ 

A directory has to be empty before you delete it, or else you get an error message:

dh-rahaeli@hack:~$ rmdir new-directory
rmdir: failed to remove `new-directory': Directory not empty

If you get this error message, and you're absolutely positively sure that you want to delete the directory, you can use the 'rm' command with the -rf (recursive, force) commands. This is not a good idea, especially if you're also using wildcard matching (see below). Everyone, and I do mean everyone, has a horror story about being in the wrong directory when they do an rm -rf and losing years of data. (Mine involved all my saved email from 1992-1999. No, I didn't have backups. Sigh.)

If you're really really really sure you want to do this, it would look like this:

dh-rahaeli@hack:~$ rm -rf new-directory
dh-rahaeli@hack:~$ ls
README  apache  bin  crontab-import  dw 
dh-rahaeli@hack:~$ 

Be really really really sure.

Skimming through files

If you want to see whether a file you're looking at is really the file you think it is, you can call it up on your screen and page through it with the 'less' command:

less filename.txt

To get out of the pager, hit q.

There's an older version of 'less' called 'more'. It does slightly different things, and some of us can't retrain ourselves out of using it. For the most part, the two are functionally interchangeable, though you might prefer one or the other.

Wildcards

Most commands will accept wildcard matching. The wildcard operator is *. You can combine it with other bits and pieces to (for instance) only show you things that match certain patterns. For instance, if you only want to see a listing of files that end in .text:

ls *.text

If you only want to see files that end in .bml:

ls *.bml

If you only want to see files that have "pic" somewhere in the name:

ls *pic*

Be very, very, very careful if you are combining wildcard matching with the 'rm' command. It's a very good idea to use the -i flag if you're using a wildcard match with the 'rm' command, so that it prompts you to confirm each deletion, or do a "ls" first to make the system show you what it thinks the wildcard will match.

Searching files

If you're looking for files that match a certain pattern, the command is "find". It's a complex command that takes a bunch of different possible arguments, but here are a few useful bits of syntax:

find . -name "*.text"
# finds all files ending in .text, starting in the current directory and working through all subdirs
 
find . -name "*.bml"
# finds all files ending in .bml, starting in the current directory and working through all subdirs
 
find . -newer filename.txt
# finds all files that have been modified more recently than filename.txt
 
find -name *.orig -o -name *.rej | xargs rm
# finds all .orig and .rej files, then deletes them

The second way to search files involves looking into the actual files. This is done with "grep" (don't ask what it stands for), or "rgrep", which works recursively (goes into each of the directories and subdirectories). It's a good idea to use the -i flag, which means "case insensitive" -- that way, it treats "friend" and "Friend" as the same thing.

The arguments it takes are: rgrep (flags) (what to search for) (where to search). In practice, you generally want to search everything, so you can use the wildcard (*). An example, where we search everything for the string "friend" to see if we've forgotten to change any of them in the code:

rgrep -i friend *

If you want to search for multiple words, you need to put them in quotes, like such:

rgrep -i "friend page" * 
# will find any instance of "friend page"

If you want to search for strings with quotes, or things that contain periods, <> signs, the pipe (|), parentheses, the asterisk (not used as a wildcard), the slash, -- or any special symbol that means something different to the system -- you need to escape those characters with a backslash (\):

rgrep -i "Kirk\/Spock" * 
# will find any instances of "Kirk/Spock"

Redirecting output

By default, the results of a command will be displayed on your screen. You might not necessarily want that, though. For instance, if you do a search, you might get thousands of matches that will then scroll rapidly by on your terminal and leave you having to scroll back up.

You can redirect any output to another program, or to a file. The syntax is slightly different for each. If you want to redirect the output of one program to another program, use the pipe (|):

ls -al | more

That takes the output of the first command (ls -al) and sends it through the second command (more).

To redirect output to a file, use the greater-than symbol (>):

ls -al > directorylist.txt

That takes the output of the first command (ls -al) and sends it to the file named directorylist.txt. (Note that it will overwrite the contents of directorylist.txt if it already exists, so be careful!)

You can combine > with relative directory paths, too. For instance, even if you're 10 directories deep, if you want to output the directory listing to a file in your home directory, you could do:

ls -al > ~/directorylist.txt

(You'll see that structure in some of the 'how to generate a patch' instructions: a command such as "hg diff > ~/bug1234.patch").

Getting help

If you ever get stuck and can't remember how to work a command, you can get help on it. The help files are known as 'man pages' (for manual), and are accessed with the 'man' command. For instance, to get help on the help command, you'd type:

man man

To exit the help viewer, hit q. (The help viewer actually uses the "less" utility discussed above, so anything that works in "less" will also work here.) Not all of the man pages for the basic utilities are necessarily helpful, but in a pinch they might give you a hand. (They're mostly useful for remembering what the heck the eighty billion flags a command can take happen to be.)

Outside Resources

Good tutorials on Unix/Linux work:

• http://www.ee.surrey.ac.uk/Teaching/Unix/