Below is a list of command line tricks I've found to be very useful in web development that a surprising number of web developers don't know about.
1. ln -s /some/destination name_of_link
This is pretty straightforward: it'll create a symbolic link from name_of_link to /some/destination. If I'm sitting in /www/matthew/ and I entered the above command, a new file will be created inside that directory that moves the user to /some/destination if they CD to name_of_link. (It's a shortcut, basically).
If I'm working on a web server where the docroot is buried deep in a directory structure, I'll often create a symbolic link in something like /www to save myself a few keystrokes when accessing it.
Also, apache won't follow symbolic links unless you tell it to do so using the following directive: Options +FollowSymLinks
2. tail -f /some/fileThis tails a file: it'll read what's at the end and output it to your terminal. The -f parameter tells it to keep outputting as the file grows. I use this a lot when examining log files that are constantly being written to.
3. ctrl+z and bgctrl+z is the long lost brother of ctrl+c. It's a signal that tells the process to suspend execution, while ctrl+c terminates it. Ctrl+z is useful if you execute a process but you want to get control of your shell; it'll suspend the process and send it to the background. Think of ctrl+z like minimizing a window, except once it's minimized it's not doing anything anymore. If you want the process to stay in the background but continue running, that's where bg (background) comes in: typing bg once a process has been suspended makes the process resume but still keeps it in the background.
I often combine ctrl+z and bg with tail:
shell> perl -e 'while() { print "."; sleep(1); }' > bunch_of_dots.log
[hit ctrl+z, execution is suspended]
shell> bg [process is now in the background and running]
shell>tail -f bunch_of_dots.log [show me the printed dots]4. fg and jobs
fg is used if you background a process but want to bring it to the foreground. If you have multiple processes in the background, type 'jobs' into your terminal to see them all. This will display a list of processes that are in the background with each process assigned a number (Note: these are not the same as pids). Typing fg [some number] will resume the process you sent to the background where [some number] matches the number reported by 'jobs.'
A good example of using fg is if you're working with an interactive program, such as the mysql command line, but you want to drop back to the shell.
mysql>SELECT foo FROM bar;
mysql> [hit ctrl+z]
[1]+ Stopped
shell>pwd /home/matt/stuff
shell>jobs
[1]+ Stopped mysql -u matt -p matts_db
shell>fg 1
mysql>[hooray, we're back in the mysql command line]
You can omit the job number when running 'fg,' it'll just foreground the first process you sent to the background. So if you only have one process in the background you can just type 'fg' without any extra numbers.
5. Hit the freakin tab key!
Assuming your shell is configured properly, hitting tab will auto-complete with whatever it thinks you need. I've encountered a surprising number of developers who don't know about this and move around servers at glacial speeds. If you aren't familiar with the tab auto-complete, get to a terminal right now and type cd [tap the tab key], you should see a list of available files or commands.
6. scp
This securely copies a file across a network using SSH for data transfer. For example:
scp [email protected]:~/secret_stuff /some/destination
This would connect via SSH to example.com as the user 'matt' and copy the file 'secure_stuff' from my home directory on example.com to /some/destination on the machine I'm currently on. It's saved me a ton of time when transferring sensitive files from one machine to another.
7. screenScreen serves two purposes for me:
- It allows things to persist even after I've disconnected
- It allows me to access multiple terminal sessions from inside a single terminal window
Suppose I'm using the wifi connection at a coffee shop and I want to execute a binary that's going to take several hours to run but I'm not going to be able to stay connected for that entire time. Before running the binary, I enter 'screen' into my terminal. This effectively creates a new session that will persist even if I get disconnected. Once this screen has been launched, I execute my binary and exit the screen by pressing ctrl+a+d. If I want to resume the screen later from a different location I simply log into my server and then type 'screen -x,' and my binary should still be running.
Screen is also useful if you want to save some precious real estate and run a bunch of different sessions from the same terminal window. Suppose you have a server at home that you like to SSH into and play /bin/fortune on, and server at work that you want to log into and tail your apache logs.
shell>screen -S homeserver [This would create a screen named 'homeserver' and attach to it]
shell> ssh matt@home [you're in the 'homeserver' screen, now ssh to my box at home]
matt@home>[we're ssh'd in to my box at home, now press ctrl+a+d to detach from this screen]
laptop>screen -x homeserver [this will re-attach to my home server]
The same goes for your work server:
screen -S workserver etc.. screen -x workserver
Running 'screen -list' should show homeserver and workserver listed. Remember that once inside a screen, you type ctrl+a+d to gracefully detach from it. There's an army of other things you can do with screen, I highly recommend reading the man page or the screen user manual. Also, screen is not a built-in command so you might have to install it first.
8. Create a file (the fast way) with touch
shell>touch foobar.log
Bam! You've successfully created foobar.log. It's much faster than running vi, saving, and exiting. Touching an existing file updates the time it was last modified.
9. Gut a file from the inside out
cat /dev/null > foobar.log
This is a nifty trick to clear out the contents of a file but not remove the file itself. Suppose foobar.log has gotten ridiculously huge and you want to remove all the stuff inside it but you don't want to delete the file, simply cat /dev/null and direct the output to your file. In other words, take /dev/null (a black hole of nothingness) and stuff it inside foobar.log.
10. Backup and reload your databases
mysqldump > mydb.sql
mysql < mydb.sql
This is mysql specific, but this article is directed at web developers and I've been asked this question enough times I figured it warranted being listed here.This takes somedatabase and dumps the SQL to my_database.sql:
mysqldump -u someuser -p somedatabase > my_database.sql
To reload this data, simply direct it back in:
mysql -u someuser -p some_other_database < my_database.sql
mysqldump has a ton of options, for more information check the mysqldump man page
11. Make a directory tree (the fast way)
mkdir -p a/deep/directory/tree
Passing -p to mkdir tells it to create the parent directory if it doesn't already exist. This allows you to create a hierarchy of folders without having to go into each one and issue separate mkdir commands.
12. killall: kill a process by nameRunning 'kill' will terminate a process but you have to know the PID (process ID) first. Killall allows you to kill a process but not look up the PID. For example:
killall -9 httpd
This will kill all the processes that are httpd. The -9 basically kills the process with extreme prejudice, it will die immediately. Be careful when using this on processes that need to do cleanup before they die.
13. Alias: Create your own commandsEver find yourself typing in really long commands over and over again? You can save yourself some time using alias. For example, here's how I'd create a command called 'mattsdb' that calls my database with all the parameters I need:
shell> alias mattsdb='mysql -u matt -p mattsdb -h example.com'
Now if I type 'mattsdb' the mysql command line will come up. Unfortunately if I were to log out and then log in again this alias would not persist. If you want your aliases to last beyond your current session, you'll have to tweak the config files for your shell. I primarily use bash and keep my aliases persistent by adding them to the bottom of ~/.bash_profile
Miscellaneous Tricks
Here's a few random gems that I've found to be very useful. As always, I highly recommend reading over all their respective manuals .
- ls -lSr List files (sorted by size)
- df -h Show available disk space in human-readable format
- du -sh /some/dir Show how much space /some/dir is taking up
- ps aux | grep blah List all the running processes but only show ones that contain 'blah'
- wget -spider https://0at.org Fetch pages and behave like a web spider: don't download the pages, just check to see if they are there
- ab - Apache benchmark, use this if you want a quick n' dirty way to benchmark how well your site performs under a heavy load.
- perl -i -pe 's/foo/bar/gi' * search through * and replace all foo with bar
I have found symbolic links to be very useful as I test new code for some of my websites. For www.foo.com I will typically have a beta.foo.com that points to a different set of code. For example, I have a website that has a lot of user-uploaded content like photos, documents, etc. When I want to test new code for the site I make a copy of the code and place it in a beta folder and make my changes there. I do not copy the user-uploaded content but, instead, I create a symbolic link in the beta folder that points to the user-uploaded content. This saves room on my server and is a lot faster than copying gigabytes of data to my beta folder and keeping the data synched-up.
The $_ variable holds the last command line argument entered.
$ ls /some/really/long/path
... then
$ cd $_
will put you in /some/really/long/path, for example. I find I use this all the time.
Heads & Tails & Stuff
Depending upon your flavour of unix / linux you can use the pipe sign | to redirect output from one process to the next.
So tail -f |pg allows you page the output or
tail -f >> myfile appends to the ascii file myfile.
ps -ef |grep 'root' | tail -f |pg lets you keep an eye on the processes spawned by root.
head filename shows you the top so many lines of filename.
head -18 filename outputs the first 18 (or nn) lines of filename.
Whilst touch filename creates an empty file if one doesn't exist, touching an existing one updates the filename date and time stamp.
unix, vi and regular expression handling is extremely powerful stuff.
something for your .bashrc:
alias log='tail -f /var/log/apache2/access.log | egrep --line-buffered -v "(\.(ico|png|gif|css|jpg|css|js|mp3|zip|txt)|images|Gigabot|mp3player|files|Googlebot|Jeeves|Slurp|msnbot)" | cut -d " " -f 1,7,11,12-32'
to replace recursive in *.html files, with multiline support:
find ./dir/ -name *.html -exec perl -pi -e 'undef $/; s/REPLACE-THIS/WITH-THAT/s' {} \;
instead of cat /dev/null > file.log to empty a file:
echo "" > file
to execute a php script more than one time:
<?php
if( !is_numeric( $argv[1] ) ) { $argv[1] = 0;}$l = $argv[1] * 5000;
$q = 'SELECT * FROM table LIMIT '.$l.',5000';
...
?>
...and execute it with
screen php file.php 0
screen php file.php 1
screen php file.php 2
etc. ;o)
if you want to see the disc usage in diff folders use:
du -h -s *
now a small bash script to backup folders, add it to your crontab for every 4 days or so...
#!/bin/bash#Programm backup.sh
tar -czf /backup/backup.html.`date +%d.%m.%G`.tar.gz /var/www/web1/html > log.backupif [ "$?" -ne 0 ]; then echo "error" mail your@email -sBACKUP-ALERT-HTML < last.logelse find /backup/backup.html.\* -mtime +13 -exec rm '{}' \;fiexit
this script packs everything in /var/www/web1/html (like if you use confixx as panel) into backup.dd.mm.yyyy.tar.gz and if everything is okay it deletes the backup file older then 13 days (2 weeks cycle)
if something is not okay you get an email ;)
greets, tobsn.
(@ theonlybushilike.com - webuildspam.com)
* Quickly substitute text within a bunch of files
perl -p -i.bak -e 's/old/new/g' *.html
* Copy a directory keeping permissions intact (assuming you're root)
tar -cf directory | (cd /new/place && tar -xf -)
* Capture your session with the "script" command.
On#9 (and #8, I guess), it's much simpler to do
> foobar.log
The reason you're doing that is so that the inode isn't changed. If you were to remove it, the inode is probably still open by the application so the file on disk continues to grow until the file is closed (probably be restarting the app)
On #12 also look at "pkill", which lets you do the same as killall but by specifying parts of the process name (be careful, of course :) )
Also, web devs should be forced to watch the output of mytop (or tail -f the slow-query log) while developing :)
Sean
Wouldn't doing a "cp -Rp /source/files /destination/files" work to copy a directory with permissions? Oh wait, does that only keep file owners and not permissions?
cp -Rp should work, although there are few caveats with setuid and setgid files.
The neat thing about using tar and pipes is that you can combine it with ssh to copy a directory tree to a remote box without having to initiate an ftp or scp transfer for each file:
(cd /src; tar -cvf - foo) | (ssh other.machine 'cd /dst; tar -xf -')
Ac cool...I'll add that to me SSH cook book.
tar -C /src -cf - foo | ssh user@host tar -C /dest -xvf -
The '-C' is simpler than launching a subshell. I also like putting the '-v' on the destination side as a small sanity check.
You can go the other way with:
ssh user@host tar -C /src -cf - foo | tar -C /dest -xvf -
Depending on connection speed b/w hosts, it may be helpful to throw in -z on both sides of tar.
You can do a remote backup with:
tar -C /backmeup -czf - foo | ssh user@host cat \> /archive/foo.tar.gz
Excellent, thanks Sean!
In fact, I use that enough to have an alias:
alias what='ps auwwwx | grep -v grep | grep'
what java
what httpd
...
excellent collection of tips... not all are new, but I found some nuggest on this page meanwhile...
and I really didn't know about "screen" cool!
Another one that might be interesting:
shell> [...] &
Adding the & at the end causes the process to be run in the background, so it will continue executing even after you exit the shell. I use this for scripts that take a long time to finish.
Great post, Oatmeal!
I am sure there are lots of Linux people reading the blog. I specially appreciate part on the screen (which I have heard of, but never tried) and explanation of bg/fg/ctrl+z combinations, which I don't use, because I am using Yakuake with a tabbed shell window, but certainly could come handy sometimes when remotely logged.
Small tip for me for batch resize of the images in the directory and directories below the current one (requires mogrify):
find . -name "*jpg" -exec mogrify -resize 350 -quality 70 {} \;
If you want to kill some processes using grep (more flexible than killall) :
$ kill -9 `ps aux | grep proc | awk '{ print $2 }'`
This will kill all the processes matching "proc".
Easiest way to truncate a file
: > file
Exactly
Gut a file from the inside out
cat /dev/null > foobar.log
**can be replaced with...
> foobar.log
If you don't have access to sessions... like on AIX... open another instance of the shell (ie run ksh, bash, zsh... and exit when you want to go back to the previous directory)
**Get all processes from the user name and search for a process that is running
ps -fu <userName> | grep process
**learn the FOR command structure...
for variableName in `cat some.file`do
something with $variableName
done
Love the article, I have used linux on the desktop, solely, for a year and a bit now, and I still didn't know a couple of these!
One extra for you though:
Forget scp, use rsync. Does very much the same thing, but uses diffs/checksums to compare files so it only transfers changes. A LOT faster :)
If you ever find yourself on a non-Linux, non-FreeBSD system, be very careful with killall. On some of them, it does exactly that - kills everything. If you're root, that's roughly equivalent to a shutdown.
Want the to show the processes that contain "blah", but not the grep process?
ps auxwww | grep [b]lah
or
ps auxwww | grep blah | grep -v grep
Great tips, thanks :)
On many systems, you can use ‘pkill’ instead of ‘killall’ (indeed, there are many systems on which ‘pkill’ exists and ‘killall’ doesn't or is actively harmful). The rule of thumb is to try ‘pkill’ first, and if that doesn't exist, look for ‘killall’. (‘pkill’ is actually more powerful, if memory serves; I think it can perform regular expression matching and so forth).
Do not ever run ‘killall’ as root on a Solaris host.
Also, this commenting system is completely broken in Safari.
step #13 is extremely insecure almost making scp pointless.
If you create an alias and use the option mysql -p you should NEVER actually enter the password, because any user regardless of their user/group access to your files they can now see your mysql username/password/hostname just by using the ps command. This example is even explicity warned about in the man mysql pages as being insecure.
The option mysql -p does not explicitely state your password, it makes the mysql client prompt you for your password. Running "ps" would reveal "mysql -u someuser -p somedatabasename -h somehost." The password would not be visible.
Hi,
Someone knows how to show which command was used to run a process that is running ?
Thanks !
history
ps aux
ps aux | grep yourcommand
I think it is better for web developer to use windows-based or web based application rather than using command line or shell scripting...
anyway developer dosent need to be a system engineer
I think this is a concept fault ...
For sending stuff from the web (a command line for the web), try futz.me
https://futz.me
cd alone will take you to your home directory. cd followed by a hyphen "cd -" will take to the last directory you were in.
Nice post :)
I assume you know this being your an SEO firm, but you got nice link bait from Lifehacker, https://lifehacker.com/software/command-line/thirteen-useful-terminal-commands-234145.php. I didn't notice a trackback in this thread of comments.
update
something very useful -> if you want to merge two files:
cat file1 >> file2
file 1 will be added after the end of file2.
if you want to delete a big cache folder or something else and "rm" replys you that there are to much files in the directory use this in the directory:
echo * | xargs rm
if you want to see the keywords from google in your access_log try this:
cat your_access_log | grep ".google." | cut -d " " -f 11 | grep "q=" | cut -d "q" -f 2 | cut -d "=" -f 2 | cut -d "&" -f 1 | php5.1 -R '$k[] = urldecode($argn);' -E 'print_r($k);'
-R is for "Run PHP <code> for every input line" $argn is the var that you get from the pipe each time and -E is executed at the End of the feed. you can also add utf8_decode() around the urlencode (for expl. european users with a diff charset)
Nice tips. Quite general really.
UNIX is very powerful for manipulating
text files which web sites essentially are.
Have a look at:
https://www.pixelbeat.org/cmdline.html
https://www.pixelbeat.org/lkdb/screen.html
The equivalent postgresql dump would be using the pg_dump command. If you want to dump all databases use pg_dumpall (go figure) and redirect it to a file. To restore any of these files do psql -f filename.sql [database].
I usually use pg_dump -ROx to dump one particular database adding s for schema, a for data, d for inserts (instead of from stdin) and t for a specific table.
Was it the number 13.. the list factor.. or something else.. magic happened today and this post got 990 diggs..
Linux + list format ( top XX of something ). Diggers love lists, it's much easier to read and digest than a 15 paragraph blog post.
That "screen" trick is neat. Hadn't seen that one.
Most command line junkies know up arrow and down arrow for moving through previous commands. However many people don't seem to know that you can use ctrl-a to go to the start of a line if you want to change something quickly (like you forgot to -p on your mkdir).
An even quicker way to repeat commands is using the "!". !! will repeat the last command, !scp will repeat the entire last scp command. !s would do it as well provided that scp was the last command you used that started with an s.
Finally the following to remove files in a directory that were created more than x number of days ago (great for managing Maildir).
find ./ -ctime +x | xargs | rm -f -r
Best to practice that last one first with a | more instead of the rm. I figured that one out trying to battle the "too many files" problem. Using echo in that case never occured to me. Great tip!
However many people don't seem to know that you can use ctrl-a to go to the start of a line if you want to change something quickly (like you forgot to -p on your mkdir).
Whoah, sweet..that's a new one to me that I've wished for many, many times.
This is definitely one of those blog posts where reading the comments is just as valuable (if not more valuable) than the blog post itself!
you know... ctrl+e is for jump back to the end... ;)
That is a great tip too. I guess I don't find myself using it as much becuase you can press return in the middle of a command and it will still execute.
For instance, if I make a mistake I press up arrow, ctrl-a, fix the mistake and press return without going to the end of the line again.
You briefly mentioned "| more" but I think it would be good to give a little more information about it's usage as it can help less experienced linux users greatly.
You can tack "| more" onto pretty much any command that returns too many lines to fit on one screen. For example, "ls -l | more" will list the contents of a directory until the data fills the screen and then it will pause and wait for you to press either ENTER to move ahead one line at a time or SPACEBAR to move forward a page at a time.
I prefer to pipe stuff to less. It allows you to go both down a list and back up, something more doesn't allow for.
Sometimes less is more...
I realize that my syntax for the find command had a mistake!
Should be:
find ./ -ctime +x | xargs rm -f -r
so to delete files created five days ago or more:
find ./ -ctime +5 | xargs rm -f -r
Home works for me, but I know that prints hex char in some consoles.
This is confusing for a non-programmmer like me... :(
That's where this old standby comes in handy:
#man foo
And as many male (and some female) *nix geeks will tell you, the mystery of the opposite sex cannot be found here:
#man woman
Too bad it's not that easy :(
This may seem crazy, but one of my most used commands is plain old "clear" (which I don't see a lot of people use). It's great when you're working in a Terminal with transparency turned on and you need to clear the terminal so you can see what's behind the window. I use it a lot when learning how to run certain code for tasks and the text is in a browser window; I don't need to keep switching back and forth because I can see it all through the terminal :)
I use ctrl+l to clear
Those are great tips.
One other tip to remember... Everyone should know how to use find, especially with -exec. For example: find all the files recursively that contain foo:
find . -exec grep foo {} \;
Or, update the modification date for all files in the current directory and below:
find . -exec touch {} \;
I could go on, but you get the point.
_kirk
Never forget xargs. And if you're using the GNU tools (if you're on a Linux box, or possibly a later OS X, you're using the GNU tools) don't forget -print0 and -0. By using xargs you keep the number of processes down, and by doing so it's potentially faster. And find -print0 and xargs -0 deal with naughty characters in filenames. There are times when you'll HAVE to use -exec, but...
find . | xargs grep foo
find . -print0 | xargs -0 grep foo
find . | xargs touch
find . -print0 | xargs -0 touch
a bit fancier:
find . -type f -print0 | xargs -0 touch
Just a few nights ago, I discovered I had my wonky *n?x-y CD ripper misconfigured for about a month and it hadn't been applying ReplayGain to my MP3s. So I did this:
find . -type f -iname "*.mp3" -mtime 30 -or -ctime 30 -print0 | xargs -0 mp3gain -r -k
it worked beautifully. found a list of files that had been created or modified in the last 30 days, and passed the list to mp3gain.
or when I used a script to create a bunch of files and wanted to add them to a subversion repo:
svn status | grep ^? | sed s/?// | xargs svn add
trust me, it works beautifully.
you can get fancier; check the manpages for details. bonus points if you add sed, awk, grep, etc to the list.
Even more efficient than the above approaches is recursive grep, especially if you know somthing about the type of files you are looking in:
grep --include=*.php -r foo dir1 dir2 ...
List just the directories in the current directory
ls -d */
List directories by size in meg with total as current (.)
du -sm $(find $1 -type d -maxdepth 1 -xdev) | sort -g
Touch is especially useful for creating those annoying verification files for Google WT and Yahoo SiteExplorer, touch google28938234657876.html in the root of your site, and your done :)
is "tail -f" on linux the same as "tail -F" on BSD? ( https://www.freebsd.org/cgi/man.cgi?query=tail&sektion=1&apropos=0&manpath=netbsd )
I think so, I use tail -f all the time on freebsd