[awk] Display every (n+1)-th line of a file (n=0,1,2….)

As an example, the following displays every first line and every 3rd line after that in a file file.txt:

$ awk 'NR%3==1' file.txt

So, if file.txt has 1 in the first line, 2 in the second and so on, then the above command will display 1 in the first line, 4 in the second, 7 in the third line and so on until it reaches the end of the file.

Credit: Command-line-fu

[awk] Output formatting

If you use print in awk you’ll lose the formatting of the input file — most of the times that may be OK, but not always. The solution is simple — replace print by printf. Here’s an example with strings in the input fields:

$ awk '{printf "%-20s%-20s%-20s\n", $1, $2, $3}' yourfile.dat

.

“%…s” means a character string.

20 means a length of 20 characters.

awk defaults to right-alignment (presumably for columns of figures) so you need -20 for left-alignment.

Credit: here.

[awk] Command line calculator using awk

This is an update on our bash command line calculator posted a few days ago — except for the fact that this time we’ll use awk to do the calculation instead of bc. As I mentioned in that post, you may use python or ruby (irb) to do the same thing, but these tricks may be useful if you don’t  have ruby or python installed (bc and awk, in general, come by default in any Unix or GNU/Linux distro).

First, create (or rewrite if you used our last trick) function “?”  as follows and put it in your ~/.bashrc file:

? () { awk "BEGIN{ print $* }" ;}

and make sure to reload your ~/.bashrc file (do the similar thing if you're using any other shell). [NOTE: ZSH does not like ``?'' as a function, so you might consider replacing it with something reasonable, e.g., ``compute'']


Now, if you want to calculate an expression, do it, for example, as

$ ? "2*3+4.0*(9.9+8.1)"

and don’t forget the quotes.

.

The advantage of this over bc is that you can use more arithmetic and trigonometric functions (link):

              atan2(y,x)     Arctan of y/x between -pi and pi.

	      cos(x)	     Cosine function, x in radians.

	      exp(x)	     Exponential function.

	      int(x)	     Returns x truncated towards zero.

	      log(x)	     Natural logarithm.

	      rand()	     Returns a random number between zero and one.

	      sin(x)	     Sine function, x in radians.

	      sqrt(x)	     Returns square root of x.

.
You may include variables as well in the function definition itself:

? () { awk "BEGIN{ pi = 4.0*atan2(1.0,1.0); degree = pi/180.0; print $* }" ;}

where we have defined the variable pi and degree (such that tan(pi/4.0) = 1.0 and pi radians is equivalent to 180 degrees) to be used later, e.g.

$ ? "cos(pi)"
$ ? "cos(90*degree)"

and I’m sure that you’ll get -1 and 0 (within the machine precision), respectively, as the answer!

(You may find some more interesting calculator related tricks posted in this blog scattered in different pages)

Credit: here and here (via LifeHacker).

UPDATE: I just realized that I can use the calc package (besides bc, awk, python and irb) to do the command line math wizardry more efficiently (and let’s take that as the end to this command line calculator series!). It has a larger set of built-in functions. You may grab the source code from the maintainer’s website and follow my instructions to install it on your system [although binaries are also readily available, e.g. apcalc package for Ubuntu 9.10 (Karmic Koala)].

[awk] Add another column from a second file

The problem: I have  two files, file-1 and file-2, each of which has two columns; and I want to add the second column of the second file as the third column in the first file.
Let’s say the following is the content of file-1:

A   1
B   2
C   3

and that of file-2:

D   4
E   5
F   6

and I want to have something like this in file-3

A   1   4
B   2   5
C   3   6

Of course, the actual data are not as simple as the above!

Solution:

$ awk '{str1=$1; str2=$2; getline < "file-2"; print str1" \t "str2" \t "$2 > "file-3"}' file-1

I inserted the tab characters (\t) just to make file-3 look nice (scientists don’t care about white spaces, do they?)!

Reference: here.

[awk] Standard deviation of a column of numbers in the command line

Just picked this up from my favorite site: commandline-fu.

$ awk '{delta = $1 - avg; avg += delta / NR; mean2 += delta * ($1 - avg); } END { print sqrt(mean2 / NR); }'


Let’s test it by finding the standard deviation of 1, 2, 3, 4, and 5 whose standard deviation is sqrt(2): in BASH, pipe in the out put of the following (which just echoes out numbers from 1 through 5) to the above command to find that it is in fact sqrt(2) or 1.42421
$ for n in {1..5}; do echo $n; done