Generating Tag Cloud the Unix way

UPDATE: I've modified the script to add color and transparency :)

Ushaft's this tweet on creating tag cloud caught my attention. While there are better tools available online, i decided to give it a try in Linux :)

First we need to generate a frequency list of words out of a file. No brainer. I used the GNU/GPL license as source text.

cat /usr/share/common-licenses/GPL |
tr 'A-Z' 'a-z' |
sed 's/[^a-z ]//g' |
sed 's/\s/\n/g' |
awk '{if (length($1)>=3) print $1 }' |
sort | uniq -c | sort -nr  > freq.txt

Explanation (per line):
1. Print the license
2. Convert to lowercase
3. Remove punctuations
4. One word per line
5. Discard words with 2 or less long
6. Grab frequency and write to freq.txt

freq.txt needs some editing to remove irrelevant words like "the", "but", etc.

Now to create tag cloud, i wrote a simple awk script that outputs svg file. Saved this as gentagcloud.awk (Please change the width, height and scale variable to your taste)

#!/bin/awk -f

BEGIN { 
    WIDTH=1000
    HEIGHT=600
    SCALE=1
    
    OFS=""
    
    print "<?xml version=\"1.0\" 
        encoding=\"UTF-8\" standalone=\"no\"?>"
    print "<svg width=\"",WIDTH,"\" height=\"",
        HEIGHT,"\" version=\"1.1\" 
        xmlns=\"http://www.w3.org/2000/svg\">"
}

{ 
R = int(rand()*9)
G = int(rand()*9)
B = int(rand()*9)

print "<text style=\"fill:#",R,G,B,";opacity:0.75;
    font-size:",$1*SCALE,"px;\" 
    x=\"",rand()*(WIDTH-100),"\" 
    y=\"",rand()*HEIGHT,"\">",$2,"</text>" 
}

END{ print "</svg>" }

Now to create tag cloud,

awk -f gentagcloud.awk < freq.txt > tagcloud.svg

Tada! Tag cloud generated, view tagcloud.svg in your browser or image viewer

If you have ImageMagick installed, you can convert it directly to png too. (Transparency isnt handled properly. If you have workarounds, please comment)

awk -f gentagcloud.awk < freq.txt | convert - tagcloud.png

BTW, the tag cloud was like this: (lil' ugly, but WTH :))

Merging PDF files in Linux

Sometimes you might have to merge two or more pdf files into one single pdf document. While there are commercial softwares like Adobe Acrobat or something similar to do the job, it's much simpler in Linux using commandline. There are several softwares for the job and most of them require simple one line command!

The following commands merge first.pdf and second.pdf to merge.pdf. You can also merge more than two files using similar command.

Using Ghostscript
Ghostscript is a PostScript and PDF language interpreter and previewer. You can do quite handful tasks related to ps and pdf files with Ghostscript; merging happens to be one of them.

gs -dNOPAUSE -sDEVICE=pdfwrite -sOUTPUTFILE=merged.pdf -dBATCH first.pdf second.pdf

Using ImageMagick
ImageMagick is basically an image manipulation software but it also handles pdf files.

convert first.pdf second.pdf merged.pdf

Using PDFTk
If PDF is electronic paper, then pdftk is an electronic staple-remover, hole-punch, binder, secret-decoder-ring, and X-Ray-glasses. Pdftk is a command-line tool for doing everyday things with PDF documents.

pdftk first.pdf second.pdf cat output merged.pdf

Using pdf2ps and ps2pdf
pdf2ps and ps2pdf comes with Ghostscript. The former converts pdf file to ps and latter the reverse. We can use them to first convert pdf files to ps format, merge all ps files and then converting ps file back to pdf.

pdf2ps first.pdf first.ps
pdf2ps second.pdf second.ps
cat first.pdf second.ps > merged.ps
ps2pdf merged.ps merged.pdf


If you are looking for some GUI version, there's a Nautilus script which acts as frontend to PDFTk. If you are on KDE, Kommander script is also available (also depends on PDFTk)


NOTE: To install Ghostscript (already installed in most distros by default), ImageMagick or PDFTk in Linux,

sudo apt-get install ghostscript
sudo apt-get install imagemagick
sudo apt-get install pdftk

Another trick using ImageMagick

Last blog was about batch resizing images using ImageMagick. But ImageMagick is more than just a resizing tool. It's kinda Adobe PhotoShop (or GIMP) for command line. You might be thinkin - what might a commandline photo editing tool be possibly use of? Well, think of resizing, watermarking and putting a nice frame to images. And to make matters worse, there are around 3000 of them and are in different formats (ie, gif, jpeg, png, bmp, etc..). Well, ImageMagick can do them all in a single command line!

ImageMagick has lots and lots of cool features. I already said, it's Adobe PhotoShop for command line and I mean it! More on ImageMagick can be found on its website www.imagemagick.org.

Okay now lets get down to the Magick.. Here's the problem:

Sometimes photos taken by digital cameras are just seem faded out.. Jus like image below.



How do we fix it? Well, the simplest answer is overlay.. This can also be done in Adobe Photoshop (or GIMP). I got this idea from here.

1. Duplicate the image layer
2. Apply layer blending mode as Overlay for the upper image.

But who does the hard work of opening Photoshop (or GIMP) and all the mess when i can do it with a single command using ImageMagick. We use composite command of ImageMagick (single line).

composite -compose overlay 
<faded_image> <faded_image> <output_bright_image>

(plz note that there <faded_image> is mentioned twice as the image is overlayed upon another instance of the same image)

eg,

composite -compose overlay colorless.jpg colorless.jpg bright.jpg

Here's before and after..

Before: Faded Image

After: ImageMagick Trick

Now who says commandline is boring?

Magic of ImageMagick

Sometime back i'd written a blog about batch image resizing in linux. The basic concept of doing that was combining djpeg/cjpeg command in linux. But, well, screw that! I've a better alternative - ImageMagick.

The main drawback of using djpeg/cjpeg command was that it didn't retain the image metadata, aka EXIF (Camera name, date, Exposure, etc). So as a better and flexible alternative, Imagemagick is our tool of the trade here..

To install Imagemagick in linux,

   apt-get install imagemagick

Imagemagick is also available for Windows, Mac, FreeBSD and any platform you can think of as it is Open Source. Go to www.imagemagick.org for more.

Now the command to resize an image is (in single line)

   convert <original_image> -resize <resize_value>
    -quality <resize_value> <output_image>

where
<resize_value> can be like 50% or 800x600.. hope u got the point
<quality_value> can be like 75, 85 (ie, 0-100). This whole quality control can be taken off, in that case ImageMagick automatically sets the best value.

To resize all the images in a folder, we use the following script

#!/bin/sh

mkdir resized

for i in *.JPG
do
   echo -n "Converting $i... "
   convert $i -resize 50% resized/$i
   echo "Done"
done

When this script is run in a folder containing images, it creates a folder named resized and puts all the resized image there, plus the EXIF data are retained too.. Cool huh?