Adobe hastens release of HTML5 developer tool

  

Adobe hastens release of HTML5 developer tool | Deep Tech - CNET News: http://bit.ly/cIRi0n

Mozy online backup gets faster and goes local

Mozy online backup gets faster and goes local | Deep Tech - CNET News: http://bit.ly/aSfKCF

XProc: An XML Pipeline Language - spec finalized

XML workflow spec finalized - "XProc: An XML Pipeline Language" - : http://bit.ly/9VheAX

HTML5 Videos with Multi-Browser Support

Embed HTML5 Videos with Multi-Browser Support on Your Site - Streaming Video - Lifehacker: http://bit.ly/bZyC9k

Video on the Web - HTML5

Video on the Web - Dive Into HTML5 - Getting your video ready for HTML5: http://diveintohtml5.org/video.html

Amazon Adds Private Streaming to CloudFront

Amazon Adds Private Streaming to CloudFront - CIO.com - Business Technology Leadership: http://bit.ly/d5n3iL

Developers Benefit From New Apprentice Scheme

Future Developers Benefit From New Apprentice Scheme: http://bit.ly/cVjF4t

HTML 5: Less than it's cracked up to be

Opinion: HTML 5: Less than it's cracked up to be: http://bit.ly/bnuoki

How to view the sizes of a list of directories?

How to look at the size of a list of directories. From time to time you need to look at the sizes of different directories to see how much space they are taking up. Here's a quick easy way to do this. Combining the ls command with the du command.

prompt$ ls -d */ | xargs -n 1 du -hs

This will give you output that looks like this:

-bash-3.00$ ls -d */ | xargs -n 1 du -hs

353M temp

301M www

135M www2

310M wine

299M junk

1.6G video

86M cgi

6.9G podcasts

Now it's easy to see where all the space is being used.

You can also add this to your profile and make your own command like this:

alias lsdir='ls -d */ | xargs -n 1 du -hs'

How to read a Foxpro database with Perl?

I recently had a friend ask me to help them read a FoxPro (dbf, cdx, fpt) database. The only problem was neither of us had FoxpPro or anything that would open and read the file. All we wanted to do was read the data and output plain text. I searched the internet for a free tool to help with this task and I found zilch!!!

I did find things that would let you read the first 50 records, but if you want more you need to buy the product. I found others that places wacky watermarks all other the data until you bought the product. I did some more searching and found the answer. Perl to the rescue!!!

I found a (free) Perl library called XBase that lets you read dbf files with cdx indexes.

First I had to install the XBase perl module. I did this on my iMac like this:

$> sudo perl -MCPAN -e shell

cpan> install DBD::XBase

Once that process was complete, the rest took about 10 minutes.

Here's a sample of the script:

#!/usr/bin/perl

use strict;

use XBase;

my $table = new XBase "MYTABLE.DBF" or die XBase->errstr;

my $cursor = $table->prepare_select("FNAME", "LNAME", "MEMO");

my ($fname, $lname, $memo, $count);

$count = 0;

while ( ($fname, $lname, $memo) = $cursor->fetch) {

print "$count\t$fname\t$lname\t$memo\n";

$count++;

}

That's it! This will print out a tab delimited list of the contents of the database.

Searching for a house?

This site has a lot of really cool features and the search capability is really cool.

Checkout the site: http://www.redfin.com/home

How to listen to you iTunes music on any computer

Hi Folks,

I found this very simple easy way to listen to all the music on my home Mac while I'm at work.

It's called Simplify Media (http://www.simplifymedia.com/). It's free and easy to use. You can also invite up to 30 people to listen to your music and vice versa. I should mention that this is not a way to copy music, just listen. I enjoy listening to all the music on my home computer while I'm at work or on the road (there is also a free iPhone app). It works on Mac's and PC's.

Enjoy!!!

filtering find with egrep

Today I found that I need to find files of certain types within a directory structure. there are many kinds of files in my structure, but I only need to find the mp3, mp4, and file with the number 384 in the name. Here's how it did it:

> find . -type f | egrep "mp3|mp4|384"

This will find all files in the local dir structure and filter the output printing only the files that contain one of the strings in my egrep expression.

this is a nice easy way to find files of different types or names in one command.

How to get the current system date in Oracle?

Getting the current system date:

SQL> select sysdate from dual;

How to find the day of the month:

SQL> select extract(day from sysdate) from dual;

You can also say extract month and year the same way.

Give it a try.

Some tips for making Firefox faster

I find that my Firefox just isn't that fast these days. These quick tips seemed to help a little.

The first can be found here at forevergeek.com: Make Firefox faster

The second is can be found here at lifehacker.com: Vaccum your Firefox database

I'll post more as I find and test them.

How to run PC apps on your intel Mac?

I recently had a need to run some PC apps on my iMac. I know thee are many ways to do this, but I came accross this simple program called Crossover (http://www.codeweavers.com/products/cxmac/) that really makes it easy without the huge overhead of running a VM. I don't think this would be good for all programs, but it might just be what you need. It's fast and easy to setup. They also have a version specifically for PC games. I haven't tried this but I do have a few games that are beckoning to be tested. :)

JSLint

Hi Folks,

I just found this really cool tool for cleaning up Javascript code.

I found it very easy to use and I was surprised by what it found.

Check it out: http://www.jslint.com/

Using awk to parse apache log files!

I found this cool page that has some good examples of using awk to parse apache log files: http://bit.ly/UdxKF

You can extrapolate these examples to any other kind of text file.

Have a look, and have some fun!!! 

How to create a zipped tar file?

This is very similar to creating a tar file. The only difference is you change the file name and add a new command line option. In the last post we created a tar file of all the html files in a directory. Let's do this again only this time, we'll create a gzipped tar file.

/my/web/root> tar -cvzf mywebroot.tar.gz *.html

Note the new option (z) and the new file extention (.tar.gz)

The ability to create zipped tar files is a handy convenience, otherwise you would find yourself zipping up the tar files in an extra step. It's just easier to do it this way.

To view the contents of a zipped tar file is just as easy:

/my/web/root> tar -tzvf mywebroot.tar.gz

Give it a try!

How to create a tar archive file?

Creating tar files is essential for certain processes. It's an easy way to package up files and directories for moving on the file system or storing on tape. The simplest form of a tar file contains just files and lives as a file itself. Here is an example of that:

Say you have a directory full of html files, a web site perhaps with no directories. This is uncommon but will show a very simple tar file.

First cd into the directory: cd /my/web/root

/my/web/root> 

If you list the contents of the directory using ls, you might see something like this:

index.html one.html two.html bio.html about.html

Now let's create an archive of these files.

/my/web/root> tar -cvf mywebroot.tar *.html

This will create a file called mywebroot.tar that contains all the html files in the current directory. 

To view the contents of the file type:

/my/web/root> tar -tvf mywebroot.tar 

You will see a list like this:

-rw-r--r-- jsmith/www 2057134 2009-04-16 18:11:33 index.html

-rw-r--r-- jsmith/www       1947 2009-04-16 18:11:33 one.html

-rw-r--r-- jsmith/www         134 2009-04-16 18:11:33 two.html

-rw-r--r-- jsmith/www           57 2009-04-16 18:11:33 bio.html

-rw-r--r-- jsmith/www         713 2009-04-16 18:11:33 about.html

Tar stores information about the files as well as the files. You get the permission of the file the date stamp the owner and group as well as the size. 

How to encrypt and decrypt files on the command line?

I just found a great article on ghacks.net that explains how to do this. 

I though I'd just pass the url along, I shortened it using micURL: http://micurl.com/ynh0vz

Enjoy!!!

How to make many levels of directories easily?

This does happen from time to time. You need to create a many levels of directories on your system like this:

/someroot/dir1/dir2/dir3/dir4

You could use mkdir (make directory, see man mkdir) to create them one at a time, then cd into the one you created and create the next one. That would look like this:

> mkdir someroot

> cd somroot

> mkdir dir1

> cd dir1

You get the picture. That's a lot of typing and a waste of time.

Fortunately there is an easier way to do it. The mkdir command has a switch '-p' that will create intermediate directories if they don't already exist. Using the '-p' switch our command would look like this:

> mkdir -p /someroot/dir1/dir2/dir3/dir4

That's much better than creating them one at a time.

By the way, I should mention that cd is the command used to change directories.

See the man page for cd (man cd).

Need to prefix lines of a text file or add a new first column to a log file?

This easy but handy script will add a new column to your file. In this example the column will contain the text 'someserver-(n)' the (n) will be numbers starting from 1 to the number of lines in the file. See the output at the bottom. This is useful if you ever have to combine like files, log files are a good example.

Put this text in a file called addcolumn.pl.

#!/bin/perl

my $prefixtext = "someserver";

while ($x=<>) {

$y++;

print "$prefixtext-$y $x";

}

Now make the perl script executable like this:

> chmod +x addcolumn.pl

Now you can do something like this. Let's say you have a log file called access.log (could be an apache log file for example).

> cat access.log | addcolumn.pl > newaccess.log

the output of this script looks like this:

someserver-1 rest of line...

someserver-2 rest of line...

someserver-3 rest of line...

and will be captured in a new file called newaccess.log

Using grep to find lines that don't match your pattern!

This is actually a pretty handy thing to know. I use it to parse config files for shell scripts. The cool thing is it's very easy to do. Let's say you have a config file called my.conf that has two different kinds of lines. Lines that contain things you want and comments. 

It may look like this:

# Users home directory

HDIR='/home/users/jsmith'

# users current directory

CDIR=`pwd`

There may be reasons that I only want to know about the command lines and ignore the comments. To do this I can use grep like this:

user$ grep -v \# my.conf

output:

> HDIR='/home/users/jsmith'

> CDIR=`pwd`

Think you have a virus on your PC?

I know this is a bit off topic, but I thought it would be good to cover anyway. Getting a virus on your computer is a bummer. And yes it's still possible to get a virus even if you have antivirus software installed. It's not enough to have it installed you have to make sure the antivirus database updates itself on a daily basis. Here are (in my opinion) the most important things you can do to protect yourself from getting infected.

1) The most obvious is run antivirus software. There are several free tool available for people that will do the trick. Take a look at the virus section on this site: http://www.filehippo.com/. Personally I've used the AVG free home edition and like it, but I think any of these is better than nothing. Again, the most important thing here is to make sure it's set to receive automatic updates. 

2) You should also be running some kind of anti-spyware program. You can also find them on the site above. I have used Windows Defender it seems to work well, but again any of these is better than nothing. (See this article from gHacks on Windows Defender). Make sure it's set to receive automatic updates. 

3) Make sure you run windows update periodically. This can and should be configured to run automatically. 

4) Be very careful when downloading software from the web. It's very tempting to download free cool looking software, just make sure it's coming from a reputable place. 

5) Sometimes people like to disable the virus software they have while they are running certain programs like games. This is ok, just remember to turn it back on again if you start browsing the web and downloading files. A few programs allow you to disable for a certain number of minutes or hours then it will automatically startup again. I believe Norton Antivirus has this feature. 

6) Another thing that I've seen is people download crappy software and their PC starts running real slow. it's not necessarily a virus, it could just be the crappy software or maybe your PC is underpowered to run the software. Be sure to check the hardware recommendations of software you run.

This is a good time to point out another cool sites. If you think you have a virus and your virus software can't find it there are some things you can try. 

1) Download and run McAfee Avert Stinger. This is a standalone tool that can find and remove specific  viruses. 

2) You can also try to download and run another virus program. I would recommend doing this on from a cd or thumb drive if possible. There is a cool site called http://portableapps.com/ that you can download a fully functional version of ClamWin Portable. All the software from this site can be run from a thumb drive. 

3) Finally here a links the Microsoft windows XP Security center page and the Microsoft Windows Vista Security Center page. I would recommend reviewing these for your system.  

How to connect to other systems securely using ssh?

There are many ways to connect to other systems. In this post I will talk about the method I use most often, ssh. Ssh is short for secure shell. Ssh is a replacement for older unsecured connection protocols such as telent or rlogin. I wont get into the history of ssh or how it works. You can look that up in many places. Instead, I'll point you at the man page for your version of ssh (man ssh).

There are commercial versions and free versions and different ways of connecting using ssh. The main thing you need to know is that it's a secure protocol and most systems have it installed. If they don't you might want to install it or go to a system that does have it installed. Also, There usually is a corresponding sftp protocol for transferring files securely. I will say this, ssh and sftp are preferred (and a lot of times required) over their older telnet and ftp unsecured ancestors.

The first time you connect to a system using ssh you will get a message like this:

somehost$ ssh someuser@someotherhost.com

The authenticity of host 'someotherhost(1.2.3.4)' can't be established.

DSA key fingerprint is 12:cf:13:5:c7:t5:gb:bg:6y:4h:34:56:6d. (fyi - this is not the real fingerprint I change it)

Are you sure you want to continue connecting (yes/no)?

(Then next time you will not get this message just a shell prompt)

Type yes and hit return, you should see:

Warning: Permanently added 'someotherhost(1.2.3.4)' (DSA) to the list of known hosts.

someuser@someotherhost.com's password:

Enter the password.

Now you will see a prompt on the other host:

someotherhost$

You are now connected to the other host and can type commands just like you can on the originating system.

I can't say enough about man!

With all the commands I talk about in this blog. I want to encourage you to check out the man pages of the command. There is so much more information in the man pages. It can seem overwhelming at times. The more you look at them, the easier they are to read.

Here are some examples:

> man man (yes, you can do this)

> man apropos

> man more

> man ls

> man httpd

Give it a try with other commands. 

How to find a command when you're not sure what it's called?

There is a very useful command called 'apropos' that will help you find a command based on a keyword lookup. So if your nor sure of the command or you want to find other commands that might be similar, apropos will help you. Here's an example:

> apropos more

bzless          bzmore (1)      - file perusal filter for crt viewing of bzip2 compressed text

bzmore          bzmore (1)      - file perusal filter for crt viewing of bzip2 compressed text

gzmore          gzmore (1)      - file perusal filter for crt viewing of compressed text

less            less (1)        - opposite of more

let             let (1)         - shell built-in function to evaluate one or more arithmetic expressions

more            more (1)        - browse or page through a text file

... ( this is a long list so I limited it) 

Another example:

If you type:

> apropos apach

you get a long list that includes:

apachectl(8)             - Apache HTTP Server Control Interface

apxs(8)                      - APache eXtenSion tool

httpd(8)                    - Apache Hypertext Transfer Protocol Server

logresolve(8)            - Resolve IP-addresses to hostnames in Apache log files

mod_perl2(3pm), mod_perl(3pm) - Embed a Perl interpreter in the Apache/2.x HTTP server

rotatelogs(8)            - Piped logging program to rotate Apache logs

All these commands relate to apache.

Play around with it and check out the man page.

How to see which apache modules are installed?

If you are using some kind of a unix system you probably have apache running somewhere. Sometimes you need to know which apache modules you have installed. There is a quick way to do this. 

To view the compiled modules:

> httpd -l 

Compiled in modules:

  core.c

  prefork.c

  http_core.c

  mod_so.c

To view the loaded modules:

> httpd -M 

Loaded Modules:

 core_module (static)

 mpm_prefork_module (static)

 http_module (static)

 so_module (static)

 authn_file_module (shared)

 authn_dbm_module (shared)

 authn_anon_module (shared)

 authn_dbd_module (shared)

 authn_default_module (shared)

 authz_host_module (shared)

... (this list is very long so I'm only showing the top 10)

When's the last time you backed up your data?

I'm going to make this easy for you. There is no more important thing you can do than backup your data. Both at home and at work. I'm less concerned about the office unless your self employed or run a small business. You should however make sure your office has a backup policy and/or a process in place that is regularly checked. If not you may want to think about coming up with your own (check with your boss first, there may be rules surrounding backing up company data). for home and small business users there are several easy and cheap ways to backup your data. At home I use an online backup service called Mozy (www.mozy.com). Mozy is very inexpensive (that's just my opinion) and I can store unlimited amounts of data on their systems. At the moment I have approx. 80 GB backed up there now. There are many of these types of services out there at very reasonable costs. Just Google 'online backup' and you'll see all kinds of services. PC Magazine did a comparison of these services, here's a link to the article. 

Another alternative is to buy an external hard drive and simply copy your data over. I have an iMac at home so I use an external hard drive with Time Machine in addition to my Mozy service. You can never have enough backups. There are many software products out there that can help you manage your backups for the PC, MAC, LINUX, and UNIX. 

 

How to look at individual columns of a tab delimited file?

We have already seen this in action. The command is cut and it can be used to look at the individual columns of a tab delimited file like so:

> cut -f1,2,5,7 somefile.tab

You can in fact look at any text file using cut. There is a switch that lets you specify the delimiting character. For example let's say you wanted to look at a coma delimited file. The command line would look like this:

> cut -d',' -f1,2,8,15

Give it a try!

10 Good UNIX usage habits from ibm.com

I found this page on the ibm.com site that lists 10 good UNIX usage habits. It's worth a read, there's some goods tips in here.

Learn 10 good UNIX usage habits

I don't have Unix, Linux, or OSX what can I do?

You do have some options. Here are two options that will get you going fast on a PC running the windows operating system. The first is Cygwin a windows program that emulates a linux environment ( I used this for years when I had a windows PC at home). The second is Knoppix. Knoppix is a full blown linux operating system that can be run from a CD or DVD leaving your existing windows drive untouched. 

Here are links to these systems:

Cygwin 

Knoppix

You can also use a program like VMWare to run other unix/linux distributions on your PC.

Here is a guide to installing linux on your PC with VMWare. 

Depending on your hardware it can be tricky to install linux on a PC. I highly recommend that you backup your system before attempting any kind of install. 

How do I list the size of directories?

Sometimes it's really convenient to know how much storage space is being used by users or even your own directories. Here is an easy way to figure this out. Here we are using a new command 'du' which shows us disk usage for a given directory. When we chain this with ls -d and just look at directories we can tell how much space each directory is using.

ls -d */ | xargs -n 1 du -hs

The output looks like this:

12M dir1

1.7G dir2

726K dir3

219M dir4

Here's another trick. You can add this as an alias to your shell. Add this line to your shell's personal settings file (this is .bashrc or .profiles or ....)

alias lsd='ls -d */ | xargs -n 1 du -hs'

Note - you can call it anything you like. I just use lsd because it's like ls for directories.

Now you can just type lsd when you want to run this.

How to look at the contents of a gzip file without unzipping the file?

Sometimes you need to look at the contents of a gzip file or files without uncompressing the files. You can do this with gunzip. There is a switch in gunzip '-c' that allows you to stream the contents of the file to the screen without first unzipping the file. Here's an example:

> gunzip -c somebigzipfile.gz

The contents of the file will print to the screen. This can be combind with other tools such as grep, cut, and more. Here are some examples:

> gunzip -csomebigzipfile.gz | grep 'the brown fox jumps' 

This will only print lines that contain the text 'the brown fox jumps'.

> gunzip -c somebigzipfile.gz | more

This will show you the lines in the file one page at a time allowing you to move through the file at you own pace. 

> gunzip -c somebigzipfile.gz | cut -f1,2,3 | more

This will only look at the first three fields (assuming this is a tab delimited file) one page at a time. 

How to find files that do not look like ...?

I know we have covered finding files, but it occurred to me that sometimes I need to find files that don't match a pattern. For example I have drives full of zipped files that all end in '.gz'. These files are placed there via a process that runs on a schedule. Sometimes the files are not zipped for whatever reason. So I need to find the files that are not zipped. That is all the files that don't have the extension '.gz'. When I find them I want to zip them. Here's how we can use the find command to accomplish this task:

First lets see which ones are not zipped:

find . ! -name '*.gz' -print (this will print out all the files so I can take a look at them)

Now lets find them and zip them

find . ! -name '*.gz' -exec gzip {} \;

The '!' tells find to locate files that DO NOT match the pattern *.gz. Then the exec part calls gzip to zip up the files.

Commands used in this post: find, gzip

How do I make my script executable?

Say you created a fancy shell script and want to be able to execute it just like a regular command. All you need to do is change the file access permissions on the script. Sounds more complicated that it really is. The command to use is called chmod. You can get fancy with the command (like many unix commands). In our case we are going to keep it simple. 

Say you have a shell script called dosomething.sh . If you run ls -l on the file it will look something like this: 

-rw-rw-r-- 1 jsmith users 0 2009-05-12 19:27 dosomething.sh

Now let's change the permissions and make the file executable:

> chmod +x dosomething.sh

Now if we run ls -l we get:

-rwxrwxr-x 1 jsmith users 0 2009-05-12 19:29 dosomething.sh

See how the 'x' was added to the third place in each section. I should mention that there are three sections to file permissions, Owner, Group, Everyone. Each section has three settings read (r), write(w), and executable (x). We'll cover this more in another post. 

Commands use in this post: chmod, ls

How do I know what directory I am in?

If you're like me, you are in and out of so many directories all the time you forget which directory you are in. Fortunately, there is a command that will tell you exactly that. It's called pwd, that's short for print working directory. At the command line just type pwd like this and it will print out where you are:

> pwd

/usr/users/eparker/some/dir/

We can build on what we just learned about shell scripts here. If we wanted to have a script become aware of what directory it's in we would do something like this:

#!/bin/sh

DIR=`pwd`

echo $DIR

Give that a try.

Commands used in this post: pwd

Shell scripting!!!

If you are going to spend anytime using unix, linux, or OSX you may have the need to write a shell script or two or three.... Shell scripts are very handy for doing all sorts of tasks that are either time consuming or need to happen routinely via cron (we'll cover cron in a later post). I am not going to go into the "how to" of shell scripts. That topic has been covered numerous times on the web in much more detail and knowledge than I have. Here are some links to sites that I think are a great place to start learning shell scripting. There are plenty of examples and code snippets in these links. 

Advanced Bash Scripting Guide

Linux Shell Scripting Guide - A Beginner's handbook

Shell Scripting Primer (Apple OSX)

Wicked Cool Shell Scripts

Get to know Linux: Bash Scripting Basics

I will give you an example of how you might want to use a shell script. 

Let's say you constructed a really long (and not so friendly) command line and you don't want to have to recreate it each time you need it. For example let's take one that we used in an earlier post:

find . -name *.gz | perl -p -e 's/^\.//g' | cut -d'/' -f3-15 | sort | uniq | perl -n -e 'chomp; split("/"); print @_[$#_]."\n";' | perl -p -e 's/\.gz$//g'

Remember that one?

We can put this in a shell script, here's what that would look like.

Create a file called myscript.sh and paste the following text into it:

#!/bin/sh

# Remember this is all one line not two

find . -name *.gz | perl -p -e 's/^\.//g' | cut -d'/' -f3-15 | sort | uniq | perl -n -e 'chomp; split("/"); print @_[$#_]."\n";' | perl -p -e 's/\.gz$//g'

Next make the script executable (we'll cover what that means later):

> chmod +x myscript.sh

Now you can just type ./myscript.sh to execute the script. Note the './' before the script name. We need this here because the shell we are running (where you see the command line) needs to know where the file is. I'll cover that later, for now just have fun running shell scripts. 

Commands used in this post: find, cat, sort, uniq, cut, perl, chmod

How to find out more about a command?

There is a command called man that will let you view the man (online manual) page for a command. Just type man and you will see a screen that shows info about the command you listed.

Here are some examples:

> man ls (show info about the ls command)

> man man (show info about the man command)

I should mention that man has different sections. When we use it like this we are looking at section 1 which is for general commands. There are other sections for system calls and c library functions that I wont get into in this post. Here's one example of how to look at the different sections:

> man 3 fscanf (section 3 is for c library functions)

Commands used in this post: man

how to pipe data to a program that only allows command line input?

This come sup all the time because allot of programs don't allow you to simply pipe data to them. Remember when we pipe data we use the "|" symbol to chain commands like this: sort | uniq. When we have a program that doesn't allow this we can use a program called xargs. xargs allows you to take the output from one program and pass it to another program as a command line argument. I have used xargs in examples of other blog posts. Here is how it works:

let's say I want to pass the names of all the files in a directory to a progrma that only takes input from the command line. I would do the following:

> ls | more | xargs -n 1 someprogram

This will take the file names and pass them to the program someprogram on at a time.

Commands used in this post: ls, more, xargs

How to find a command that you used before?

I use this all the time because I can't seem to remember anything these days.
There is a command called "history" that will print out the last 500 (I think) commands that were entered on the command line.

Just type as follows:

> history

You will see a list like this:

737 cd
738 history > eric_history.txt
739 more eric_history.txt
740 wc eric_history.txt
741 history
742 ls
743 more eric_history.txt
744 more eric_history.txt
745 ls
746 who
747 fi
748 date
749 ls
750 history

The cool thing is when you see the command you need you can run it by typing ! followed by the number. For example, say I wanted to run line 743 above.
I would type:

> !743 ( This is very handy.)

From time to time I backup my history file using something like line 738 above.

Commands used in this post: history

Using ls and cut to list files with a twist

Suppose you need to list files without the extension?
There are too many reasons why you might want to do this and it turns out there's a very easy way to do it. Suppose you have a directory with a bunch of files some called xyz.html and others called xyz.html.new. See examples below followed by output:

> ls
> xyz.html
> xyz.html.new

> ls | cut -d'.' -f1
> xyz
> xyz

> ls | cut -d'.' -f1,2
> xyz.html
> xyz.html

You could take this further and pipe it to sort | uniq like so:

> ls | cut -d'.' -f1,2 | sort | uniq
> xyz.html

These types of commands can be very useful for working with all kinds of files and data.

Commands used in this post: ls, cut, sort, uniq

how to look at the last n lines of a log file?

This comes up everyday. It's very useful to be able to look at the last n lines of some log file. I use tail for this with the -f option like this:

> tail -50f myfile.log

This will tail the last 50 lines of the file. The -f option without a number only looks at the last 10 lines of the file. Adding the extra lines makes it easier to find what you're looking for.

Commands used in this post: tail

Another way to find text inside gzip files.

zgrep is another program you can use to find text inside gzip files. I've only seen it available on linux but you may find a port for other platforms. zgrep is a very handy tool for searching through gzip files. It works pretty much the same way as grep only you don't have to unzip the files.

Example:

Let's say I was searching for an IP address in apache log files.

> zgrep -e '1.2.3.4' *.gz

You can also use the -l option just like grep.

Commands used in this post: zgrep

Searching for text in gzip files?

If you have a bunch of files (say log files) that all have a standard naming format.

Here is a sample list, I wont list them all but you get the idea:

access_04-01-09.gz
access_04-02-09.gz
access_04-03-09.gz
access_04-04-09.gz
access_04-05-09.gz
...
access_04-30-09.gz

And you only need to select say the 10th through the 20th. You could list them all out but that's not really efficient. A better way is to do something like this:

Say you want to examine the contents of these gzip files.

gunzip -c access_04-[10-20]-09.gz | grep 'some text to search for'

This would look at just the files in the range 10-20.

You could also say something like this:

gunzip -c access_04-[01,03,15]-09.gz | grep 'some text to search for'

This would look at 01,03, and 15.

Or look at all the access files like this:

gunzip -c access_*.gz | grep 'some text to search for'

Commands used in this post: gunzip, grep 

find all gzip files, print a list of files, remove duplicates and strip off the '.gz' extention

I know, this looks long, but if you break it down into each part and execute you can quickly see how it works. It's really not that bad. In this example we are processing gzip files, but this could be used on any kind of file.

find . -name *.gz | perl -p -e 's/^\.//g' | cut -d'/' -f3-15 | sort | uniq | perl -n -e 'chomp; split("/"); print @_[$#_]."\n";' | perl -p -e 's/\.gz$//g'

Part 1) find the files:

find . -name *.gz

Part 2) remove the '.' from the beginning of the files added by find. (you don't need this if you specify the full path):

perl -p -e 's/^\.//g'

Part 3) break the path strings on the '/' char and print fields 3-15 (cut is very useful command):

cut -d'/' -f3-15 (note you file will have different field numbers)

Part 4 & 5) we've seen these before sort and remove duplicate entries:

sort | uniq

Part 6) get the actual filename from the path:

perl -n -e 'chomp; split("/"); print @_[$#_]."\n";'

Part 7) remove the '.gz' extention from the name:

perl -p -e 's/\.gz$//g'

put them all together and you have a really long command line that really should be made into a shell script.

Commands used in this post: find, cat, sort, uniq, cut, perl

How to convert a tab delimited file to a csv file?

This is a simple Perl script to convert a tab delimited file to a csv file.

!/usr/bin/perl -w

while(<>){
s/\t/","/g;
chomp;
print "\"" . $_ . "\"\n";
}

Place this code in a file and cat another file to it lie this:

> cat somefile.tab | perl tab2csv.pl > somefile.csv

You could also do this with perl from the command line like this:

cat somefile.tab | perl -p -e 's/\t/","/g' > somefile.csv

I put it in a file so I don't have to keep writing the line above.

Commands used in this post: cat, perl 

Now what, you found the files you wanted?

If you used the find command in the last post to find files and direct them to a file like so:

find /some/root -name '*.*' -exec egrep -l -e 'string to find' {} \; > files.txt

you will now have a file that contains a list of files with full paths.

/some/root/file1.xyz
/some/root/file2.xyz
/some/root/file3.xyz

All these files contain the pattern you searched for in the egrep command. There may be duplicates in the list of files. You can get rid of them like so:

cat files.txt | sort | uniq > files2.txt

To see the files output on the screen just leave off the '> files2.txt'

these commands put together with the pipe '|' list all the files, then sort them, then remove the duplicates and redirect the clean output to a new file. This is very very useful for all sorts of applications. I probably use these commands several times a week.

You can also cat the files to some other script like so:

cat files.txt | xargs -n 1 perl myperlscript.pl

Here I use the xargs command to take the files names from the cat and turn them into command line arguments in the perl script.

Commands used in this post: find, cat, sort, uniq, egrep, xargs, perl

Finding files and files that contain some string

How to find files is a very basic thing to know on a unix system. Fortunately it's very easy using the find command. Here's an example of how to use the find command to find log files for instance: 

find . -name '*.log' -print

This will look through all the directories using the current directory as the root for files that match the pattern *.log. It will print them out with paths relative to the root. for example the output of the command above might look like this:

./dir1/dir2/access.log

./dir1/dir3/dir4/temp.log

./dir5/dir6/error.log

If you put the full path to the root in the find command like this:

find /home/users/jsmith -name '*.log' -print

The out put will look like this:

/home/users/jsmith/dir1/dir2/access.log

/home/users/jsmith/dir1/dir3/dir4/temp.log

/home/users/jsmith/dir5/dir6/error.log

Now let's say that you want to find only files that contain a particular string.

This is relatively easy thing to do. Here's one way to do it, I'm sure there are more.

find . -name '*.*' -exec egrep -l -e 'string to find' {} \;

This will give you a list of file names. To see the actual line in the file remove the -l switch.

Click here to see more info about egrep.

Here is a great page that has lots of examples of the find command.

Commands used in this post: find, egrep