find all gzip files, print a list of files, remove duplicates and strip off the '.gz' extention

I know, this looks long, but if you break it down into each part and execute you can quickly see how it works. It's really not that bad. In this example we are processing gzip files, but this could be used on any kind of file.

find . -name *.gz | perl -p -e 's/^\.//g' | cut -d'/' -f3-15 | sort | uniq | perl -n -e 'chomp; split("/"); print @_[$#_]."\n";' | perl -p -e 's/\.gz$//g'

Part 1) find the files:

find . -name *.gz

Part 2) remove the '.' from the beginning of the files added by find. (you don't need this if you specify the full path):

perl -p -e 's/^\.//g'

Part 3) break the path strings on the '/' char and print fields 3-15 (cut is very useful command):

cut -d'/' -f3-15 (note you file will have different field numbers)

Part 4 & 5) we've seen these before sort and remove duplicate entries:

sort | uniq

Part 6) get the actual filename from the path:

perl -n -e 'chomp; split("/"); print @_[$#_]."\n";'

Part 7) remove the '.gz' extention from the name:

perl -p -e 's/\.gz$//g'

put them all together and you have a really long command line that really should be made into a shell script.

Commands used in this post: find, cat, sort, uniq, cut, perl