Dan Harkless' Software
This is a collection of software I've written to solve specific problems.
Some of it is UNIX-specific, but most of it should also work on Windows,
assuming you have ActiveState Perl
and/or Cygwin installed.
Everything is released under the terms of the GNU General Public License.
In addition to the below original software, in the past I have also been a
developer and co-maintainer of GNU wget and nmh. Unfortunately recently I have
not had time to keep up with either project 
, but I'm planning to become
involved in nmh again as soon as I am able.
I've also contributed fixes and/or new features to
Apache,
BIND,
Cistron RADIUS,
CVS,
Douglas K. Rand's syslogd,
dump-contacts2db,
EGD,
GNU
Emacs,
patch, and
wdiff;
LEAF,
MRTG,
Perl modules
CGI,
Cwd,
Image::Size,
Lingua::EN::Squeeze,
and Mail::SPF::Query;
Mozilla Firefox,
pidentd,
Plum Hall C Validation Suite,
sendmail and Milters
dnsbl-milter
and
spf-milter,
Snort,
SSH.com's ssh,
tcsh,
TeX4ht,
trn,
XEmacs,
XFree86 xterm,
and others.
Android
| reformat_contacts2.db_phone_numbers_to_use_parens |
|
The VCF import function in the Android Contacts app reformats (###)
###-#### phone numbers to ###-###-####, which I find harder to read. If
you run this script on your Android device after the VCF contact list is
imported, this will be fixed. |
Audio/Video
Bookmarklets
| wikipedia_expand_all.js |
|
Activating while on a Wikipedia page will cause all the unexpanded
"[show]" sections to be expanded. |
CGI Scripts
| gen_form_search_db |
|
Takes a TAB-delimited database and a database description file in my
proprietary plain-text format and displays a table of the data and/or a
form for searching it. The search form uses checkboxes for enumerated
values and text-entry boxes for arbitrary text fields. You can search for
plain-text strings or Perl
regular expressions, and database fields can be sorted in any order or
omitted entirely. Results can be output one record per row or one record
per column. Check out my Buying
Guides or the "Have" links on my Possessions page to see it in action. |
| GET_to_POST |
|
Converts a GET request to this script into a POST request to the specified
URL, redirecting automatically via JavaScript, if desired. |
| tryout_inlines |
|
Allows you to easily try out a bunch of different alternatives for some
piece of inline media on an HTML page. For instance, if you're not sure
what background graphic you want to use on a page, you can use this script
to try out several. |
DNS
| ifup_nsupdate |
|
Does a TSIG-signed
dynamic DNS update using BIND's nsupdate
command when your network interface comes up (and has a new IP address).
Useful for PPP, PPPoE, and other systems with dynamically-assigned IPs.
Written for use with LEAF
(specifically Bering) and is the main component of nsupdate.lrp, but should also work on normal Linux
installations (particularly Debian) with little or no modification. |
| watch_whois_record |
|
Called as a cron job, periodically checks to see if a given
domain name has become available (or has otherwise changed) by checking for
a particular string in its WHOIS record. |
Emacs
| csh-mode.el |
|
The csh-mode.el and csh-mode2.el available in the old Emacs Lisp Archive are
extremely primitive compared to, say, Gary Ellison's ksh-mode.
When I bemoaned this fact on comp.emacs,
Carlo Migliorini sent me a copy of ksh-mode.el he had done a
search-and-replace on to produce csh-mode.el. I have done
extensive further polishing, resulting in a mode that is even superior to
ksh-mode 2.6 in some ways. I probably should have called this
file tcsh-mode.el, as it has support both for true csh and the
additional features of tcsh. |
Email: Anti-spam
| email_form_data |
|
Yet another web form data emailer along the lines of Reuven M. Lerner's
original 1994 form-mail.pl script (but minus the security holes that so many of
them have). Has some nice features (including anti-spam measures) to make
it more feasible to replace mailto: links with calls to this
script. |
| spfilter2bind |
|
Converts an spfilter-format
DNSBL input file into a BIND zone file. I wrote this
because the spfilter.pl script contained bugs that prevented
BIND from successfully loading zone files created by it, and because I
wanted a much more simple and lightweight option to use in conjunction with
my run_if_modified script. |
| test_DNSBL |
|
Looks up one or more IP addresses on the specified DNS Blackhole List.
Optionally uses monitor_file to keep tabs on
whether the DNSBL stays operational. |
Email: General
| email_address |
|
Takes a full email address, which may include a real name, such as
Dan Harkless <dan@wave.eng.uci.edu>, and
prints just the actual address part -- dan@wave.eng.uci.edu
in this case. |
| rename_to_DATE--SUBJ |
|
Takes a group of mail message files (as from MH/nmh, or .eml
files from Outlook Express) and renames them to be of the form
YYYY-MM-DD_HH:MM[:SS]--SUBJECT.
This was written for the purpose of converting an archive of security
vulnerability emails into a group of files downloadable from a
website. |
| sms_biff |
|
Just as xbiff extends the biff concept (of
instant notification of email arrivial) from tty to X11, this script
extends the concept to SMS messaging. If you set up your email filtering
to call this script on a copy of incoming messages, it'll send a terse
version of each message to your cell phone via SMS. Here's an example of calling this script
from procmail (or see
below for a more complex
example).
|
| treocentral_wmexperts_notification |
|
Takes a TreoCentral.com or WMExperts.com "Reply to post" or "New
Private Message" email on stdin and filters it to contain just
the URL (and subject), so that when an SMS notification of the email is
sent to you by my sms_biff script (or equivalent),
you'll be able to browse to the thread / private message directly from the
SMS message on your Treo. Here's an example of
calling this script in conjunction with sms_biff from procmail.
|
Email: MH/nmh
| compp |
|
A wrapper around MH/nmh's comp for composing PGP-encrypted
messages. Requires medit. |
| find_correspondents |
|
If you use MH/nmh or some other mail system that keeps one message per
file, named numerically, you can use this script to find everyone who has
emailed you at a particular address or addresses. This is useful, for
instance, if you need to inform everyone of a new address. |
| forwp |
|
A wrapper around MH/nmh's forw for forwarding and
PGP-encrypting messages. Requires medit. |
| medit |
|
An emulation of Jim Hester's 1986 medit C program. When used
as your MH/nmh Editor, prompts you for header information and then spawns
your favorite editor to edit the actual body. Also fills out headers
automagically when possible and handles MIME-decoding and PGP encrypting /
decrypting. |
| replm |
|
A wrapper around MH/nmh's repl for replying to MIME-encoded
messages. Requires medit. |
| replp |
|
A wrapper around MH/nmh's repl for replying to PGP-encrypted
messages. Requires medit and an
mhl.replp file. |
| show_JIS_email |
|
A program to be called from MH/nmh's mhshow-charset-iso-2022-jp entry to
view JIS-encoded Japanese emails on AIX (or on other OSes with some
tweaking). Requires Ken Lunde's
jconv program. |
| showp |
|
A wrapper around MH/nmh's show for showing PGP-encrypted
messages. Requires mhl.headers and
mhl.body, which come with
nmh. |
General commandline utilities
| backslashify |
|
This filter puts backslashes in front of characters that are special to
commands that process regular expressions. Can backslashify all extended
regexp characters, [$()*+.?[\^|], or just basic regexp
characters, [$*.[\^]. Especially useful in scripts where an
egrep or sed expression includes a variable whose
contents are unpredictable and may contain special characters
(say, a filename like "c++filt.c"). |
| bgrep.c |
|
Rather than using find . -type f -exec egrep pattern {} \;
-print, use find * -type f -exec bgrep --blank
pattern {} \;. For each file with egrep
matches, a terminal-width boldface banner will be printed
before the matches (rather than afterwards, as with
find's -print), containing the filename,
right-justified. The --blank flag causes a blank line to be
printed between each file, for even more readability. (Another option that
came into existence after I wrote bgrep is to use the -H option in GNU grep
2.1+ to prefix each match line with the filename, as in find . -type f
-exec egrep -H pattern {} \;, but bgrep still wins out on
the readability front.) |
| count |
|
Like sort -u, but with instance counts. Also handy for
sorting IP addresses if you can't remember GNU sort's sort -k 1,1 -k
2,2 -k 3,3 -k 4,4 -n -t . -u syntax. |
| d2u |
|
A wrapper around dos2unix which preserves file ownership,
permissions, and modification time, and which won't mangle binary
files. |
| deduplicate |
|
Traverses one or more file hierarchies, identifying duplicate files by
SHA-3 hash. Optionally, duplicate files can be deleted, or hard-linked or
symlinked to the master copy. |
| fstype.c |
|
Prints what kind of filesystem a file or files live in (e.g.
autofs, cdfs, ext2/3/4,
hfs, jfs, nfs, tmpfs,
ufs, etc.), or else the filesystem magic numbers. |
| grep16 |
|
Greps for a Perl regular
expression in each file specified on the commandline, or stdin if no
files are specified. The regular expression is searched for in each file
both as-is and converted to a UTF-16 string with NULs between each
character. Input files can be big and/or little endian.
|
| longest_line.c |
|
Prints the length (including the trailing \n, if any) of the
longest line in the specified file (or stdin, if none
specified). |
| lookin |
|
Looks in all files in a hierarchy matching a given pattern (e.g.
*.[ch]) for a specified regular expression by having
find call bgrep. |
| mirror |
|
Mirrors a directory tree onto another machine. Unlike rdist,
it builds up a secondary tree and then switches out the old one for the new
one at the last second so that the tree isn't in an inconsistent state
during the transfer. |
| most.c |
|
A pager with several improvements over more -- it properly
displays bold and underlined text, allows backwards movement even on
stdin, completely completely consumes stdin to
allow percentage displayed, and highlights the last line on each page like
trn -m so you don't lose your place on the final, partial
page. less now has
all those capabilities as well, but one important feature most
has that neither more nor less does is
that if you copy & paste a long, wrapping line, it will preserve it
as-is rather than breaking it up into multiple newline-terminated
screen-width lines. |
| total |
|
Takes a column of numbers on stdin and outputs their total on
stdout (simple but very useful). |
| uniq_dirs |
|
Takes a list of pathnames and outputs an alpha-sorted list of all the
unique directories that contain those files. The pathnames should be
newline-separated, and can be given on stdin, or can be specified inside
one or more input files given on the commandline. I wrote this to avoid
the limitations that arise when calling dirname from
xargs. |
Images
| image_album |
|
Generates HTML on the fly for navigation of albums of related images; for
example, online photo albums. Supports several different viewing modes
appropriate for different display sizes and connection speeds, and allows
the viewing of additional image info (such as EXIF data). To see it in
action, check out my online
photo albums, or those of my brother
Steve, brother Patrick, Mom, or Dad.
|
| image_album_prep |
|
Prepares a group of image files for use with my image_album CGI script (or other presentation
methods). Can edit caption files, add comments, restore file timestamps
from EXIF data, create reduced-size copies of images (e.g. for thumbnails),
rename images to prefixNNN.extension,
remove embedded EXIF thumbnails, losslessly rotate portrait shots into the
correct orientation, and more. Some of these abilities require particular
external programs to be available, as specified in the documentation. |
| image_info |
|
A standalone script interface to the Perl CPAN module Image::ExifTool.
Prints out information (such as comments, dimensions, and EXIF fields) for
the image files specified on the commandline. The data is formatted such
that you can easily grep for particular values of particular
attributes in a range of files. |
| image_info_modtime |
|
Sets the modification time for one or more image files to the value of the
embedded timestamp field specified (or the EXIF
DateTimeOriginal field by default). The embedded timestamp
can be interpreted as being in the local time zone or as UTC. |
| rdjpgcom_wrapper |
|
A wrapper around rdjpgcom
that allows the specification of multiple files on the commandline and that
prints each comment with a prefix of "filename: ".
If a file has no comment, "No JPEG comment" is printed
(rdjpcom is simply silent on such files). |
System administrator utilities
| access_log_domains |
|
Outputs all the unique domains found among website visitors in an Apache
access_log. For each one, outputs the hostname and timestamp
of the last visitor from that domain, and optionally the hostname /
timestamp of the first visitor as well. |
| booby.c |
|
A little boobytrap. If you find an evil binary on your system, like some
kind of cracker tool, and rather than deleting it you want to be notified
next time someone uses it, you can move it to some odd location and replace
it with an executable compiled from this file. The wrapper will do a
little snooping and then run the real executable as if nothing's amiss.
Strings used by the program are obfuscated so it's not easy to determine
the executable's nature with the strings command. Basically
you're taking an unimportant known-compromised machine and transforming it
into a poor man's honeypot. |
| check_df |
|
A script you can call from cron to output (and thus email) a
df listing if any of the filesystems' usage is greater than or
equal to the percentage you specify on the commandline. |
| crypt8 |
|
Prompts for a string to run through crypt() and prints the
result on stdout. This can come in handy if you need to
change a password but you can't (or don't want to) use the
passwd program, e.g. because hackers have broken on to your
system and deleted it. You can also manually specify a salt, e.g. to
verify if someone's password is set to a given value without having to
attempt a log-on. |
|
|
|
A shell account customer wanted to know if he could have access to his
http://shell_machine/~user log files,
and I was surprised to discover that Apache doesn't give you any way to set up
per-user logs. The best solution would be a module that would enable this,
but this program was easier to write. It writes the logs as
user_access_log and
user_error_log in a directory you specify, which
users should have read-access to but not write-access to. |
| monitor_file |
|
Monitors a file (or pseudo-file, like a /proc entry) to watch
for changes. If there are any, diff output will be displayed,
or sent to the email address specified. |
| rpm-V_all |
|
Runs 'rpm -V' on all packages installed on the system, prompting the user
for their desired action when any differences are found. Actions include
getting package info, reinstalling, reverifying, uninstalling, and
ignoring. |
Timestamps
| run_if_modified |
|
Examines the modification timestamp on a specified file, then runs a
specified update command. If the update command has modified the
timestamp, a second specified command will then be run. One example of
where you might use this is in a cron job that periodically
retrieves a DNS zone file via rsync or wget (such
as a DNS
Blackhole List that doesn't offer traditional zone transfers), then
restarts your nameserver if the zone has been modified. |
| timepreserve.c |
|
Preserves the access and modification timestamps of one or more files
across the execution of a commandline that would otherwise modify
them. |
Windows utilities
| MBProbe_critical.sh |
|
A Cygwin shell script to
be called from MBProbe when
measurements go into the Critical zone. Quickly and unceremoniously shuts
down the machine. This script is necessary because on most machines
MBProbe's builtin Shutdown feature only shuts down to the "It is now safe
to shutdown your computer" screen, rather than powering off. |
| MBProbe_warning.pl |
|
A Perl script to be called from MBProbe when measurements go
into the Warning zone. Emails you the first and last few lines from
MBProbe's Event and History logs. |
| reverse_lookup |
|
Does NetBIOS and DNS reverse lookups on a list of IP addresses. |
In addition to the above, I've also squirrelled away some old
software that used to be on this page but is now either obsolete or just
no longer of use to me in my current computing environments. A few things
there might still be of use to someone, though, particularly if they're on an
old system.
Dan Harkless
Page created: August 21, 1997