NAME - Read a list of email file specs and turn them into a token file


The following options are supported

Case Sensitive Tokens (--case or -c)

Tokens are not considered case sensitive by default. If you desire that the tokens Hello and hello be considered different, turn case sensitive tokens on.

The following example will read the file token_list.dat if it exists, and then read all the files matching the file spec. The files will be tokenized with case sensitive and the new file written out.

example: perl -w -c -o token_list.dat -s ./good_files/*.msg

Case Sensitive File Search (--file_case or -f)

Files are specified based on ``file specifications''. By default, the file specs are assumed to be case insensitive. In the UNIX world, this may make a difference so you can turn case sensitivity on with this option.

The following example will read the file token_list.dat if it exists, and then read all the files matching the file spec. This will not match files matching *.MSG. The files will be tokenized and the new file written out. The path portion will be case sensitive based on the operating system used. If you are using Linux and you have the directories good_files and Good_Files, the second directory will not be searched even with the default case insensitive file matching.

example: perl -w -f -o token_list.dat -s ./good_files/*.msg

Help (-h or -?)

Print useage instructions

example: perl -w -h

Log File Name (-l or --log)

If a logfile is specified, then this is used as the logfile name. By default, the log tokenize_file.log is created.

Log Configuration Files (--log_cfg)

You can create a configuration file for your logger and then configure your log object by simply telling it to read the specified configuration file. To create an initial configuration file, write a perl script that creates a logger, configures the logger, and then use the write_to_file('log_cfg.dat') method.

This provides complete control over how the logger is configured. You can set screen and file output levels, for example.

example: perl -w -o token_list.dat -s ./*.msg --log_cfg ~andy/logs/default_log.dat

Log File Directory (--log_dir)

This allows you to specify which directory contains the log

example: perl -w -o token_list.dat -s ./*.msg --log_dir ~andy/logs

Token file (-o or --out)

This provides a method of specifying the name of the token data file. If the file exists, then it is read. If it does not, then it is created.

example: perl -w -o token_list.dat -s ./good_files/*.msg

Recurse Directories (-r or --recurse)

This causes all directories under the specified directory to be searched for the given file spec. In the example below, all files will extension msg will be included regardless of their directory.

example: perl -w -r -o token_list.dat -s ./good_files/*.msg

File Specs (-s or --spec)

This specifies the file specs to search. If you desire to have three sets of file specs, then include the spec parameter three times.

example: perl -w -r -o token_list.dat -s ./good_files/*.msg -s ~andy/home/msg/*.msg


Copyright 1998-2002, Andrew Pitonyak (

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

Modification History

September 10, 2002

Version 1.00 First release