tokenize_file.pl - Read a list of email file specs and turn them into a token file
The following options are supported
Tokens are not considered case sensitive by default. If you desire that the tokens Hello and hello be considered different, turn case sensitive tokens on.
The following example will read the file token_list.dat if it exists, and then read all the files matching the file spec. The files will be tokenized with case sensitive and the new file written out.
example: perl -w tokenize_file.pl -c -o token_list.dat -s ./good_files/*.msg
Files are specified based on ``file specifications''. By default, the file specs are assumed to be case insensitive. In the UNIX world, this may make a difference so you can turn case sensitivity on with this option.
The following example will read the file token_list.dat if it exists, and then read all the files matching the file spec. This will not match files matching *.MSG. The files will be tokenized and the new file written out. The path portion will be case sensitive based on the operating system used. If you are using Linux and you have the directories good_files and Good_Files, the second directory will not be searched even with the default case insensitive file matching.
example: perl -w tokenize_file.pl -f -o token_list.dat -s ./good_files/*.msg
Print useage instructions
example: perl -w tokenize_file.pl -h
If a logfile is specified, then this is used as the logfile name. By default, the log tokenize_file.log is created.
You can create a configuration file for your logger and then configure
your log object by simply telling it to read the specified configuration file.
To create an initial configuration file, write a perl script that
creates a logger, configures the logger, and then use the write_to_file('log_cfg.dat')
method.
This provides complete control over how the logger is configured. You can set screen and file output levels, for example.
example: perl -w tokenize_file.pl -o token_list.dat -s ./*.msg --log_cfg ~andy/logs/default_log.dat
This allows you to specify which directory contains the log
example: perl -w tokenize_file.pl -o token_list.dat -s ./*.msg --log_dir ~andy/logs
This provides a method of specifying the name of the token data file. If the file exists, then it is read. If it does not, then it is created.
example: perl -w tokenize_file.pl -o token_list.dat -s ./good_files/*.msg
This causes all directories under the specified directory to be searched for the given file spec. In the example below, all files will extension msg will be included regardless of their directory.
example: perl -w tokenize_file.pl -r -o token_list.dat -s ./good_files/*.msg
This specifies the file specs to search. If you desire to have three sets of file specs, then include the spec parameter three times.
example: perl -w tokenize_file.pl -r -o token_list.dat -s ./good_files/*.msg -s ~andy/home/msg/*.msg
Copyright 1998-2002, Andrew Pitonyak (perlboy@pitonyak.org)
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
Version 1.00 First release