Introduction - If you have any usage issues, please Google them yourself
Using the UnderC Tokenizer Class
It s often necessary to parse complex text files, where standard i/o
is too clumsy. C programmers fall back on strtok(), but this can be
tricky to use properly. Besides, you are still responsible for keeping
strtok() fed with new input, and I don t like the schlepp.
Tokenizer is a text-parsing input stream, modelled after the (undocumented)
VCL TParser class, and based on the UnderC tokenizing preprocessor front-end.