whitespace_tokenizer.cpp File Reference

#include <cassert>
#include <cctype>
#include "analyzers/tokenizers/whitespace_tokenizer.h"
#include "corpus/document.h"
#include "io/mmap_file.h"

Namespaces
	meta
	The ModErn Text Analysis toolkit is a suite of natural language processing, classification, information retreival, data mining, and other applications of text processing.
	meta::analyzers
	Contains various ways to segment text and deal with preprocessed files (POS tags, parse trees, etc).
	meta::analyzers::tokenizers
	Contains tokenizers that start off a filter chain.

Detailed Description

Author

Chase Geigle