icu_tokenizer.cpp File Reference

#include <algorithm>
#include <deque>
#include <unicode/utf.h>
#include <unicode/uchar.h>
#include "analyzers/tokenizers/icu_tokenizer.h"
#include "util/pimpl.tcc"
#include "utf/segmenter.h"

Classes
class	meta::analyzers::tokenizers::icu_tokenizer::impl
	Implementation class for the icu_tokenizer. More...

Namespaces
	meta
	The ModErn Text Analysis toolkit is a suite of natural language processing, classification, information retreival, data mining, and other applications of text processing.
	meta::analyzers
	Contains various ways to segment text and deal with preprocessed files (POS tags, parse trees, etc).
	meta::analyzers::tokenizers
	Contains tokenizers that start off a filter chain.

Detailed Description

Author

Chase Geigle