meta::analyzers::tokenizers::icu_tokenizer::impl Class Reference

Implementation class for the icu_tokenizer. More...

Public Member Functions
void	set_content (std::string content)
std::string	next ()
	operator bool () const
	True if tokens is not empty.

Private Attributes
utf::segmenter	segmenter_
	UTF segmenter to use for this tokenizer.
std::deque< std::string >	tokens_
	Buffered tokens.

Detailed Description

Implementation class for the icu_tokenizer.

Member Function Documentation

void meta::analyzers::tokenizers::icu_tokenizer::impl::set_content ( std::string  content )

inline

Parameters

content

The string content to set TODO: can we make this be a streaming API instead of buffering all of the tokens?

std::string meta::analyzers::tokenizers::icu_tokenizer::impl::next ( )

inline

Returns

the next token

---

The documentation for this class was generated from the following file:

• /home/chase/projects/meta/src/analyzers/tokenizers/icu_tokenizer.cpp