ModErn Text Analysis
META Enumerates Textual Applications
|
Filter that removes "non-letter" characters from tokens. More...
#include <alpha_filter.h>
Public Member Functions | |
alpha_filter (std::unique_ptr< token_stream > source) | |
Constructs an alpha filter reading tokens from the given source. More... | |
alpha_filter (const alpha_filter &other) | |
Copy constructor. More... | |
void | set_content (const std::string &content) override |
Sets the content for the beginning of the filter chain. More... | |
std::string | next () override |
Obtains the next token in the sequence. | |
operator bool () const override | |
Determines whether there are more tokens available in the stream. | |
Public Member Functions inherited from meta::util::multilevel_clonable< Root, Base, Derived > | |
virtual std::unique_ptr< Root > | clone () const |
Clones the given object. More... | |
Static Public Attributes | |
static const std::string | id = "alpha" |
Identifier for this filter. | |
Private Member Functions | |
void | next_token () |
Finds the next valid token for this filter. | |
Private Attributes | |
std::unique_ptr< token_stream > | source_ |
The source to read tokens from. | |
util::optional< std::string > | token_ |
The buffered token. | |
Filter that removes "non-letter" characters from tokens.
"Letterness" is determined by the Unicode properties of each codepoint in the token.
meta::analyzers::filters::alpha_filter::alpha_filter | ( | std::unique_ptr< token_stream > | source | ) |
Constructs an alpha filter reading tokens from the given source.
source | The source to construct the filter from |
meta::analyzers::filters::alpha_filter::alpha_filter | ( | const alpha_filter & | other | ) |
Copy constructor.
other | The alpha_filter to copy into this one |
|
override |
Sets the content for the beginning of the filter chain.
content | The string content to set |