ModErn Text Analysis
META Enumerates Textual Applications
|
Filter that removes any empty sentences from the token stream. More...
#include <empty_sentence_filter.h>
Public Member Functions | |
empty_sentence_filter (std::unique_ptr< token_stream > source) | |
Constructs an empty_sentence_filter which reads tokens from the given source. More... | |
empty_sentence_filter (const empty_sentence_filter &other) | |
Copy constructor. More... | |
void | set_content (const std::string &content) override |
Sets the content for the beginning of the filter chain. More... | |
std::string | next () override |
Obtains the next token in the sequence. | |
operator bool () const override | |
Determines whether there are more tokens available in the stream. | |
Public Member Functions inherited from meta::util::multilevel_clonable< Root, Base, Derived > | |
virtual std::unique_ptr< Root > | clone () const |
Clones the given object. More... | |
Static Public Attributes | |
static const std::string | id = "empty-sentence" |
Identifier for this filter. | |
Private Member Functions | |
void | next_token () |
Finds the next valid token for this filter. | |
Private Attributes | |
std::unique_ptr< token_stream > | source_ |
The source to read tokens from. | |
util::optional< std::string > | first_ |
Keeps track of the left hand side of a potentially empty sentence. | |
util::optional< std::string > | second_ |
Keeps track of the right hand side of a potentially empty sentence. | |
Filter that removes any empty sentences from the token stream.
Empty sentences can be caused by filters in the filter chain that follow sentence boundary detection.
meta::analyzers::filters::empty_sentence_filter::empty_sentence_filter | ( | std::unique_ptr< token_stream > | source | ) |
Constructs an empty_sentence_filter which reads tokens from the given source.
source | The source to construct the filter from |
meta::analyzers::filters::empty_sentence_filter::empty_sentence_filter | ( | const empty_sentence_filter & | other | ) |
Copy constructor.
other | The empty_sentence_filter to copy into this one |
|
override |
Sets the content for the beginning of the filter chain.
content | The string content to set |