ModErn Text Analysis
META Enumerates Textual Applications
|
Base class that represents a stream of tokens that have been extracted from a document. More...
#include <token_stream.h>
Classes | |
class | token_stream_exception |
Basic exception class for token stream interactions. More... | |
Public Member Functions | |
virtual std::string | next ()=0 |
Obtains the next token in the sequence. | |
virtual | operator bool () const =0 |
Determines whether there are more tokens available in the stream. | |
virtual void | set_content (const std::string &content)=0 |
Sets the content for the stream. More... | |
virtual | ~token_stream ()=default |
Destructor. | |
virtual std::unique_ptr< token_stream > | clone () const =0 |
Clones the given token stream. More... | |
Base class that represents a stream of tokens that have been extracted from a document.
These tokens may be raw tokens (in the case of a tokenizer class) or filtered tokens (from the filter classes).
|
pure virtual |
Sets the content for the stream.
content | The string content to set |
|
pure virtual |
Clones the given token stream.