Base class that represents a stream of tokens that have been extracted from a document. More...

Classes
class	token_stream_exception
	Basic exception class for token stream interactions. More...

Public Member Functions
virtual std::string	next ()=0
	Obtains the next token in the sequence.

virtual	operator bool () const =0
	Determines whether there are more tokens available in the stream.

virtual void	set_content (const std::string &content)=0
	Sets the content for the stream. More...

virtual	~token_stream ()=default
	Destructor.

virtual std::unique_ptr< token_stream >	clone () const =0
	Clones the given token stream. More...

Detailed Description

Base class that represents a stream of tokens that have been extracted from a document.

These tokens may be raw tokens (in the case of a tokenizer class) or filtered tokens (from the filter classes).

Member Function Documentation

virtual void meta::analyzers::token_stream::set_content ( const std::string & content )

pure virtual

Sets the content for the stream.

Parameters

content The string content to set

virtual std::unique_ptr<token_stream> meta::analyzers::token_stream::clone ( ) const

pure virtual

Clones the given token stream.

The documentation for this class was generated from the following file: