grammarscope.io.read.AbstractTokenizer< T > Class Template Reference

Inheritance diagram for grammarscope.io.read.AbstractTokenizer< T >:

Collaboration diagram for grammarscope.io.read.AbstractTokenizer< T >:

Public Member Functions
T	next ()
boolean	hasNext ()
void	remove ()
T	peek ()
List< T >	tokenize ()

Protected Member Functions
abstract T	getNext ()

Protected Attributes
T	nextToken

Detailed Description

An abstract tokenizer. Tokenizers extending AbstractTokenizer need only implement the getNext() method. This implementation does not allow null tokens, since null is used in the protected nextToken field to signify that no more tokens are available.

Author

Teg Grenager (grena.nosp@m.ger@.nosp@m.stanf.nosp@m.ord..nosp@m.edu)

Member Function Documentation

getNext()

abstract T grammarscope.io.read.AbstractTokenizer< T >.getNext ( )

abstractprotected

Internally fetches the next token.

Returns

the next token in the token stream, or null if none exists.

hasNext()

boolean grammarscope.io.read.AbstractTokenizer< T >.hasNext

(

)

Returns true if this Tokenizer has more elements.

Implements grammarscope.io.read.Tokenizer< T >.

next()

T grammarscope.io.read.AbstractTokenizer< T >.next

(

)

Returns the next token from this Tokenizer.

Returns

the next token in the token stream.

Exceptions

java.util.NoSuchElementException

if the token stream has no more tokens.

Implements grammarscope.io.read.Tokenizer< T >.

peek()

T grammarscope.io.read.AbstractTokenizer< T >.peek

(

)

This is an optional operation, by default supported.

Returns

The next token in the token stream.

Exceptions

java.util.NoSuchElementException

if the token stream has no more tokens.

Implements grammarscope.io.read.Tokenizer< T >.

remove()

void grammarscope.io.read.AbstractTokenizer< T >.remove

(

)

This is an optional operation, by default not supported.

Implements grammarscope.io.read.Tokenizer< T >.

tokenize()

List<T> grammarscope.io.read.AbstractTokenizer< T >.tokenize

(

)

Returns text as a List of tokens.

Returns

A list of all tokens remaining in the underlying Reader

Implements grammarscope.io.read.Tokenizer< T >.

Member Data Documentation

nextToken

T grammarscope.io.read.AbstractTokenizer< T >.nextToken

protected

---

The documentation for this class was generated from the following file:

• grammarscope/library/src/main/java/grammarscope/io/read/AbstractTokenizer.java