grammarscope.io.read.Tokenizer< T > Interface Template Reference

Inheritance diagram for grammarscope.io.read.Tokenizer< T >:

Collaboration diagram for grammarscope.io.read.Tokenizer< T >:

Public Member Functions
T	next ()
boolean	hasNext ()
void	remove ()
T	peek ()
List< T >	tokenize ()

Detailed Description

Tokenizers break up text into individual Objects. These objects may be Strings, Words, or other Objects. A Tokenizer extends the Iterator interface, but provides a lookahead operation peek(). An implementation of this interface is expected to have a constructor that takes a single argument, a Reader.

Author

Teg Grenager (grena.nosp@m.ger@.nosp@m.stanf.nosp@m.ord..nosp@m.edu)

Member Function Documentation

hasNext()

boolean grammarscope.io.read.Tokenizer< T >.hasNext

(

)

Returns true if and only if this Tokenizer has more elements.

Implemented in grammarscope.io.read.AbstractTokenizer< T >.

next()

T grammarscope.io.read.Tokenizer< T >.next

(

)

Returns the next token from this Tokenizer.

Returns

the next token in the token stream.

Exceptions

java.util.NoSuchElementException

if the token stream has no more tokens.

Implemented in grammarscope.io.read.AbstractTokenizer< T >.

peek()

T grammarscope.io.read.Tokenizer< T >.peek

(

)

Returns the next token, without removing it, from the Tokenizer, so that the same token will be again returned on the next call to next() or peek().

Returns

the next token in the token stream.

Exceptions

java.util.NoSuchElementException

if the token stream has no more tokens.

Implemented in grammarscope.io.read.AbstractTokenizer< T >.

remove()

void grammarscope.io.read.Tokenizer< T >.remove

(

)

Removes from the underlying collection the last element returned by the iterator. This is an optional operation for Iterators - a Tokenizer normally would not support it. This method can be called only once per call to next.

Implemented in grammarscope.io.read.AbstractTokenizer< T >.

tokenize()

List<T> grammarscope.io.read.Tokenizer< T >.tokenize

(

)

Returns all tokens of this Tokenizer as a List for convenience.

Returns

A list of all the tokens

Implemented in grammarscope.io.read.AbstractTokenizer< T >.

---

The documentation for this interface was generated from the following file:

• grammarscope/library/src/main/java/grammarscope/io/read/Tokenizer.java