grammarscope.io.read

Interface Tokenizer<T>

All Superinterfaces:

java.util.Iterator<T>

All Known Implementing Classes:

AbstractTokenizer, PennTreebankTokenizer, TokenizerAdapter

public interface Tokenizer<T>
extends java.util.Iterator<T>

Tokenizers break up text into individual Objects. These objects may be Strings, Words, or other Objects. A Tokenizer extends the Iterator interface, but provides a lookahead operation peek(). An implementation of this interface is expected to have a constructor that takes a single argument, a Reader.

Author:

Teg Grenager (grenager@stanford.edu)

Method Summary

Modifier and Type	Method and Description
boolean	hasNext() Returns true if and only if this Tokenizer has more elements.
T	next() Returns the next token from this Tokenizer.
T	peek() Returns the next token, without removing it, from the Tokenizer, so that the same token will be again returned on the next call to next() or peek().
void	remove() Removes from the underlying collection the last element returned by the iterator.
java.util.List<T>	tokenize() Returns all tokens of this Tokenizer as a List for convenience.

Methods inherited from interface java.util.Iterator

forEachRemaining

Method Detail

next

T next()

Returns the next token from this Tokenizer.

Specified by:

next in interface java.util.Iterator<T>

Returns:

the next token in the token stream.

Throws:

java.util.NoSuchElementException - if the token stream has no more tokens.

hasNext

boolean hasNext()

Returns true if and only if this Tokenizer has more elements.

Specified by:

hasNext in interface java.util.Iterator<T>

remove

void remove()

Removes from the underlying collection the last element returned by the iterator. This is an optional operation for Iterators - a Tokenizer normally would not support it. This method can be called only once per call to next.

Specified by:

remove in interface java.util.Iterator<T>

peek

T peek()

Returns the next token, without removing it, from the Tokenizer, so that the same token will be again returned on the next call to next() or peek().

Returns:

the next token in the token stream.

Throws:

java.util.NoSuchElementException - if the token stream has no more tokens.

tokenize

java.util.List<T> tokenize()

Returns all tokens of this Tokenizer as a List for convenience.

Returns:

A list of all the tokens