grammarscope.io.read

Class PennTreebankTokenizer

java.lang.Object

grammarscope.io.read.AbstractTokenizer<java.lang.String>

grammarscope.io.read.TokenizerAdapter

grammarscope.io.read.PennTreebankTokenizer

All Implemented Interfaces:

Tokenizer<java.lang.String>, java.util.Iterator<java.lang.String>

public class PennTreebankTokenizer
extends TokenizerAdapter

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	PennTreebankTokenizer.EnglishTreebankStreamTokenizer A StreamTokenizer for PennTreebank trees.

Field Summary

Fields inherited from class grammarscope.io.read.TokenizerAdapter

eolString, st

Fields inherited from class grammarscope.io.read.AbstractTokenizer

nextToken

Constructor Summary

Constructors

Constructor and Description
PennTreebankTokenizer(java.io.Reader r)

Method Summary

Methods inherited from class grammarscope.io.read.TokenizerAdapter

getNext, isEol, setEolString

Methods inherited from class grammarscope.io.read.AbstractTokenizer

hasNext, next, peek, remove, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Constructor Detail

PennTreebankTokenizer

public PennTreebankTokenizer(java.io.Reader r)