grammarscope.io.read.TokenizerAdapter Class Reference

Inheritance diagram for grammarscope.io.read.TokenizerAdapter:

Collaboration diagram for grammarscope.io.read.TokenizerAdapter:

Public Member Functions
	TokenizerAdapter (final StreamTokenizer st)
String	getNext ()
void	setEolString (final String eolString)
boolean	isEol (final String str)
Public Member Functions inherited from grammarscope.io.read.AbstractTokenizer< String >
T	next ()
boolean	hasNext ()
void	remove ()
T	peek ()
List< T >	tokenize ()

Protected Attributes
final StreamTokenizer	st
String	eolString = "<EOL>"
Protected Attributes inherited from grammarscope.io.read.AbstractTokenizer< String >
T	nextToken

Additional Inherited Members
Protected Member Functions inherited from grammarscope.io.read.AbstractTokenizer< String >
abstract T	getNext ()

Detailed Description

This class adapts between a java.io.StreamTokenizer and a edu.stanford.nlp.process.Tokenizer.

Author

Christopher Manning

Version

2004/04/01

Constructor & Destructor Documentation

TokenizerAdapter()

grammarscope.io.read.TokenizerAdapter.TokenizerAdapter

(

final StreamTokenizer

st

)

Create a new TokenizerAdaptor. In general, it is recommended that the passed in StreamTokenizer should have had resetSyntax() done to it, so that numbers are returned as entered as tokens of type String, though this code will cope as best it can.

Parameters

st	The internal java.io.StreamTokenizer

Member Function Documentation

getNext()

String grammarscope.io.read.TokenizerAdapter.getNext

(

)

Internally fetches the next token.

Returns

the next token in the token stream, or null if none exists.

isEol()

boolean grammarscope.io.read.TokenizerAdapter.isEol

(

final String

str

)

Say whether the String is the end-of-line token for this tokenizer.

Parameters

str	The String being tested

Returns

Whether it is the end-of-line token

setEolString()

void grammarscope.io.read.TokenizerAdapter.setEolString

(

final String

eolString

)

Set the String returned when the inner tokenizer returns an end-of-line token. This will only happen if the inner tokenizer has been set to eolIsSignificant(true).

Parameters

eolString

The String used to represent eol. It is not allowed to be null (which would confuse line ends and file end)

Member Data Documentation

eolString

String grammarscope.io.read.TokenizerAdapter.eolString = "<EOL>"

protected

st

final StreamTokenizer grammarscope.io.read.TokenizerAdapter.st

protected

---

The documentation for this class was generated from the following file:

• grammarscope/library/src/main/java/grammarscope/io/read/TokenizerAdapter.java