GrammarScope  4.0.0
Public Member Functions | Protected Attributes | List of all members
grammarscope.io.read.TokenizerAdapter Class Reference
Inheritance diagram for grammarscope.io.read.TokenizerAdapter:
Inheritance graph
[legend]
Collaboration diagram for grammarscope.io.read.TokenizerAdapter:
Collaboration graph
[legend]

Public Member Functions

 TokenizerAdapter (final StreamTokenizer st)
 
String getNext ()
 
void setEolString (final String eolString)
 
boolean isEol (final String str)
 
- Public Member Functions inherited from grammarscope.io.read.AbstractTokenizer< String >
next ()
 
boolean hasNext ()
 
void remove ()
 
peek ()
 
List< T > tokenize ()
 

Protected Attributes

final StreamTokenizer st
 
String eolString = "<EOL>"
 
- Protected Attributes inherited from grammarscope.io.read.AbstractTokenizer< String >
nextToken
 

Additional Inherited Members

- Protected Member Functions inherited from grammarscope.io.read.AbstractTokenizer< String >
abstract T getNext ()
 

Detailed Description

This class adapts between a java.io.StreamTokenizer and a edu.stanford.nlp.process.Tokenizer.

Author
Christopher Manning
Version
2004/04/01

Constructor & Destructor Documentation

◆ TokenizerAdapter()

grammarscope.io.read.TokenizerAdapter.TokenizerAdapter ( final StreamTokenizer  st)

Create a new TokenizerAdaptor. In general, it is recommended that the passed in StreamTokenizer should have had resetSyntax() done to it, so that numbers are returned as entered as tokens of type String, though this code will cope as best it can.

Parameters
stThe internal java.io.StreamTokenizer

Member Function Documentation

◆ getNext()

String grammarscope.io.read.TokenizerAdapter.getNext ( )

Internally fetches the next token.

Returns
the next token in the token stream, or null if none exists.

◆ isEol()

boolean grammarscope.io.read.TokenizerAdapter.isEol ( final String  str)

Say whether the String is the end-of-line token for this tokenizer.

Parameters
strThe String being tested
Returns
Whether it is the end-of-line token

◆ setEolString()

void grammarscope.io.read.TokenizerAdapter.setEolString ( final String  eolString)

Set the String returned when the inner tokenizer returns an end-of-line token. This will only happen if the inner tokenizer has been set to eolIsSignificant(true).

Parameters
eolStringThe String used to represent eol. It is not allowed to be null (which would confuse line ends and file end)

Member Data Documentation

◆ eolString

String grammarscope.io.read.TokenizerAdapter.eolString = "<EOL>"
protected

◆ st

final StreamTokenizer grammarscope.io.read.TokenizerAdapter.st
protected

The documentation for this class was generated from the following file: