|
Parsing Engine | |||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectdanbikel.lisp.WordTokenizer
public class WordTokenizer
A simple tokenizer for words only (no numbers and no significant eol chars).
Many of the methods and some of the data members of StreamTokenizer
exist in this class, but at present, this class
does not extend StreamTokenizer
. This class recognizes
comments as beginning with an optionally-specified comment character.
A comment is a line where the first non-whitespace character is the
comment character.
Field Summary | |
---|---|
StreamTokenizer |
javadocHack
Included as a public data member so that javadoc can resolve external links to members of the StreamTokenizer class. |
String |
sval
Contains the most recent word tokenized by this tokenizer. |
int |
ttype
The type of the last token read, using the type definitions in StreamTokenizer . |
Constructor Summary | |
---|---|
WordTokenizer(Reader inStream)
Creates a new tokenizer object. |
Method Summary | |
---|---|
void |
close()
Closes the underlying stream. |
void |
commentChar(int ch)
Specifies a character to be treated as the start of a comment on the current line. |
int |
lineno()
Returns the line number of the underlying character stream. |
int |
nextToken()
Reads the next token from the underlying character stream and returns its type, which is also stored in ttype . |
long |
numCharsRead()
Returns the number of characters read from the underlying reader for this word tokenizer. |
void |
ordinaryChar(char ch)
Specifies a character to treated as a token delimiter, to be contained in ttype after it is read. |
void |
ordinaryChars(int low,
int hi)
Specifies a range of characters to treated as a token delimiter, to be contained in ttype after it is read. |
void |
pushBack()
Causes the most recent token read (either a word or ordinary character) to be pushed back, so that it is the next token returned by nextToken() . |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public StreamTokenizer javadocHack
StreamTokenizer
class.
public int ttype
StreamTokenizer
. Because this tokenizer only reads ordinary
characters and words, the value of ttype
will only ever be
StreamTokenizer.TT_EOF
, StreamTokenizer.TT_WORD
or an ordinary character.
ordinaryChar(char)
,
ordinaryChars(int, int)
public String sval
Constructor Detail |
---|
public WordTokenizer(Reader inStream)
inStream
- the stream to be tokenized.Method Detail |
---|
public void commentChar(int ch)
ch
- the character to be treated as the start of a single-line commentpublic void ordinaryChar(char ch)
ttype
after it is read. The character
must be in the range of 0 <= ch <= Byte.MAX_VALUE
.
ch
- the character to be testedpublic void ordinaryChars(int low, int hi)
ttype
after it is read. The characters
must be in the range of 0 <= ch <= Byte.MAX_VALUE
.
low
- the lowest-valued character in a range to be treated as
token delimitershi
- the highest-valued character in a range to be treated as
token delimiterspublic int nextToken() throws IOException
ttype
.
ttype
)
IOException
- if there was a problem reading the next token
from the underlying streampublic void pushBack()
nextToken()
.
public int lineno()
public void close() throws IOException
IOException
- if the underlying stream throws an
IOException
while being closedpublic long numCharsRead()
|
Parsing Engine | |||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |