Parsing Engine

danbikel.lisp
Class SexpTokenizer

java.lang.Object
  extended by danbikel.lisp.WordTokenizer
      extended by danbikel.lisp.SexpTokenizer

public class SexpTokenizer
extends WordTokenizer

A class for tokenizing simple S-expressions, where there are only strings delimited by whitespace or parentheses (as implemented by WordTokenizer). Comments are lines where the first non-whitespace character is a semicolon (the character ';').


Field Summary
 
Fields inherited from class danbikel.lisp.WordTokenizer
javadocHack, sval, ttype
 
Constructor Summary
SexpTokenizer(File file, String encoding, int bufSize)
          Convenience constructor, creating a SexpTokenizer around a BufferedReader around a FileInputStream.
SexpTokenizer(File file, String encoding, int bufSize, boolean comments)
          Convenience constructor, creating a SexpTokenizer around a BufferedReader around a FileInputStream.
SexpTokenizer(InputStream stream, String encoding, int bufSize)
          Convenience constructor, creating a SexpTokenizer around a BufferedReader around a InputStreamReader.
SexpTokenizer(InputStream stream, String encoding, int bufSize, boolean comments)
          Convenience constructor, creating a SexpTokenizer around a BufferedReader around a InputStreamReader.
SexpTokenizer(Reader inStream)
          Constructs a SexpTokenizer with the specified stream and the default comment-recognition option, which is true.
SexpTokenizer(Reader inStream, boolean comments)
          Constructs a SexpTokenizer with the specified stream and comment-recognition option.
SexpTokenizer(Reader inStream, boolean comments, char[] ordinary)
          Constructs a SexpTokenizer with the specified stream, comment-recognition option and set of ordinary characters, which are actually those characters to be treated as metacharacters (i.e., not part of tokens, but delimiters of tokens).
SexpTokenizer(String filename, String encoding, int bufSize)
          Convenience constructor, creating a SexpTokenizer around a BufferedReader around a FileInputStream.
SexpTokenizer(String filename, String encoding, int bufSize, boolean comments)
          Convenience constructor, creating a SexpTokenizer around a BufferedReader around a FileInputStream.
SexpTokenizer(URL url, String encoding, int bufSize)
          Convenience constructor, creating a SexpTokenizer around a BufferedReader around an InputStreamReader around the stream created by calling url.openStream().
SexpTokenizer(URL url, String encoding, int bufSize, boolean comments)
          Convenience constructor, creating a SexpTokenizer around a BufferedReader around a InputStreamReader around the stream created by calling url.openStream().
 
Method Summary
 
Methods inherited from class danbikel.lisp.WordTokenizer
close, commentChar, lineno, nextToken, numCharsRead, ordinaryChar, ordinaryChars, pushBack
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

SexpTokenizer

public SexpTokenizer(Reader inStream,
                     boolean comments)
Constructs a SexpTokenizer with the specified stream and comment-recognition option.

Parameters:
inStream - the input stream for this tokenizer
comments - indicates whether to recognizes comment lines
See Also:
WordTokenizer.commentChar(int)

SexpTokenizer

public SexpTokenizer(Reader inStream,
                     boolean comments,
                     char[] ordinary)
Constructs a SexpTokenizer with the specified stream, comment-recognition option and set of ordinary characters, which are actually those characters to be treated as metacharacters (i.e., not part of tokens, but delimiters of tokens).

Parameters:
inStream - the input stream for this tokenizer
comments - indicates whether to recognizes comment lines
ordinary - an array of all characters to be treated as metacharacters
See Also:
WordTokenizer.ordinaryChar(char), WordTokenizer.commentChar(int)

SexpTokenizer

public SexpTokenizer(Reader inStream)
Constructs a SexpTokenizer with the specified stream and the default comment-recognition option, which is true.

Parameters:
inStream - the input stream for this tokenizer

SexpTokenizer

public SexpTokenizer(String filename,
                     String encoding,
                     int bufSize)
              throws UnsupportedEncodingException,
                     FileNotFoundException,
                     IOException
Convenience constructor, creating a SexpTokenizer around a BufferedReader around a FileInputStream. The tokenizer will recognize comment lines.

Parameters:
filename - the name of the file containing S-expressions
encoding - the encoding of the file to be tokenized
bufSize - the size of the buffer of the BufferedReader that will be created
Throws:
UnsupportedEncodingException - if the specified encoding is not supported
FileNotFoundException - if the specified file does not exist
IOException

SexpTokenizer

public SexpTokenizer(String filename,
                     String encoding,
                     int bufSize,
                     boolean comments)
              throws UnsupportedEncodingException,
                     FileNotFoundException,
                     IOException
Convenience constructor, creating a SexpTokenizer around a BufferedReader around a FileInputStream.

Parameters:
filename - the name of the file containing S-expressions
encoding - the encoding of the file to be tokenized
bufSize - the size of the buffer of the BufferedReader that will be created
comments - whether this tokenizer will recognize comments
Throws:
UnsupportedEncodingException - if the specified encoding is not supported
FileNotFoundException - if the specified file does not exist
IOException

SexpTokenizer

public SexpTokenizer(File file,
                     String encoding,
                     int bufSize)
              throws UnsupportedEncodingException,
                     FileNotFoundException,
                     IOException
Convenience constructor, creating a SexpTokenizer around a BufferedReader around a FileInputStream. The tokenizer will recognize comment lines.

Parameters:
file - the file containing S-expressions
encoding - the encoding of the file to be tokenized
bufSize - the size of the buffer of the BufferedReader that will be created
Throws:
UnsupportedEncodingException - if the specified encoding is not supported
FileNotFoundException - if the specified file does not exist
IOException

SexpTokenizer

public SexpTokenizer(File file,
                     String encoding,
                     int bufSize,
                     boolean comments)
              throws UnsupportedEncodingException,
                     FileNotFoundException,
                     IOException
Convenience constructor, creating a SexpTokenizer around a BufferedReader around a FileInputStream.

Parameters:
file - the file containing S-expressions
encoding - the encoding of the file to be tokenized
bufSize - the size of the buffer of the BufferedReader that will be created
comments - whether this tokenizer will recognize comments
Throws:
UnsupportedEncodingException - if the specified encoding is not supported
FileNotFoundException - if the specified file does not exist
IOException

SexpTokenizer

public SexpTokenizer(InputStream stream,
                     String encoding,
                     int bufSize)
              throws UnsupportedEncodingException
Convenience constructor, creating a SexpTokenizer around a BufferedReader around a InputStreamReader. The tokenizer will recognize comment lines.

Parameters:
stream - the stream of bytes, encoded with encoding, containing S-expressions
encoding - the encoding of the file to be tokenized
bufSize - the size of the buffer of the BufferedReader that will be created
Throws:
UnsupportedEncodingException - if the specified encoding is not supported

SexpTokenizer

public SexpTokenizer(InputStream stream,
                     String encoding,
                     int bufSize,
                     boolean comments)
              throws UnsupportedEncodingException
Convenience constructor, creating a SexpTokenizer around a BufferedReader around a InputStreamReader.

Parameters:
stream - the stream of bytes, encoded with encoding, containing S-expressions
encoding - the encoding of the file to be tokenized
bufSize - the size of the buffer of the BufferedReader that will be created
comments - whether this tokenizer will recognize comments
Throws:
UnsupportedEncodingException - if the specified encoding is not supported

SexpTokenizer

public SexpTokenizer(URL url,
                     String encoding,
                     int bufSize)
              throws UnsupportedEncodingException,
                     IOException
Convenience constructor, creating a SexpTokenizer around a BufferedReader around an InputStreamReader around the stream created by calling url.openStream(). The tokenizer will recognize comment lines.

Parameters:
url - the url from which to get the stream containing S-expressions
encoding - the encoding of the file to be tokenized
bufSize - the size of the buffer of the BufferedReader
Throws:
UnsupportedEncodingException - if the specified encoding is not supported
IOException - if the underlying call to URL.openStream() or the construction of any of the readers throws an IOException

SexpTokenizer

public SexpTokenizer(URL url,
                     String encoding,
                     int bufSize,
                     boolean comments)
              throws UnsupportedEncodingException,
                     IOException
Convenience constructor, creating a SexpTokenizer around a BufferedReader around a InputStreamReader around the stream created by calling url.openStream().

Parameters:
url - the url from which to get the stream containing S-expressions
encoding - the encoding of the file to be tokenized
bufSize - the size of the buffer of the BufferedReader
comments - whether this tokenizer will recognize comments
Throws:
UnsupportedEncodingException - if the specified encoding is not supported
IOException - if the underlying call to URL.openStream() or the construction of any of the readers throws an IOException

Parsing Engine

Author: Dan Bikel.