| Author: | Paul McGuire |
|---|---|
| Address: | ptmcg@users.sourceforge.net |
| Revision: | 1.4 |
| Date: | December, 2005 |
| Copyright: | Copyright © 2003-2005 Paul McGuire. |
| abstract: | This document provides how-to instructions for the pyparsing library, an easy-to-use Python module for constructing and executing basic text parsers. The pyparsing module is useful for evaluating user-definable expressions, processing custom application language commands, or extracting data from formatted reports. |
|---|
To parse an incoming data string, the client code must follow these steps:
The following complete Python program will parse the greeting "Hello, World!", or any other greeting of the form "<salutation>, <addressee>!":
from pyparsing import Word, alphas greet = Word( alphas ) + "," + Word( alphas ) + "!" greeting = greet.parseString( "Hello, World!" ) print greeting
The parsed tokens are returned in the following form:
['Hello', ',', 'World', '!']
The pyparsing module can be used to interpret simple command strings or algebraic expressions, or can be used to extract data from text reports with complicated format and structure ("screen or report scraping"). However, it is possible that your defined matching patterns may accept invalid inputs. Use pyparsing to extract data from strings assumed to be well-formatted.
To keep up the readability of your code, use the +, |, ^, and ~ operators to combine expressions. You can also combine string literals with ParseExpressions - they will be automatically converted to Literal objects. For example:
integer = Word( nums ) # simple unsigned integer variable = Word( alphas, max=1 ) # single letter variable, such as x, z, m, etc. arithOp = Word( "+-*/", max=1 ) # arithmetic operators equation = variable + "=" + integer + arithOp + integer # will match "x=2+2", etc.
In the definition of equation, the string "=" will get added as a Literal("="), but in a more readable way.
The pyparsing module's default behavior is to ignore whitespace. This is the case for 99% of all parsers ever written. This allows you to write simple, clean, grammars, such as the above equation, without having to clutter it up with extraneous ws markers. The equation grammar will successfully parse all of the following statements:
x=2+2 x = 2+2 a = 10 * 4 r= 1234/ 100000
Of course, it is quite simple to extend this example to support more elaborate expressions, with nesting with parentheses, floating point numbers, scientific notation, and named constants (such as e or pi). See fourFn.py, included in the examples directory.
MatchFirst expressions are matched left-to-right, and the first match found will skip all later expressions within, so be sure to define less-specific patterns after more-specific patterns. If you are not sure which expressions are most specific, use Or expressions (defined using the ^ operator) - they will always match the longest expression, although they are more compute-intensive.
Or expressions will evaluate all of the specified subexpressions to determine which is the "best" match, that is, which matches the longest string in the input data. In case of a tie, the left-most expression in the Or list will win.
If parsing the contents of an entire file, pass it to the parseFile method using:
expr.parseFile( sourceFile )
ParseExceptions will report the location where an expected token or expression failed to match. For example, if we tried to use our "Hello, World!" parser to parse "Hello World!" (leaving out the separating comma), we would get an exception, with the message:
pyparsing.ParseException: Expected "," (6), (1,7)
In the case of complex expressions, the reported location may not be exactly where you would expect. See more information under ParseException .
Use the Group class to enclose logical groups of tokens within a sublist. This will help organize your results into more hierarchical form (the default behavior is to return matching tokens as a flat list of matching input strings).
Punctuation may be significant for matching, but is rarely of much interest in the parsed results. Use the suppress() method to keep these tokens from cluttering up your returned lists of tokens. For example, delimitedList() matches a succession of one or more expressions, separated by delimiters (commas by default), but only returns a list of the actual expressions - the delimiters are used for parsing, but are suppressed from the returned output.
Parse actions can be used to convert values from strings to other data types (ints, floats, booleans, etc.).
Be careful when defining parse actions that modify global variables or data structures (as in fourFn.py), especially for low level tokens or expressions that may occur within an And expression; an early element of an And may match, but the overall expression may fail.
Performance of pyparsing may be slow for complex grammars and/or large input strings. The psyco package can be used to improve the speed of the pyparsing module with no changes to grammar or program logic - observed improvments have been in the 20-50% range.
ParserElement - abstract base class for all pyparsing classes; methods for code to use are:
parseString( sourceString ) - only called once, on the overall matching pattern; returns a ParseResults object that makes the matched tokens available as a list, and optionally as a dictionary, or as an object with named attributes
parseFile( sourceFile ) - a convenience function, that accepts an input file object or filename. The file contents are passed as a string to parseString().
scanString( sourceString ) - generator function, used to find and extract matching text in the given source string; for each matched text, returns a tuple of:
scanString allows you to scan through the input source string for random matches, instead of exhaustively defining the grammar for the entire source text (as would be required with parseString).
transformString( sourceString ) - convenience wrapper function for scanString, to process the input source string, and replace matching text with the tokens returned from parse actions defined in the grammar (see setParseAction).
setName( name ) - associate a short descriptive name for this element, useful in displaying exceptions and trace information
setResultsName( string, listAllMatches=False ) - name to be given to tokens matching the element; if multiple tokens within a repetition group (such as ZeroOrMore or delimitedList) the default is to return only the last matching token - if listAllMatches is set to True, then a list of matching tokens is returned. Note: setResultsName returns a copy of the element so that a single basic element can be referenced multiple times and given different names within a complex grammar.
setParseAction( fn ) - function to call after successful matching of the element; the function is defined as fn( s, loc, toks ), where:
fn can return a modified toks list, to perform conversion, or string modifications. For brevity, fn may also be a lambda - here is an example of using a parse action to convert matched integer tokens from strings to integers:
intNumber = Word(nums).setParseAction( lambda s,l,t: [ int(t[0]) ] )
If fn does not modify the toks list, it does not need to return anything at all.
copy() - returns a copy of a ParserElement; can be used to use the same parse expression in different places in a grammar, with different parse actions attached to each
leaveWhiteSpace() - change default behavior of skipping whitespace before starting matching (mostly used internally to the pyparsing module, rarely used by client code)
setWhitespaceChars( chars ) - define the set of chars to be ignored as whitespace before trying to match a specific ParserElement, in place of the default set of whitespace (space, tab, newline, and return)
setDefaultWhitespaceChars( chars ) - class-level method to override the default set of whitespace chars for all subsequently created ParserElements (including copies); useful when defining grammars that treat one or more of the default whitespace characters as significant (such as a line-sensitive grammar, to omit newline from the list of ignorable whitespace)
suppress() - convenience function to suppress the output of the given element, instead of wrapping it with a Suppress object.
ignore( expr ) - function to specify parse expression to be ignored while matching defined patterns; can be called repeatedly to specify multiple expressions; useful to specify patterns of comment syntax, for example
setDebug( dbgFlag=True ) - function to enable/disable tracing output when trying to match this element
validate() - function to verify that the defined grammar does not contain infinitely recursive constructs
Word - one or more contiguous characters; construct with a string containing the set of allowed initial characters, and an optional second string of allowed body characters; for instance, a common Word construct is to match a code identifier - in C, a valid identifier must start with an alphabetic character or an underscore ('_'), followed by a body that can also include numeric digits. That is, a, i, MAX_LENGTH, _a1, b_109_, and plan9FromOuterSpace are all valid identifiers; 9b7z, $a, .section, and 0debug are not. To define an identifier using a Word, use either of the following:
- Word( alphas+"_", alphanums+"_" )
- Word( srange("[a-zA-Z_]"), srange("[a-zA-Z0-9_]") )
If only one string given, it specifies that the same character set defined for the initial character is used for the word body; for instance, to define an identifier that can only be composed of capital letters and underscores, use:
- Word( "ABCDEFGHIJKLMNOPQRSTUVWXYZ" )
- Word( srange("[A-Z]")
A Word may also be constructed with any of the following optional parameters:
If exact is specified, it will override any values for min or max.
CharsNotIn - similar to Word, but matches characters not in the given constructor string (accepts only one string for both initial and body characters); also supports min, max, and exact optional parameters.
Regex - a powerful construct, that accepts a regular expression to be matched at the current parse position; accepts an optional flags parameter, corresponding to the flags parameter in the re.compile method; if the expression includes named sub-fields, they will be represented in the returned ParseResults
SkipTo - skips ahead in the input string, accepting any characters up to the specified pattern; may be constructed with the following optional parameters:
ParseResults - class used to contain and manage the lists of tokens created from parsing the input using the user-defined parse expression. ParseResults can be accessed in a number of ways:
ParseResults can also be converted to an ordinary list of strings by calling asList(). Note that this will strip the results of any field names that have been defined for any embedded parse elements. (The pprint module is especially good at printing out the nested contents given by asList().)
Finally, ParseResults can be converted to an XML string by calling asXML(). Where possible, results will be tagged using the results names defined for the respective ParseExpressions. asXML() takes two optional arguments:
ParseException - exception returned when a grammar parse fails; ParseExceptions have attributes loc, msg, line, lineno, and column; to view the text line and location where the reported ParseException occurs, use:
except ParseException, err:
print err.line
print " "*(err.column-1) + "^"
print err
RecursiveGrammarException - exception returned by validate() if the grammar contains a recursive infinite loop, such as:
badGrammar = Forward()
goodToken = Literal("A")
badGrammar << Optional(goodToken) + badGrammar
You can also get some insights into the parsing logic using diagnostic parse actions, and setDebug(), or test the matching of expression fragments by testing them using scanString().
removeQuotes - removes the first and last characters of a quoted string; useful to remove the delimiting quotes from quoted strings
replaceWith(replString) - returns a parse action that simply returns the replString; useful when using transformString, or converting HTML entities, as in:
nbsp = Literal(" ").setParseAction( replaceWith("<BLANK>") )
alphas - same as string.letters
nums - same as string.digits
alphanums - a string containing alphas + nums
alphas8bit - a string containing alphabetic 8-bit characters:
ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþ
printables - same as string.printable, minus the space (' ') character
empty - a global Empty(); will always match
sglQuotedString - a string of characters enclosed in 's; may include whitespace, but not newlines
dblQuotedString - a string of characters enclosed in "s; may include whitespace, but not newlines
quotedString - sglQuotedString | dblQuotedString
cStyleComment - a comment block delimited by '/*' and '*/' sequences; can span multiple lines, but does not support nesting of comments
htmlComment - a comment block delimited by '<!--' and '-->' sequences; can span multiple lines, but does not support nesting of comments
commaSeparatedList - similar to delimitedList, except that the list expressions can be any text value, or a quoted string; quoted strings can safely include commas without incorrectly breaking the string into two tokens
restOfLine - all remaining printable characters up to but not including the next newline