Earley parser


In computer science, the Earley parser is an algorithm for parsing strings that belong to a given context-free language, though it may suffer problems with certain nullable grammars. The algorithm, named after its inventor Jay Earley, is a chart parser that uses dynamic programming; it is mainly used for parsing in computational linguistics. It was first introduced in his dissertation in 1968.
Earley parsers are appealing because they can parse all context-free languages, unlike LR parsers and LL parsers, which are more typically used in compilers but which can only handle restricted classes of languages. The Earley parser executes in cubic time in the general case, where n is the length of the parsed string, quadratic time for unambiguous grammars, and linear time for all deterministic context-free grammars. It performs particularly well when the rules are written left-recursively.

Earley recogniser

The following algorithm describes the Earley recogniser. The recogniser can be modified to create a parse tree as it recognises, and in that way can be turned into a parser.

The algorithm

In the following descriptions, α, β, and γ represent any string of terminals/nonterminals, X and Y represent single nonterminals, and a represents a terminal symbol.
Earley's algorithm is a top-down dynamic programming algorithm. In the following, we use Earley's dot notation: given a production X → αβ, the notation X → α • β represents a condition in which α has already been parsed and β is expected.
Input position 0 is the position prior to input. Input position n is the position after accepting the nth token. For every input position, the parser generates a state set. Each state is a tuple, consisting of
  • the production currently being matched
  • the current position in that production
  • the position i in the input at which the matching of this production began: the origin position
A state is finished when its current position is the last position of the right side of the production, that is, when there is no symbol to the right of the dot • in the visual representation of the state.
The state set at input position k is called S. The parser is seeded with S consisting of only the top-level rule. The parser then repeatedly executes three operations: prediction, scanning, and completion.
  • Prediction: For every state in S of the form , add to S for every production in the grammar with Y on the left-hand side.
  • Scanning: If a is the next symbol in the input stream, for every state in S of the form, add to S.
  • Completion: For every state in S of the form, find all states in S of the form and add to S.
Duplicate states are not added to the state set, only new ones. These three operations are repeated until no new states can be added to the set. The set is generally implemented as a queue of states to process, with the operation to be performed depending on what kind of state it is.
The algorithm accepts if ends up in S, where is the top level-rule and n the input length, otherwise it rejects.

Pseudocode

Adapted from Speech and Language Processing by Daniel Jurafsky and James H. Martin,

DECLARE ARRAY S;
function INIT
S ← CREATE_ARRAY
for k ← from 0 to LENGTH do
S ← EMPTY_ORDERED_SET
function EARLEY_PARSE
INIT
ADD_TO_SET
for k ← from 0 to LENGTH do
for each state in S do // S can expand during this loop
if not FINISHED then
if NEXT_ELEMENT_OF is a nonterminal then
PREDICTOR // non_terminal
else do
SCANNER // terminal
else do
COMPLETER
end
end
return chart
procedure PREDICTOR
for each in GRAMMAR_RULES_FOR do
ADD_TO_SET
end
procedure SCANNER
if j < LENGTH and a ⊂ PARTS_OF_SPEECH then
ADD_TO_SET
end
procedure COMPLETER
for each in S do
ADD_TO_SET
end

Example

Consider the following simple grammar for arithmetic expressions:

::= # the start rule
::= "+" |
::= "*" |
::= "1" | "2" | "3" | "4"

With the input:
2 + 3 * 4
This is the sequence of state sets:
The state represents a completed parse. This state also appears in S and S, which are complete sentences.

Constructing the parse forest

Earley's dissertation briefly describes an algorithm for constructing parse trees by adding a set of pointers from each non-terminal in an Earley item back to the items that caused it to be recognized. But Tomita noticed that this does not take into account the relations between symbols, so if we consider the grammar S → SS | b and the string bbb, it only notes that each S can match one or two b's, and thus produces spurious derivations for bb and bbbb as well as the two correct derivations for bbb.
Another method is to build the parse forest as you go, augmenting each Earley item with a pointer to a shared packed parse forest node labelled with a triple where s is a symbol or an LR item, and i and j give the section of the input string derived by this node. A node's contents are either a pair of child pointers giving a single derivation, or a list of "packed" nodes each containing a pair of pointers and representing one derivation. SPPF nodes are unique, but may contain more than one derivation for ambiguous parses. So even if an operation does not add an Earley item, it may still add a derivation to the item's parse forest.
  • Predicted items have a null SPPF pointer.
  • The scanner creates an SPPF node representing the non-terminal it is scanning.
  • Then when the scanner or completer advance an item, they add a derivation whose children are the node from the item whose dot was advanced, and the one for the new symbol that was advanced over.
SPPF nodes are never labeled with a completed LR item: instead they are labelled with the symbol that is produced so that all derivations are combined under one node regardless of which alternative production they come from.

Optimizations

Philippe McLean and R. Nigel Horspool in their paper combine Earley parsing with LR parsing and achieve an improvement in an order of magnitude.

Other reference materials

Category:Parsing algorithms
Category:Dynamic programming