Earley parser

Earley parsers are appealing because they can parse all context-free languages, unlike LR parsers and LL parsers, which are more typically used in compilers but which can only handle restricted classes of languages. However, the original Earley algorithm contained a bug, and its run time and memory use were high for the computers available at the time it was published. Earley parsing has since become a viable solution, as corrections and optimizations of the algorithm have been found and computers have become far more powerful.

Earley parsers perform particularly well when the rules are written left-recursively. Some grammars can be automatically rewritten so that the Earley algorithm parses the language they describe more efficiently.

The worst-case costs of the original Earley algorithm are: ${\displaystyle O(n)}$ linear time and space for non-right recursive LR(k) grammars, ${\displaystyle O(n^2)}$ quadratic time and space for unambiguous grammars, ${\displaystyle O(n^3)}$ cubic time and ${\displaystyle O(n^2)}$ quadratic space for all other context-free grammars, where ${\displaystyle n}$ is the length of the parsed string.

Earley recogniser

The following algorithm describes the Earley recogniser. The recogniser can be modified to create a parse tree as it recognises, and in that way can be turned into a parser.

In the following descriptions, α, β, and γ represent any string of terminals/nonterminals (including the empty string), X and Y represent single nonterminals, and a represents a terminal symbol.

In the following, we use Earley's dot notation: given a production X → αβ, the notation X → α • β represents a condition in which α has already been parsed and β is expected.

Input position 0 is the position prior to input. Input position n is the position after accepting the nth token. (Informally, input positions can be thought of as locations at token boundaries.) For every input position, the parser generates a state set. Each state is a tuple (X → α • β, i), consisting of

• the production currently being matched (X → α β)
• the current position in that production (visually represented by the dot •)
• the position i in the input at which the matching of this production began: the origin position

(Earley's original algorithm included a look-ahead in the state; later research showed this to have little practical effect on the parsing efficiency, and it has subsequently been dropped from most implementations.)

A state is finished when its current position is the last position of the right side of the production, that is, when there is no symbol to the right of the dot • in the visual representation of the state.

The state set at input position k is called S(k). The parser is seeded with S(0) consisting of only the top-level rule. The parser then repeatedly executes three operations: prediction, scanning, and completion.

• Prediction: For every state in S(k) of the form (X → α • Y β, j) (where j is the origin position as above), add (Y → • γ, k) to S(k) for every production in the grammar with Y on the left-hand side (Y → γ).
• Scanning: If a is the next symbol in the input stream, for every state in S(k) of the form (X → α • a β, j), add (X → α a • β, j) to S(k+1).
• Completion: For every state in S(k) of the form (Y → γ •, j), find all states in S(j) of the form (X → α • Y β, i) and add (X → α Y • β, i) to S(k).

Duplicate states are not added to the state set, only new ones. These three operations are repeated until no new states can be added to the set. The set is generally implemented as a queue of states to process, with the operation to be performed depending on what kind of state it is.

The algorithm accepts if (X → γ •, 0) ends up in S(n), where (X → γ) is the top level-rule and n the input length, otherwise it rejects.

Pseudocode

Adapted from Speech and Language Processing by Daniel Jurafsky and James H. Martin,

<span id="mweA"></span><span class="n" id="mweQ">DECLARE</span><span class="w" id="mweg"> </span><span class="k" id="mwew">ARRAY</span><span class="w" id="mwfA"> </span><span class="n" id="mwfQ">S</span><span class="o" id="mwfg">;</span> <span class="k" id="mwfw">function</span><span class="w" id="mwgA"> </span><span class="nf" id="mwgQ">INIT</span><span class="p" id="mwgg">(</span><span class="n" id="mwgw">words</span><span class="p" id="mwhA">)</span> <span class="w" id="mwhQ"> </span><span class="n" id="mwhg">S</span><span class="w" id="mwhw"> </span><span class="err" id="mwiA">←</span><span class="w" id="mwiQ"> </span><span class="n" id="mwig">CREATE_ARRAY</span><span class="p" id="mwiw">(</span><span class="nb" id="mwjA">LENGTH</span><span class="p" id="mwjQ">(</span><span class="n" id="mwjg">words</span><span class="p" id="mwjw">)</span><span class="w" id="mwkA"> </span><span class="o" id="mwkQ">+</span><span class="w" id="mwkg"> </span><span class="mi" id="mwkw">1</span><span class="p" id="mwlA">)</span> <span class="w" id="mwlQ"> </span><span class="k" id="mwlg">for</span><span class="w" id="mwlw"> </span><span class="n" id="mwmA">k</span><span class="w" id="mwmQ"> </span><span class="err" id="mwmg">←</span><span class="w" id="mwmw"> </span><span class="n" id="mwnA">from</span><span class="w" id="mwnQ"> </span><span class="mi" id="mwng">0</span><span class="w" id="mwnw"> </span><span class="k" id="mwoA">to</span><span class="w" id="mwoQ"> </span><span class="nb" id="mwog">LENGTH</span><span class="p" id="mwow">(</span><span class="n" id="mwpA">words</span><span class="p" id="mwpQ">)</span><span class="w" id="mwpg"> </span><span class="k" id="mwpw">do</span> <span class="w" id="mwqA"> </span><span class="n" id="mwqQ">S</span><span class="p" id="mwqg">[</span><span class="n" id="mwqw">k</span><span class="p" id="mwrA">]</span><span class="w" id="mwrQ"> </span><span class="err" id="mwrg">←</span><span class="w" id="mwrw"> </span><span class="n" id="mwsA">EMPTY_ORDERED_SET</span> <span class="k" id="mwsQ">function</span><span class="w" id="mwsg"> </span><span class="nf" id="mwsw">EARLEY_PARSE</span><span class="p" id="mwtA">(</span><span class="n" id="mwtQ">words</span><span class="o" id="mwtg">,</span><span class="w" id="mwtw"> </span><span class="n" id="mwuA">grammar</span><span class="p" id="mwuQ">)</span> <span class="w" id="mwug"> </span><span class="n" id="mwuw">INIT</span><span class="p" id="mwvA">(</span><span class="n" id="mwvQ">words</span><span class="p" id="mwvg">)</span> <span class="w" id="mwvw"> </span><span class="n" id="mwwA">ADD_TO_SET</span><span class="p" id="mwwQ">((</span><span class="err" id="mwwg">γ</span><span class="w" id="mwww"> </span><span class="err" id="mwxA">→</span><span class="w" id="mwxQ"> </span><span class="err" id="mwxg">•</span><span class="n" id="mwxw">S</span><span class="o" id="mwyA">,</span><span class="w" id="mwyQ"> </span><span class="mi" id="mwyg">0</span><span class="p" id="mwyw">)</span><span class="o" id="mwzA">,</span><span class="w" id="mwzQ"> </span><span class="n" id="mwzg">S</span><span class="p" id="mwzw">[</span><span class="mi" id="mw0A">0</span><span class="p" id="mw0Q">])</span> <span class="w" id="mw0g"> </span><span class="k" id="mw0w">for</span><span class="w" id="mw1A"> </span><span class="n" id="mw1Q">k</span><span class="w" id="mw1g"> </span><span class="err" id="mw1w">←</span><span class="w" id="mw2A"> </span><span class="n" id="mw2Q">from</span><span class="w" id="mw2g"> </span><span class="mi" id="mw2w">0</span><span class="w" id="mw3A"> </span><span class="k" id="mw3Q">to</span><span class="w" id="mw3g"> </span><span class="nb" id="mw3w">LENGTH</span><span class="p" id="mw4A">(</span><span class="n" id="mw4Q">words</span><span class="p" id="mw4g">)</span><span class="w" id="mw4w"> </span><span class="k" id="mw5A">do</span> <span class="w" id="mw5Q"> </span><span class="k" id="mw5g">for</span><span class="w" id="mw5w"> </span><span class="n" id="mw6A">each</span><span class="w" id="mw6Q"> </span><span class="n" id="mw6g">state</span><span class="w" id="mw6w"> </span><span class="k" id="mw7A">in</span><span class="w" id="mw7Q"> </span><span class="n" id="mw7g">S</span><span class="p" id="mw7w">[</span><span class="n" id="mw8A">k</span><span class="p" id="mw8Q">]</span><span class="w" id="mw8g"> </span><span class="k" id="mw8w">do</span><span class="w" id="mw9A"> </span><span class="c1" id="mw9Q">// S[k] can expand during this loop</span> <span class="w" id="mw9g"> </span><span class="k" id="mw9w">if</span><span class="w" id="mw-A"> </span><span class="k" id="mw-Q">not</span><span class="w" id="mw-g"> </span><span class="n" id="mw-w">FINISHED</span><span class="p" id="mw_A">(</span><span class="n" id="mw_Q">state</span><span class="p" id="mw_g">)</span><span class="w" id="mw_w"> </span><span class="k" id="mwAQA">then</span> <span class="w" id="mwAQE"> </span><span class="k" id="mwAQI">if</span><span class="w" id="mwAQM"> </span><span class="n" id="mwAQQ">NEXT_ELEMENT_OF</span><span class="p" id="mwAQU">(</span><span class="n" id="mwAQY">state</span><span class="p" id="mwAQc">)</span><span class="w" id="mwAQg"> </span><span class="k" id="mwAQk">is</span><span class="w" id="mwAQo"> </span><span class="n" id="mwAQs">a</span><span class="w" id="mwAQw"> </span><span class="n" id="mwAQ0">nonterminal</span><span class="w" id="mwAQ4"> </span><span class="k" id="mwAQ8">then</span> <span class="w" id="mwARA"> </span><span class="n" id="mwARE">PREDICTOR</span><span class="p" id="mwARI">(</span><span class="n" id="mwARM">state</span><span class="o" id="mwARQ">,</span><span class="w" id="mwARU"> </span><span class="n" id="mwARY">k</span><span class="o" id="mwARc">,</span><span class="w" id="mwARg"> </span><span class="n" id="mwARk">grammar</span><span class="p" id="mwARo">)</span><span class="w" id="mwARs"> </span><span class="c1" id="mwARw">// non_terminal</span> <span class="w" id="mwAR0"> </span><span class="k" id="mwAR4">else</span><span class="w" id="mwAR8"> </span><span class="k" id="mwASA">do</span> <span class="w" id="mwASE"> </span><span class="n" id="mwASI">SCANNER</span><span class="p" id="mwASM">(</span><span class="n" id="mwASQ">state</span><span class="o" id="mwASU">,</span><span class="w" id="mwASY"> </span><span class="n" id="mwASc">k</span><span class="o" id="mwASg">,</span><span class="w" id="mwASk"> </span><span class="n" id="mwASo">words</span><span class="p" id="mwASs">)</span><span class="w" id="mwASw"> </span><span class="c1" id="mwAS0">// terminal</span> <span class="w" id="mwAS4"> </span><span class="k" id="mwAS8">else</span><span class="w" id="mwATA"> </span><span class="k" id="mwATE">do</span> <span class="w" id="mwATI"> </span><span class="n" id="mwATM">COMPLETER</span><span class="p" id="mwATQ">(</span><span class="n" id="mwATU">state</span><span class="o" id="mwATY">,</span><span class="w" id="mwATc"> </span><span class="n" id="mwATg">k</span><span class="p" id="mwATk">)</span> <span class="w" id="mwATo"> </span><span class="k" id="mwATs">end</span> <span class="w" id="mwATw"> </span><span class="k" id="mwAT0">end</span> <span class="w" id="mwAT4"> </span><span class="n" id="mwAT8">return</span><span class="w" id="mwAUA"> </span><span class="n" id="mwAUE">chart</span> <span class="k" id="mwAUI">procedure</span><span class="w" id="mwAUM"> </span><span class="nf" id="mwAUQ">PREDICTOR</span><span class="p" id="mwAUU">((</span><span class="n" id="mwAUY">A</span><span class="w" id="mwAUc"> </span><span class="err" id="mwAUg">→</span><span class="w" id="mwAUk"> </span><span class="err" id="mwAUo">α•</span><span class="n" id="mwAUs">B</span><span class="err" id="mwAUw">β</span><span class="o" id="mwAU0">,</span><span class="w" id="mwAU4"> </span><span class="n" id="mwAU8">j</span><span class="p" id="mwAVA">)</span><span class="o" id="mwAVE">,</span><span class="w" id="mwAVI"> </span><span class="n" id="mwAVM">k</span><span class="o" id="mwAVQ">,</span><span class="w" id="mwAVU"> </span><span class="n" id="mwAVY">grammar</span><span class="p" id="mwAVc">)</span> <span class="w" id="mwAVg"> </span><span class="k" id="mwAVk">for</span><span class="w" id="mwAVo"> </span><span class="n" id="mwAVs">each</span><span class="w" id="mwAVw"> </span><span class="p" id="mwAV0">(</span><span class="n" id="mwAV4">B</span><span class="w" id="mwAV8"> </span><span class="err" id="mwAWA">→</span><span class="w" id="mwAWE"> </span><span class="err" id="mwAWI">γ</span><span class="p" id="mwAWM">)</span><span class="w" id="mwAWQ"> </span><span class="k" id="mwAWU">in</span><span class="w" id="mwAWY"> </span><span class="n" id="mwAWc">GRAMMAR_RULES_FOR</span><span class="p" id="mwAWg">(</span><span class="n" id="mwAWk">B</span><span class="o" id="mwAWo">,</span><span class="w" id="mwAWs"> </span><span class="n" id="mwAWw">grammar</span><span class="p" id="mwAW0">)</span><span class="w" id="mwAW4"> </span><span class="k" id="mwAW8">do</span> <span class="w" id="mwAXA"> </span><span class="n" id="mwAXE">ADD_TO_SET</span><span class="p" id="mwAXI">((</span><span class="n" id="mwAXM">B</span><span class="w" id="mwAXQ"> </span><span class="err" id="mwAXU">→</span><span class="w" id="mwAXY"> </span><span class="err" id="mwAXc">•γ</span><span class="o" id="mwAXg">,</span><span class="w" id="mwAXk"> </span><span class="n" id="mwAXo">k</span><span class="p" id="mwAXs">)</span><span class="o" id="mwAXw">,</span><span class="w" id="mwAX0"> </span><span class="n" id="mwAX4">S</span><span class="p" id="mwAX8">[</span><span class="n" id="mwAYA">k</span><span class="p" id="mwAYE">])</span> <span class="w" id="mwAYI"> </span><span class="k" id="mwAYM">end</span> <span class="k" id="mwAYQ">procedure</span><span class="w" id="mwAYU"> </span><span class="nf" id="mwAYY">SCANNER</span><span class="p" id="mwAYc">((</span><span class="n" id="mwAYg">A</span><span class="w" id="mwAYk"> </span><span class="err" id="mwAYo">→</span><span class="w" id="mwAYs"> </span><span class="err" id="mwAYw">α•</span><span class="n" id="mwAY0">a</span><span class="err" id="mwAY4">β</span><span class="o" id="mwAY8">,</span><span class="w" id="mwAZA"> </span><span class="n" id="mwAZE">j</span><span class="p" id="mwAZI">)</span><span class="o" id="mwAZM">,</span><span class="w" id="mwAZQ"> </span><span class="n" id="mwAZU">k</span><span class="o" id="mwAZY">,</span><span class="w" id="mwAZc"> </span><span class="n" id="mwAZg">words</span><span class="p" id="mwAZk">)</span> <span class="w" id="mwAZo"> </span><span class="k" id="mwAZs">if</span><span class="w" id="mwAZw"> </span><span class="n" id="mwAZ0">k</span><span class="w" id="mwAZ4"> </span><span class="o" id="mwAZ8"><</span><span class="w" id="mwAaA"> </span><span class="nb" id="mwAaE">LENGTH</span><span class="p" id="mwAaI">(</span><span class="n" id="mwAaM">words</span><span class="p" id="mwAaQ">)</span><span class="w" id="mwAaU"> </span><span class="k" id="mwAaY">and</span><span class="w" id="mwAac"> </span><span class="n" id="mwAag">a</span><span class="w" id="mwAak"> </span><span class="err" id="mwAao">⊂</span><span class="w" id="mwAas"> </span><span class="n" id="mwAaw">PARTS_OF_SPEECH</span><span class="p" id="mwAa0">(</span><span class="n" id="mwAa4">words</span><span class="p" id="mwAa8">[</span><span class="n" id="mwAbA">k</span><span class="p" id="mwAbE">])</span><span class="w" id="mwAbI"> </span><span class="k" id="mwAbM">then</span> <span class="w" id="mwAbQ"> </span><span class="n" id="mwAbU">ADD_TO_SET</span><span class="p" id="mwAbY">((</span><span class="n" id="mwAbc">A</span><span class="w" id="mwAbg"> </span><span class="err" id="mwAbk">→</span><span class="w" id="mwAbo"> </span><span class="err" id="mwAbs">α</span><span class="n" id="mwAbw">a</span><span class="err" id="mwAb0">•β</span><span class="o" id="mwAb4">,</span><span class="w" id="mwAb8"> </span><span class="n" id="mwAcA">j</span><span class="p" id="mwAcE">)</span><span class="o" id="mwAcI">,</span><span class="w" id="mwAcM"> </span><span class="n" id="mwAcQ">S</span><span class="p" id="mwAcU">[</span><span class="n" id="mwAcY">k</span><span class="o" id="mwAcc">+</span><span class="mi" id="mwAcg">1</span><span class="p" id="mwAck">])</span> <span class="w" id="mwAco"> </span><span class="k" id="mwAcs">end</span> <span class="k" id="mwAcw">procedure</span><span class="w" id="mwAc0"> </span><span class="nf" id="mwAc4">COMPLETER</span><span class="p" id="mwAc8">((</span><span class="n" id="mwAdA">B</span><span class="w" id="mwAdE"> </span><span class="err" id="mwAdI">→</span><span class="w" id="mwAdM"> </span><span class="err" id="mwAdQ">γ•</span><span class="o" id="mwAdU">,</span><span class="w" id="mwAdY"> </span><span class="n" id="mwAdc">x</span><span class="p" id="mwAdg">)</span><span class="o" id="mwAdk">,</span><span class="w" id="mwAdo"> </span><span class="n" id="mwAds">k</span><span class="p" id="mwAdw">)</span> <span class="w" id="mwAd0"> </span><span class="k" id="mwAd4">for</span><span class="w" id="mwAd8"> </span><span class="n" id="mwAeA">each</span><span class="w" id="mwAeE"> </span><span class="p" id="mwAeI">(</span><span class="n" id="mwAeM">A</span><span class="w" id="mwAeQ"> </span><span class="err" id="mwAeU">→</span><span class="w" id="mwAeY"> </span><span class="err" id="mwAec">α•</span><span class="n" id="mwAeg">B</span><span class="err" id="mwAek">β</span><span class="o" id="mwAeo">,</span><span class="w" id="mwAes"> </span><span class="n" id="mwAew">j</span><span class="p" id="mwAe0">)</span><span class="w" id="mwAe4"> </span><span class="k" id="mwAe8">in</span><span class="w" id="mwAfA"> </span><span class="n" id="mwAfE">S</span><span class="p" id="mwAfI">[</span><span class="n" id="mwAfM">x</span><span class="p" id="mwAfQ">]</span><span class="w" id="mwAfU"> </span><span class="k" id="mwAfY">do</span> <span class="w" id="mwAfc"> </span><span class="n" id="mwAfg">ADD_TO_SET</span><span class="p" id="mwAfk">((</span><span class="n" id="mwAfo">A</span><span class="w" id="mwAfs"> </span><span class="err" id="mwAfw">→</span><span class="w" id="mwAf0"> </span><span class="err" id="mwAf4">α</span><span class="n" id="mwAf8">B</span><span class="err" id="mwAgA">•β</span><span class="o" id="mwAgE">,</span><span class="w" id="mwAgI"> </span><span class="n" id="mwAgM">j</span><span class="p" id="mwAgQ">)</span><span class="o" id="mwAgU">,</span><span class="w" id="mwAgY"> </span><span class="n" id="mwAgc">S</span><span class="p" id="mwAgg">[</span><span class="n" id="mwAgk">k</span><span class="p" id="mwAgo">])</span> <span class="w" id="mwAgs"> </span><span class="k" id="mwAgw">end</span>