N-gram models and FSTs revisited

What do n-gram models miss? Two random sentences from Alice in Wonderland:
1. At least I know who I WAS when I got up this morning, but I think I must have been changed several times since then.
2. The Cat's head began fading away the moment he was gone, and, by the time he had come back with the Duchess, it had entirely disappeared.
Sentences generated by an n-gram model trained on Alice in Wonderland.
Morphology and FSTs

Segmentation, embedding, and recursion

A complicated English sentence
They have done so very reluctantly and only after courts throughout the nation, and the attorneys general of 40 states, questioned the legality of a securitized system of homeownership that has impoverished tens of millions.
Rudimentary segmentation: finding noun phrases
[They] have done so very reluctantly and only after [courts throughout the nation], and [the attorneys general of 40 states], questioned [the legality of a securitized system of homeownership that has impoverished tens of millions].
Noun phrases can contain noun phrases.
Handling recursion in segmentation, or parsing more generally, requires that we go beyond finite state automata.
But lots of useful information can be gleaned from a sentence ignoring recursion.
Partial parsing as an alternative

Chunking

Goal: identify constituents of a given set of types, for example, noun phrase, prepositional phrase, where the constituent definitions exclude the possibility of recursion

Part-of-speech tagging

>>> sent = '''The disastrous disarray in the housing industry is a direct
	result of decisions taken during the deregulation frenzy
	of the Clinton presidency'''
>>> text = nltk.word_tokenize(sent)
>>> pos = nltk.pos_tag(text)
>>> pos
[('The', 'DT'), ('disastrous', 'JJ'), ('disarray', 'NN'), ('in', 'IN'), ('the', 'DT'),
 ('housing', 'NN'), ('industry', 'NN'), ('is', 'VBZ'), ('a', 'DT'), ('direct', 'JJ'),
 ('result', 'NN'), ('of', 'IN'), ('decisions', 'NNS'), ('taken', 'VBN'), ('during', 'IN'),
 ('the', 'DT'), ('deregulation', 'NN'), ('frenzy', 'NN'), ('of', 'IN'), ('the', 'DT'),
 ('Clinton', 'NNP'), ('presidency', 'NN')]

Identifying chunks as sequences of POS tags

Chunk grammar: sequence of rules consisting of regular expressions

>>> grammar = r"""
  NP:  {<DT|PP\$>?<JJ>*<NN|NNP>?<NN>}   # chunk determiner/possessive, adjectives and noun-noun pairs
"""
>>> chunker = nltk.RegexpParser(grammar)
>>> parse = chunker.parse(pos)
>>> print parse
(S
  (NP The/DT disastrous/JJ disarray/NN)
  in/IN
  (NP the/DT housing/NN industry/NN)
  is/VBZ
  (NP a/DT direct/JJ result/NN)
  of/IN
  decisions/NNS
  taken/VBN
  during/IN
  (NP the/DT deregulation/NN frenzy/NN)
  of/IN
  (NP the/DT Clinton/NNP presidency/NN))
>>> parse.draw()

Chunking with multiple passes through the input

Training a chunker

Chunking based on more than the current POS tag
- Joey/NN sold/VBD the/DT farmer/NN rice/NN
- Joey/NN broke/VBD the/DT computer/NN monitor/NN

IOB tags ("inside", "outside", "beginning")

We PRP B-NP
saw VBD O
the DT B-NP
little JJ I-NP
yellow JJ I-NP
dog NN I-NP

N-gram taggers
Training corpus: the CoNLL 2000 shared task
Training an n-gram chunker
- N-gram chunkers:
- Backoff chunker: assigning an IOB tag when the context is not in the context-tag dictionary
Code for training an n-gram chunker in NLTK: chunk_train.py