{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 3: Open Information Extraction\n",
    "(Due 2019-4-18 12:00)\n",
    "\n",
    "In this lab you will be implementing an *open information extraction* system, almost entirely from scratch. Information extraction takes a body of freeform text and extracts the contained information in a computer interpretable form. The word *open* simply means that the text/facts are arbitrary, so it will work with any input rather than a specific domain (e.g. legal texts).\n",
    "\n",
    "As an example, given the input:\n",
    "\n",
    "> \"Trolls really don't like the sun.\"\n",
    "  \n",
    "\n",
    "you may extract the \"fact\":\n",
    "```\n",
    "('Trolls', 'do not like', 'the sun')\n",
    "```\n",
    "\n",
    "The approach is based on the paper \"*Identifying Relations for Open Information Extraction*\", by Fader, Soderland & Etzioni. Parts have been updated with more recent, or more ML, techniques however (Q3 and Q4 match the paper; Q1 and Q2 don't). You don't have to read the paper as this workbook takes you through the process; this lab is also intended as a tutorial/opportunity to see one way such a system works. Some parts are rule based, as that's still often the case.\n",
    "\n",
    "The steps of the system are as follows:\n",
    "*  Tokenise and split on sentences *(provided)*\n",
    "\n",
    "\n",
    "1. Part of speech tagging - token level\n",
    "2. Part of speech tagging - sentence level\n",
    "3. Named entity resolution\n",
    "4. Relation extraction\n",
    "\n",
    "\n",
    "*  Summarise \"*20,000 leagues under the seas*\" by Jules Verne *(provided)*\n",
    "\n",
    "A simple NLP library, called `ogonek`, is provided. It has some basic functionality that you will require; this is for two reasons:\n",
    "1. This has to run on BUCS computers, and normal NLP libraries do not.\n",
    "2. To keep certain low level functionality consistent, so automarking is easier.\n",
    "\n",
    "Its documentation can be found below in a markdown cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Marking and Submission\n",
    "\n",
    "This lab exercise is marked, and contributes to your final grade. For this lab exercise there are 4 places where you are expected to enter your own code, for 30 marks overall. Every place you have to add code is indicated by\n",
    "\n",
    "`# **************************************************************** n marks`\n",
    "\n",
    "with instructions above the code block.\n",
    "\n",
    "Remember that the amount of work you put in should be proportional to the number of marks! Also remember that if you get stuck you may want to try the next question, even if you have to sacrifice marks by using libraries/the wrong algorithm etc. to handle an earlier question.\n",
    "\n",
    "Please submit your completed workbook using Moodle before **12:00 on the 18th April 2019**. The workbook you submit must be an `.ipynb` file, which is saved into the directory you're running Jupyter; alternatively you can download it from the menu above using `File -> Download As -> Notebook (.ipynb)`. Remember to save your work regularly (Save and checkpoint in the File menu, the icon of a floppy disk, or Ctrl-S); the version you submit should have all code blocks showing the results (if any) of execution below them. We will try to get you feedback within two weeks, though reality and extensions often impinge on that hope."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Auto marking\n",
    "\n",
    "This lab will be (partially) automarked, though a human will check when a part does not work, to assign partial credit. For this reason please do not change the interfaces that have been provided to you. Do remember that marking code is hard and time consuming; please try and be neat!\n",
    "\n",
    "Do check your entire workbook runs before submission: Click on *Restart & Run All* under the *Kernel* menu, and verify all code cells complete as you expect (this may take some time)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ogonek\n",
    "\n",
    "A tiny NLP library, that contains exactly the functionality I don't want you to implement for this coursework!\n",
    "\n",
    "\n",
    "\n",
    "### Tokenisation and sentence splitting\n",
    "`ogonek.Tokenise()`\n",
    "\n",
    "A class that tokenises some text and splits it into sentences. Construct an instance with `tokens = ogonek.Tokenise('My text')`; it then has the same interface as a list of lists:\n",
    "* `len(tokens)`: Number of extracted sentences (not words)\n",
    "* `tokens[i]`: Sentence i, where i ranges from 0 to one less than `len(tokens)`. A sentence is a list of tokens.\n",
    "\n",
    "\n",
    "\n",
    "### Word vectors\n",
    "`ogonek.Glove()`\n",
    "\n",
    "Constructing a `glove = ogonek.Glove()` object loads a heavily pruned Glove word vectors from the file `baby_glove.zip` into memory, and will then translate tokens into word vectors. Note that it automatically lowercases any token it is handed, so you don't need to. Has the following interface:\n",
    "* `glove.len_vec()` - Returns the length of the word vectors; should be 300.\n",
    "* `len(glove)` - Returns how many word vectors it knows of.\n",
    "* `token in glove` - Returns `True` if it has a word vector for that token, `False` otherwise.\n",
    "* `glove[token]` - Returns the word vector for the given token; raises an error if it does not have one.\n",
    "* `glove.decode(token)` - Returns the word vector for the given token, but if the word vector is unknown returns a vector of zeros instead (silent failure).\n",
    "* `glove.decodes(list of tokens)` - Returns a list of word vectors, one for each token. Has the same silent failure behaviour as `decode`.\n",
    "\n",
    "\n",
    "\n",
    "### Groningen Meaning Bank dataset\n",
    "`ogonek.GMB()`\n",
    "\n",
    "Provides access to the Groningen Meaning Bank dataset, which is supplied in the file `ner_dataset.csv`. Replicated the interface of the tokenisation system as far as it can. Construct with `gmb = ogonek.GMB()` the  use:\n",
    "* `len(gmb)`: Number of sentences (not words) in data set\n",
    "* `gmb[i]`: Sentence i, where i ranges from 0 to one less than `len(gmb)`. A sentence is a list of tokens.\n",
    "* `gmb.pos(i)`: A list of POS tags that match with sentence i. Note that these are the full Penn Treebank tags (not the reduced set used below).\n",
    "* `gmb.ner(i)`: A list of named entities that match with sentence i. Using outside-inside scheme.\n",
    "\n",
    "\n",
    "\n",
    "### Pretty printing\n",
    "\n",
    "`ogonek.aligned_print(*)` takes multiple lists and prints them out, aligning them so that all elements in position 0 of all lists are aligned vertically (extra space added as required), and then elements in position 1 and so on. For showing tags and a sentence with everything aligned. Also does word wrap and colour coding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import time\n",
    "import string\n",
    "import re\n",
    "\n",
    "import numpy\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import ogonek\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Useful variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dictionary giving descriptions of the reduced pos...\n",
    "rpos_desc = {'C' : 'Coordinating conjunction',\n",
    "             '0' : 'Cardinal number',\n",
    "             'D' : 'Determiner',\n",
    "             'E' : 'Existential there',\n",
    "             'I' : 'Preposition or subordinating conjunction',\n",
    "             'J' : 'Adjective',\n",
    "             'N' : 'Noun',\n",
    "             'P' : 'Predeterminer',\n",
    "             'S' : 'Possessive ending',\n",
    "             'M' : 'Pronoun',\n",
    "             'R' : 'Adverb',\n",
    "             'Z' : 'Particle',\n",
    "             'T' : 'to',\n",
    "             'V' : 'Verb',\n",
    "             'A' : 'Anything else',\n",
    "             '.' : 'All punctuation'}\n",
    "\n",
    "\n",
    "\n",
    "# Reduced list of part of speech tags as a list...\n",
    "num_to_rpos = ['C', '0', 'D', 'E', 'I', 'J', 'N', 'P',\n",
    "               'S', 'M', 'R', 'Z', 'T', 'V', 'A', '.']\n",
    "\n",
    "\n",
    "\n",
    "# Dictionary that maps a reduced part of speech\n",
    "# tag to it's index in the above list; useful for vectors/matrices etc...\n",
    "rpos_to_num = {'C' : 0,\n",
    "               '0' : 1,\n",
    "               'D' : 2,\n",
    "               'E' : 3,\n",
    "               'I' : 4,\n",
    "               'J' : 5,\n",
    "               'N' : 6,\n",
    "               'P' : 7,\n",
    "               'S' : 8,\n",
    "               'M' : 9,\n",
    "               'R' : 10,\n",
    "               'Z' : 11,\n",
    "               'T' : 12,\n",
    "               'V' : 13,\n",
    "               'A' : 14,\n",
    "               '.' : 15}\n",
    "\n",
    "\n",
    "\n",
    "# Dictionary that maps the full part of speech tags to the reduced set...\n",
    "pos_to_rpos = {'CC' : 'C',\n",
    "               'CD' : '0',\n",
    "               'DT' : 'D',\n",
    "               'EX' : 'E',\n",
    "               'FW' : 'A',\n",
    "               'IN' : 'I',\n",
    "               'JJ' : 'J',\n",
    "               'JJR' : 'J',\n",
    "               'JJS' : 'J',\n",
    "               'LS' : 'A',\n",
    "               'MD' : 'A',\n",
    "               'NN' : 'N',\n",
    "               'NNS' : 'N',\n",
    "               'NNP' : 'N',\n",
    "               'NNPS' : 'N',\n",
    "               'PDT' : 'P',\n",
    "               'POS' : 'S',\n",
    "               'PRP' : 'M',\n",
    "               'PRP$' : 'M',\n",
    "               'RB' : 'R',\n",
    "               'RBR' : 'R',\n",
    "               'RBS' : 'R',\n",
    "               'RP' : 'Z',\n",
    "               'SYM' : 'A',\n",
    "               'TO' : 'T',\n",
    "               'UH' : 'A',\n",
    "               'VB' : 'V',\n",
    "               'VBD' : 'V',\n",
    "               'VBG' : 'V',\n",
    "               'VBN' : 'V',\n",
    "               'VBP' : 'V',\n",
    "               'VBZ' : 'V',\n",
    "               'WDT' : 'D',\n",
    "               'WP' : 'M',\n",
    "               'WP$' : 'S',\n",
    "               'WRB' : 'R',\n",
    "               '-' : '.',\n",
    "               'LRB' : '.',\n",
    "               'RRB' : '.',\n",
    "               '``' : '.',\n",
    "               '\"' : '.',\n",
    "               '.' : '.',\n",
    "               ',' : '.',\n",
    "               ';' : '.',\n",
    "               ':' : '.',\n",
    "               '$' : '.'}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load book, tokenise and split on sentences\n",
    "The below code reads in the book, chops it down to just the text of the book, and then tokenises it using the provided `ogonek` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01. It was the regime of verticality .\n",
      "02. Now then , the tides are not strong in the Pacific , and if you can not unballast the Nautilus , which seems impossible to me , I do not see how it will float off . \"\n",
      "03. Captain Nemo left the cave , and we climbed back up the bank of shellfish in the midst of these clear waters not yet disturbed by divers at work .\n",
      "04. Likewise the pilothouse and the beacon housing were withdrawn into the hull until they lay exactly flush with it .\n",
      "05. Instead of digging all around the Nautilus , which would have entailed even greater difficulties , Captain Nemo had an immense trench outlined on the ice , eight meters from our port quarter .\n",
      "06. We would not go five miles without bumping into a fellow countryman .\n",
      "07. The oars , mast , and sail are in the skiff .\n",
      "08. Under existing conditions some ten men at the most should be enough to operate it . \"\n",
      "09. Nobody appeared on our arrival .\n",
      "10. We gasped .\n"
     ]
    }
   ],
   "source": [
    "# Loop file, only keeping lines between indicators...\n",
    "lines = []\n",
    "record = False\n",
    "\n",
    "with open('20,000 Leagues Under the Seas.txt', 'r') as fin:\n",
    "    for line in fin:\n",
    "        if record:\n",
    "            if line.startswith('***END OF THE PROJECT GUTENBERG'):\n",
    "                break\n",
    "      \n",
    "            lines.append(line)\n",
    "    \n",
    "        else:\n",
    "            if line.startswith('***START OF THE PROJECT GUTENBERG'):\n",
    "                record = True\n",
    "\n",
    "text = ''.join(lines)\n",
    "\n",
    "\n",
    "# Tokenise...\n",
    "under_the_seas = ogonek.Tokenise(text)\n",
    "\n",
    "\n",
    "# Print 10 random sentences to check it worked...\n",
    "numpy.random.seed(0)\n",
    "\n",
    "for i in range(10):\n",
    "    toks = numpy.random.choice(under_the_seas)\n",
    "    print('{:02d}. {}'.format(i+1, ' '.join(toks)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Part of speech tagging - token level\n",
    "\n",
    "The goal here is to train a classifier that indicates which of the part of speech tags (provided above) each word is. For this initial approach you're going to treat words (tokens) individually, without context. For features the Glove word vectors are going to be used (provided by `ogonek.Glove()`).\n",
    "\n",
    "Instead of training a single classifier a slight modification of a random kitchen sink for each part of speech tag is going to be used. Specifically, a logistic random kitchen sink that indicates the probability that the word should be labelled with the associated tag. This is a *one vs all* classifier - you have a classifier for every tag, run them all on each word, and then select the tag with the highest probability (it's inconsistent - they won't sum to 1!). A logistic random kitchen sink is simply a normal kitchen sink that is pushed through a sigmoid function (in neural network terms, the final layer has a non-linearity),\n",
    "$$\\operatorname{Sig}(z) = \\frac{1}{1 + e^{-z}}$$\n",
    "such that the final binary classifier is\n",
    "$$P(\\textrm{tag}) = \\operatorname{Sig}\\left(\\sum_{k \\in K} \\alpha_k \\phi\\left(\\vec{x} \\cdot \\vec{w}_k\\right)\\right)$$\n",
    "For the cost function you should maximise the log likelihood of the dataset. This will require gradient descent; Nestorov or better, including backtracking line search for step size, to get all marks (it will be very slow otherwise). You can either differentiate yourself, copy the equations from lecture 7 of ML1 (where they were derived), or use tensor flow; your choice! The non-linearity, $\\phi(\\cdot)$ is up to you ($\\sin$ works). Remember to include the original features plus the value `1` when creating the extended feature vector (so it has a bias term). It is suggested to use 300 random features, in addition to the 300 provided by glove and bias term (total of 601), as that keeps the resulting data matrix during training small enough that it completes reasonably quickly.\n",
    "\n",
    "The Groningen Meaning Bank dataset has been provided; it can be accessed via the class `ogonek.GMB`. It includes lots of sentences, each as a list of tokens, plus part of speech tags as a list aligned with the sentence.\n",
    "Source: https://www.kaggle.com/abhinavwalia95/entity-annotated-corpus\n",
    "\n",
    "Advice:\n",
    "* Be careful of the *maximise* - the gradient descent slide deck is talking about minimisation! Just need to flip the correct signs...\n",
    "* Your choice on what to return from `train_tag_model` - tuple of model parameters or an object are both sensible choices.\n",
    "* If not using tensor flow write a function to calculate the cost and another to calculate the gradient, so you don't duplicate that code and can test it.\n",
    "* Pay attention to `split`, i.e. make sure you're training on the test set!\n",
    "* Initialise `alpha` with a very small amount of noise, e.g. Gaussian(0, 1) scaled so the vector length is 1.\n",
    "* Remember to limit iteration counts, e.g. to 256. Detect convergence as no improvement for n iterations, where n is a small number, e.g. 8. These numbers are too low really, but good enough for this exercise, and keep the runtime under 2 minutes per classifier (at least, for my implementation).\n",
    "* A good starting step size for the backtracking line search is 1.0\n",
    "* Be careful when calculating the log likelihood, as it is easy to get too close to zero/one and get NaN/infinity from `log`. You will want to `numpy.clip(p, 1e-3, 1 - 1e-3)` the calculated probabilities, to avoid this.\n",
    "* You have 16 models to train - write efficient code! Vectorisation is definitely needed.\n",
    "* For your own sanity you may want to print progress reports during training, to check it's working.\n",
    "* This is quite a subtle thing to implement - a slight mistake can cause it to utterly fail. Test and print everything if it isn't converging!\n",
    "* My model answer gets 90% correct.\n",
    "* There was a mistake in lecture 3 for the backtracking line search; make sure you have the most recent version!\n",
    "\n",
    "\n",
    "**(16 marks)**\n",
    "* 6 marks for the logistic random kitchen sink implementation.\n",
    "* 3 marks for initialising the step size with a backtracking line search.\n",
    "* 3 marks for implementing a Nesterov, or better, algorithm.\n",
    "* 4 marks for training a classifier for each part of speech, and finishing the `token_pos()` function.\n",
    "* You may use the random kitchen sink code provided in the lecture, though that is not exactly what is being asked for, so will need heavy editing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GMB sentences = 47959\n",
      "\n",
      "\u001b[0mThe U.S.  space agency is  making final preparations to launch the first \n",
      "\u001b[31mDT  NNP   NN    NN     VBZ VBG    JJ    NNS          TO VB     DT  JJ    \n",
      "\u001b[34mO   B-geo O     O      O   O      O     O            O  O      O   O     \n",
      "\u001b[0m\n",
      "\u001b[0mdirect space probe to the distant planet of Pluto . \n",
      "\u001b[31mJJ     NN    NN    TO DT  JJ      NN     IN NNP   . \n",
      "\u001b[34mO      O     O     O  O   O       O      O  B-geo O \n",
      "\u001b[0m\n",
      "\u001b[0mOn Monday , the freighter Torgelow was hijacked off the eastern coast of \n",
      "\u001b[31mIN NNP    , DT  NN        NNP      VBD VBN      IN  DT  JJ      NN    IN \n",
      "\u001b[34mO  B-tim  O O   O         B-art    O   O        O   O   O       O     O  \n",
      "\u001b[0m\n",
      "\u001b[0mSomalia . \n",
      "\u001b[31mNNP     . \n",
      "\u001b[34mB-geo   O \n",
      "\u001b[0m\n",
      "\u001b[0mChile and Bolivia are associate members . \n",
      "\u001b[31mNNP   CC  NNP     VBP JJ        NNS     . \n",
      "\u001b[34mB-gpe O   B-gpe   O   O         O       O \n",
      "\u001b[0m\n",
      "\u001b[0mVenezuela has freed 11 Colombian soldiers who had been detained after entering \n",
      "\u001b[31mNNP       VBZ VBN   CD JJ        NNS      WP  VBD VBN  VBN      IN    VBG      \n",
      "\u001b[34mB-geo     O   O     O  B-gpe     O        O   O   O    O        O     O        \n",
      "\u001b[0m\n",
      "\u001b[0mVenezuelan territory without authorization . \n",
      "\u001b[31mJJ         NN        IN      NN            . \n",
      "\u001b[34mB-gpe      O         O       O             O \n",
      "\u001b[0m\n",
      "\u001b[0mHowever , the closing figure of 12,012 points was below the record level of \n",
      "\u001b[31mRB      , DT  NN      NN     IN CD     NNS    VBD IN    DT  NN     NN    IN \n",
      "\u001b[34mO       O O   O       O      O  O      O      O   O     O   O      O     O  \n",
      "\u001b[0m\n",
      "\u001b[0m12,049 points reached during trading Wednesday . \n",
      "\u001b[31mCD     NNS    VBN     IN     NN      NNP       . \n",
      "\u001b[34mO      O      O       O      O       B-tim     O \n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Load word vectors; in a seperate cell as this takes a couple seconds...\n",
    "glove = ogonek.Glove()\n",
    "\n",
    "\n",
    "# Groningen Meaning Bank dataset - a set of sentences each tagged\n",
    "# with part of speech and named entity recognitiuon tags...\n",
    "gmb = ogonek.GMB()\n",
    "print('GMB sentences = {}'.format(len(gmb)))\n",
    "print()\n",
    "\n",
    "\n",
    "# Print out 5 random sentences from GMB with POS and NER tags, to illustrate the data...\n",
    "numpy.random.seed(1)\n",
    "for _ in range(5):\n",
    "    i = numpy.random.randint(len(gmb))\n",
    "    ogonek.aligned_print(gmb[i], gmb.pos(i), gmb.ner(i))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using 14387 sentences for training\n",
      "Training D\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.4/dist-packages/ipykernel_launcher.py:18: RuntimeWarning: overflow encountered in exp\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Selected step size = 0.0351844\n",
      "  Iter 0 ll = -233250\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.4/dist-packages/ipykernel_launcher.py:26: RuntimeWarning: overflow encountered in exp\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Finished, best ll = -53911.4\n",
      "  (took 26.6373 seconds)\n",
      "Training P\n",
      "  Selected step size = 0.512\n",
      "  Iter 0 ll = -21261.4\n",
      "  Iter 50 ll = -12807.6\n",
      "  Iter 100 ll = -7959.03\n",
      "  Iter 150 ll = -5154.89\n",
      "  Iter 200 ll = -3469.64\n",
      "  Iter 250 ll = -2578.67\n",
      "  Finished, best ll = -2412.91\n",
      "  (took 96.6122 seconds)\n",
      "Training N\n",
      "  Selected step size = 1.5325e-06\n",
      "  Iter 0 ll = -197010\n",
      "  Iter 50 ll = -65428.3\n",
      "  Iter 100 ll = -58664.6\n",
      "  Iter 150 ll = -55824.6\n",
      "  Iter 200 ll = -54268.9\n",
      "  Iter 250 ll = -53293.4\n",
      "  Finished, best ll = -53215\n",
      "  (took 114.763 seconds)\n",
      "Training Z\n",
      "  Selected step size = 0.512\n",
      "  Iter 0 ll = -25964.9\n",
      "  Iter 50 ll = -14713.8\n",
      "  Iter 100 ll = -7952.13\n",
      "  Iter 150 ll = -6032.05\n",
      "  Iter 200 ll = -4920.06\n",
      "  Iter 250 ll = -4208.67\n",
      "  Finished, best ll = -3835.7\n",
      "  (took 100.289 seconds)\n",
      "Training T\n",
      "  Selected step size = 0.4096\n",
      "  Iter 0 ll = -69049.3\n",
      "  Iter 50 ll = -11170.7\n",
      "  Iter 100 ll = -6425.73\n",
      "  Iter 150 ll = -4132.69\n",
      "  Iter 200 ll = -3006.89\n",
      "  Finished, best ll = -2951.64\n",
      "  (took 87.0055 seconds)\n",
      "Training 0\n",
      "  Selected step size = 0.32768\n",
      "  Iter 0 ll = -65744.8\n",
      "  Iter 50 ll = -8490.85\n",
      "  Iter 100 ll = -4436.59\n",
      "  Iter 150 ll = -3168.09\n",
      "  Finished, best ll = -2765.15\n",
      "  (took 73.4933 seconds)\n",
      "Training M\n",
      "  Selected step size = 0.32768\n",
      "  Iter 0 ll = -69826.6\n",
      "  Finished, best ll = -32851.4\n",
      "  (took 29.2709 seconds)\n",
      "Training .\n",
      "  Selected step size = 0.0351844\n",
      "  Iter 0 ll = -199870\n",
      "  Finished, best ll = -33479.5\n",
      "  (took 27.2261 seconds)\n",
      "Training S\n",
      "  Selected step size = 0.4096\n",
      "  Iter 0 ll = -43680.8\n",
      "  Iter 50 ll = -10956.6\n",
      "  Iter 100 ll = -5900.82\n",
      "  Iter 150 ll = -3421.3\n",
      "  Iter 200 ll = -2537.23\n",
      "  Iter 250 ll = -1825.84\n",
      "  Finished, best ll = -1798.21\n",
      "  (took 97.5427 seconds)\n",
      "Training R\n",
      "  Selected step size = 0.4096\n",
      "  Iter 0 ll = -71155.8\n",
      "  Finished, best ll = -53433.1\n",
      "  (took 28.8113 seconds)\n",
      "Training E\n",
      "  Selected step size = 0.4096\n",
      "  Iter 0 ll = -22055.7\n",
      "  Iter 50 ll = -11177.6\n",
      "  Iter 100 ll = -5141.08\n",
      "  Iter 150 ll = -3193.84\n",
      "  Iter 200 ll = -2164.27\n",
      "  Iter 250 ll = -1736.05\n",
      "  Finished, best ll = -1694.61\n",
      "  (took 97.5396 seconds)\n",
      "Training V\n",
      "  Selected step size = 1.91562e-06\n",
      "  Iter 0 ll = -213118\n",
      "  Iter 50 ll = -46981.1\n",
      "  Iter 100 ll = -43010.7\n",
      "  Iter 150 ll = -41260.6\n",
      "  Iter 200 ll = -40255.9\n",
      "  Iter 250 ll = -39599.2\n",
      "  Finished, best ll = -39546.2\n",
      "  (took 110.953 seconds)\n",
      "Training I\n",
      "  Selected step size = 1.91562e-06\n",
      "  Iter 0 ll = -197340\n",
      "  Iter 50 ll = -12042.9\n",
      "  Iter 100 ll = -8901.47\n",
      "  Iter 150 ll = -7671.5\n",
      "  Iter 200 ll = -6968.14\n",
      "  Iter 250 ll = -6496.62\n",
      "  Finished, best ll = -6457.64\n",
      "  (took 110.927 seconds)\n",
      "Training C\n",
      "  Selected step size = 0.4096\n",
      "  Iter 0 ll = -69818.8\n",
      "  Iter 50 ll = -9464.71\n",
      "  Iter 100 ll = -4629.98\n",
      "  Iter 150 ll = -2848.04\n",
      "  Iter 200 ll = -2019.23\n",
      "  Iter 250 ll = -1570.29\n",
      "  Finished, best ll = -1563.38\n",
      "  (took 96.3889 seconds)\n",
      "Training J\n",
      "  Selected step size = 0.0549756\n",
      "  Iter 0 ll = -186865\n",
      "  Finished, best ll = -61762.7\n",
      "  (took 35.9307 seconds)\n",
      "Training A\n",
      "  Selected step size = 0.512\n",
      "  Iter 0 ll = -35662\n",
      "  Iter 50 ll = -12186\n",
      "  Iter 100 ll = -5845.57\n",
      "  Iter 150 ll = -3870.24\n",
      "  Iter 200 ll = -2930.92\n",
      "  Iter 250 ll = -2233.33\n",
      "  Finished, best ll = -2178.08\n",
      "  (took 96.3168 seconds)\n"
     ]
    }
   ],
   "source": [
    "# A test/train split - train with [0:split], test with [split:len(gmb)]\n",
    "split = int(len(gmb) * 0.3) # Have a lot of data, and don't want you waiting around too long to train!\n",
    "print('Using {} sentences for training'.format(split))\n",
    "\n",
    "\n",
    "\n",
    "# **************************************************************** 16 marks\n",
    "class LRKS:\n",
    "    def __init__(self, x, y, K = 300, max_iter = 256):\n",
    "        # Extend x with K new features...\n",
    "        w = numpy.random.standard_normal((K, x.shape[1]))\n",
    "        nf = numpy.sin(numpy.einsum('ef,gf->eg', x, w))\n",
    "        ex = numpy.concatenate((numpy.ones((x.shape[0],1)), x, nf), axis=1)\n",
    "        \n",
    "        # Define error (log liklihood)...\n",
    "        def ll(alpha):\n",
    "            z = ex.dot(alpha)\n",
    "            p = numpy.clip(1 / (1 + numpy.exp(-z)), 1e-3, 1-1e-3)\n",
    "            \n",
    "            return numpy.log(y*p + (1-y)*(1-p)).sum()\n",
    "            #return (y * numpy.log(p) + (1-y) * numpy.log(1-p)).sum()\n",
    "            \n",
    "        # Define gradient...\n",
    "        def grad(alpha):\n",
    "            z = ex.dot(alpha)\n",
    "            delta = y - 1 / (1 + numpy.exp(-z))\n",
    "            return numpy.einsum('e,ef->f', delta, ex)\n",
    "        \n",
    "        # Backtracking line search to find step size (eta)...\n",
    "        alpha = numpy.random.standard_normal(ex.shape[1])\n",
    "        alpha /= numpy.sqrt(numpy.square(alpha).sum())\n",
    "        \n",
    "        g = grad(alpha)\n",
    "        \n",
    "        eta = 1.0\n",
    "        while True:\n",
    "            guess = ll(alpha) + 0.5 * eta * numpy.sqrt(numpy.square(g).sum())\n",
    "            actual = ll(alpha + eta * g)\n",
    "            if actual > guess:\n",
    "                break\n",
    "            \n",
    "            eta *= 0.8\n",
    "        print('  Selected step size = {:g}'.format(eta))\n",
    "        \n",
    "        # Nesterov gradient ascent...\n",
    "        prev_step = numpy.zeros(alpha.shape[0])\n",
    "        \n",
    "        best = ll(alpha)\n",
    "        best_i = 0\n",
    "        \n",
    "        for i in range(max_iter):\n",
    "            step = 0.8 * prev_step + eta * grad(alpha + 0.8*prev_step)\n",
    "            alpha += step\n",
    "            prev_step = step\n",
    "            \n",
    "            b = ll(alpha)\n",
    "            \n",
    "            if i%50==0:\n",
    "                print('  Iter {} ll = {:g}'.format(i, b))\n",
    "            \n",
    "            if b > best:\n",
    "                best = b\n",
    "                best_i = i\n",
    "            \n",
    "            if best_i+8<i:\n",
    "                break\n",
    "        \n",
    "        print('  Finished, best ll = {:g}'.format(best))\n",
    "        \n",
    "        # Record answer...\n",
    "        self._w = w\n",
    "        self._alpha = alpha\n",
    "\n",
    "\n",
    "    def __call__(self, x):\n",
    "        nf = numpy.sin(numpy.einsum('ef,gf->eg', x, self._w))\n",
    "        ex = numpy.concatenate((numpy.ones((x.shape[0],1)), x, nf), axis=1)\n",
    "        \n",
    "        z = ex.dot(self._alpha)\n",
    "        return 1 / (1 + numpy.exp(-z))\n",
    "\n",
    "\n",
    "\n",
    "def train_tag_model(tag):\n",
    "    start = time.time()\n",
    "    \n",
    "    # **************************************************************** some of the above marks\n",
    "    \n",
    "    # Count number of tokens...\n",
    "    count = sum(len(gmb[i]) for i in range(split))\n",
    "    \n",
    "    # Create matrices...\n",
    "    x = numpy.empty((count, glove.len_vec()))\n",
    "    y = numpy.zeros(count)\n",
    "    \n",
    "    # Loop sentences and word vector them...\n",
    "    offset = 0\n",
    "    for i in range(split):\n",
    "        for t, p in zip(gmb[i], gmb.pos(i)):\n",
    "            if t in glove:\n",
    "                x[offset,:] = glove.decode(t)\n",
    "                if pos_to_rpos[p]==tag:\n",
    "                    y[offset] = 1\n",
    "\n",
    "                offset += 1\n",
    "    \n",
    "    x = x[:offset, :]\n",
    "    y = y[:offset]\n",
    "    \n",
    "    # Train and return model...\n",
    "    ret = LRKS(x, y)\n",
    "    end = time.time()\n",
    "    print('  (took {:g} seconds)'.format(end-start))\n",
    "    return ret\n",
    "\n",
    "\n",
    "\n",
    "# Code to train a model for each reduced POS tag...\n",
    "rpos_model = {}\n",
    "for tag in rpos_desc:\n",
    "    print('Training {}'.format(tag))\n",
    "    rpos_model[tag] = train_tag_model(tag)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.4/dist-packages/ipykernel_launcher.py:80: RuntimeWarning: overflow encountered in exp\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 99%\n",
      "Percentage correct = 86.4%\n",
      "  (took 904.456 seconds)\n"
     ]
    }
   ],
   "source": [
    "# You will want to train the models above, then fill in the below function to estimate POS tags...\n",
    "def token_pos(sentence):\n",
    "    \"\"\"Given a sentence, as a list of tokens, this should return part of\n",
    "    speech tags, as a list of strings (the codes in the rpos_desc dictionary).\n",
    "    Basically calls the models for each tag and selects the tag with the\n",
    "    highest probability.\"\"\"\n",
    "    \n",
    "    # **************************************************************** some of the above marks\n",
    "    \n",
    "    # To word vector...\n",
    "    x = numpy.array(glove.decodes(sentence))\n",
    "    \n",
    "    ret = ['A'] * len(sentence)\n",
    "    ret_prob = [0.0] * len(sentence)\n",
    "    \n",
    "    for tag, model in rpos_model.items():\n",
    "        prob = model(x)\n",
    "        for i in range(len(sentence)):\n",
    "            if prob[i] > ret_prob[i]:\n",
    "                ret[i] = tag\n",
    "                ret_prob[i] = prob[i]\n",
    "    \n",
    "    return ret\n",
    "    #return ['N'] * len(sentence) # Replace me\n",
    "\n",
    "\n",
    "\n",
    "# Code to test the performance of your POS tagger...\n",
    "correct = 0\n",
    "tested = 0\n",
    "pershown = 0\n",
    "stop_percent = 100 # If you want faster feedback you can reduce this\n",
    "\n",
    "start = time.time()\n",
    "for i in range(split, len(gmb)):\n",
    "    percent = int(100 * (i - split) / (len(gmb) - split))\n",
    "    if percent>pershown:\n",
    "        pershown = percent\n",
    "        print('\\r{: 3d}%'.format(percent), end='')\n",
    "    \n",
    "    if percent>=stop_percent:\n",
    "        break\n",
    "    \n",
    "    guess = token_pos(gmb[i])\n",
    "    truth = gmb.pos(i)\n",
    "    \n",
    "    for g,t in zip(guess, truth):\n",
    "        if g==pos_to_rpos[t]:\n",
    "            correct += 1\n",
    "        tested += 1\n",
    "end = time.time()\n",
    "\n",
    "print()\n",
    "print('Percentage correct = {:.1f}%'.format(100 * correct / tested))\n",
    "print('  (took {:g} seconds)'.format(end - start))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Part of speech tagging - sentence level\n",
    "\n",
    "While the previous step works very well you need POS tags to be super accurate, as everything else depends on them. You will now introduce context/ This is done by calculating transition probabilities between tags and solving a Markov random chain using the forward-backwards algorithm (or just forward if you keep links; it's dynamic programming) to find the maximum a posteriori (MAP) POS tag assignment for the entire sentence. The adjacency matrix should contain $\\log P(\\textrm{second pos tag} | \\textrm{first pos tag})$.\n",
    "\n",
    "Advice:\n",
    "* Work in log space, to avoid underflow. Remember to `clip` the model probabilities to avoid `log(0)`.\n",
    "* Include a prior for the adjacency probabilities. Some combinations have a probability of zero, which will end badly when you take the log otherwise.\n",
    "* You need to use the probabilities from the logistic random kitchen sinks models. Remember to normalise them so they sum to 1 before taking the `log`.\n",
    "* The performance boost from this is relatively slight: Only 1.5% for the model answer. But going from 90% to 92% remains a substantial boost (it gets harder the closer you get to 100%! With tweaks, primarily training for longer with more data and random features, this approach (Q1 and Q2) can hit 99%).\n",
    "\n",
    "**(8 marks)**\n",
    "* 2 for adjacency probabilities\n",
    "* 6 marks for algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# **************************************************************** 8 marks\n",
    "\n",
    "# Calculate adjacency probabilities...\n",
    "adj_prob = numpy.ones((len(rpos_to_num), len(rpos_to_num))) # [first, second]\n",
    "\n",
    "for i in range(split):\n",
    "    p = gmb.pos(i)\n",
    "    \n",
    "    for j in range(len(p)-1):\n",
    "        first = rpos_to_num[pos_to_rpos[p[j]]]\n",
    "        second = rpos_to_num[pos_to_rpos[p[j+1]]]\n",
    "        \n",
    "        adj_prob[first, second] += 1\n",
    "\n",
    "adj_prob /= adj_prob.sum(axis=1)[:,None]\n",
    "log_adj_prob = numpy.log(adj_prob)\n",
    "\n",
    "\n",
    "\n",
    "def sentence_pos(sentence):\n",
    "    \"\"\"Given a sentence, as a list of tokens, this should return part of\n",
    "    speech tags, as a list of strings (the codes in the rpos_desc dictionary).\n",
    "    A more advanced version of token_pos that uses neighbours as well.\"\"\"\n",
    "    \n",
    "    # **************************************************************** some of the above marks\n",
    "    \n",
    "    # Declare state...\n",
    "    log_prob = numpy.zeros((len(sentence), len(rpos_to_num))) # Indexed [token, pos tag index]\n",
    "    \n",
    "    # Fill it in with the model outputs...\n",
    "    for i, tok in enumerate(sentence):\n",
    "        wv = glove.decode(tok)\n",
    "        \n",
    "        for j, t in enumerate(num_to_rpos):\n",
    "            model = rpos_model[t]\n",
    "            log_prob[i, j] = model(wv[None,:])[0]\n",
    "        \n",
    "    log_prob /= log_prob.sum(axis=1)[:,None]\n",
    "    log_prob[log_prob<1e-3] = 1e-3\n",
    "    log_prob = numpy.log(log_prob)\n",
    "    \n",
    "    # Forwards pass...\n",
    "    forwards = numpy.zeros(log_prob.shape)\n",
    "    \n",
    "    for i in range(len(sentence)-1):\n",
    "        merge = forwards[i,:,None] + log_prob[i,:,None] + log_adj_prob\n",
    "        forwards[i+1,:] = merge.max(axis=0)\n",
    "    \n",
    "    # Backwards pass...\n",
    "    backwards = numpy.zeros(log_prob.shape)\n",
    "    \n",
    "    for i in range(len(sentence)-1, 0, -1):\n",
    "        merge = backwards[None,i,:] + log_prob[None,i,:] + log_adj_prob\n",
    "        forwards[i-1,:] = merge.max(axis=1)\n",
    "    \n",
    "    # Combine then argmax to select the outputs...\n",
    "    log_prob += forwards\n",
    "    log_prob += backwards\n",
    "\n",
    "    return [num_to_rpos[i] for i in numpy.argmax(log_prob, axis=1)]\n",
    "    #return ['N'] * len(sentence) # Replace me\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.4/dist-packages/ipykernel_launcher.py:80: RuntimeWarning: overflow encountered in exp\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 99%\n",
      "Percentage correct = 89.6%\n",
      "  (took 866.693 seconds)\n"
     ]
    }
   ],
   "source": [
    "# Code to test the performance of your improved POS tagger...\n",
    "correct = 0\n",
    "tested = 0\n",
    "pershown = 0\n",
    "stop_percent = 100 # If you want faster feedback you can reduce this\n",
    "\n",
    "start = time.time()\n",
    "for i in range(split, len(gmb)):\n",
    "    percent = int(100 * (i - split) / (len(gmb) - split))\n",
    "    if percent>pershown:\n",
    "        pershown = percent\n",
    "        print('\\r{: 3d}%'.format(percent), end='')\n",
    "    \n",
    "    if percent>=stop_percent:\n",
    "        break\n",
    "    \n",
    "    guess = sentence_pos(gmb[i])\n",
    "    truth = gmb.pos(i)\n",
    "    \n",
    "    for g,t in zip(guess, truth):\n",
    "        if g==pos_to_rpos[t]:\n",
    "            correct += 1\n",
    "        tested += 1\n",
    "end = time.time()\n",
    "\n",
    "print()\n",
    "print('Percentage correct = {:.1f}%'.format(100 * correct / tested))\n",
    "print('  (took {:g} seconds)'.format(end - start))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Named entity recognition\n",
    "\n",
    "The next step is to identify names, that is the entities that \"facts\" may apply to. While training a further classifier does work (same as above, inc. dynamic programming) there would be little point in repeating the exercise. Instead, a simple rule based approach using *regular expressions* is going to be used.\n",
    "\n",
    "You will probably want to look at the Python 3 documentation:\n",
    "https://docs.python.org/3/library/re.html\n",
    "There is also the how to, a tutorial:\n",
    "https://docs.python.org/3/howto/regex.html\n",
    "\n",
    "\n",
    "Given part of speech tagging a name can be defined as:\n",
    "* An optional *determiner*, e.g. *the* (1 or none)\n",
    "* An arbitrary number of *adjectives* (could be none)\n",
    "* A single *noun*\n",
    "\n",
    "Convert this into a regular expression and finish the function `sentence_ner()` below.\n",
    "\n",
    "Advice:\n",
    "* `''.join(pos)` prepares the list of part of speech tags for applying a regular expression.\n",
    "* Check out the `re.finditer` function as it makes it very easy!\n",
    "* Fun fact: The above ML approach gets 95% accuracy on this task, 97% if you include a feature indicating if the word is uppercased, when this rule gets only 63% accuracy. However, the GMB data set is mostly news headlines, which doesn't translate well to other kinds of text. They also fail for different reasons: ML tries to balance the objectives, while this rule tends to think more things are names than really are. That works better for information extraction (sometimes raw accuracy is not a good indicator of how well a system will work!).\n",
    "\n",
    "**(2 marks)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sentence_ner(sentence, pos):\n",
    "    \"\"\"Given a sentence as a list of tokens and it's part of speech tags\n",
    "    this returns a list of the same length with True wherever it thinks\n",
    "    there is a name.\"\"\"\n",
    "    \n",
    "    ret = [False] * len(sentence)\n",
    "    \n",
    "    # **************************************************************** 2 marks\n",
    "\n",
    "    for match in re.finditer(r'D?J*N', ''.join(pos)):\n",
    "        for i in range(match.start(), match.end()):\n",
    "            ret[i] = True\n",
    "    \n",
    "    return ret\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 99%\n",
      "Percentage correct = 63.4%\n",
      "  (took 0.45142 seconds)\n"
     ]
    }
   ],
   "source": [
    "# Code to test the performance of the NER tagger...\n",
    "correct = 0\n",
    "tested = 0\n",
    "pershown = 0\n",
    "stop_percent = 100 # If you want faster feedback you can reduce this\n",
    "\n",
    "start = time.time()\n",
    "for i in range(split, len(gmb)):\n",
    "    percent = int(100 * (i - split) / (len(gmb) - split))\n",
    "    if percent>pershown:\n",
    "        pershown = percent\n",
    "        print('\\r{: 3d}%'.format(percent), end='')\n",
    "    \n",
    "    if percent>=stop_percent:\n",
    "        break\n",
    "    \n",
    "    guess = sentence_ner(gmb[i], [pos_to_rpos[p] for p in gmb.pos(i)])\n",
    "    truth = [ner!='O' for ner in gmb.ner(i)]\n",
    "    \n",
    "    for g,t in zip(guess, truth):\n",
    "        if g==t:\n",
    "            correct += 1\n",
    "        tested += 1\n",
    "end = time.time()\n",
    "\n",
    "print()\n",
    "print('Percentage correct = {:.1f}%'.format(100 * correct / tested))\n",
    "print('  (took {:g} seconds)'.format(end - start))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Relation extraction\n",
    "\n",
    "This is where the paper \"*Identifying Relations for Open Information Extraction*\" comes in, specifically one of its novel contributions. It extracts relations using this procedure:\n",
    "1. Find relation text by matching a human-designed pattern to the POS tags\n",
    "2. Identify the named entities to the left and right of the relation text.\n",
    "3. Generate the relation tuple (left named entity, relation text, right named entity)\n",
    "\n",
    "(all previous approaches found names then relations - turns out it works much better the other way around)\n",
    "\n",
    "Relation text is identified as:\n",
    "`(Ve (Wo* Pa)?)+`\n",
    "where\n",
    "* `Ve = Verb Particle? Adverb?`\n",
    "* `Wo = Noun | Adjective | Adverb | Pronoun | Determiner`\n",
    "* `Pa = Preposition or subordinating conjunction | Particle`\n",
    "* `| =` or, so either of the options\n",
    "* `? =` optional\n",
    "* `+ =` at least one, but can be many\n",
    "* `* =` an arbitrary number of repetitions, including the option for none.\n",
    "\n",
    "You will need to convert the above rules into a regular expression - this one is harder than the above! You can then run it on a sentence, and for each match identify the named entity to the left and right and create a relation from that.\n",
    "\n",
    "Advice:\n",
    "* Most sentences will only generate one relation, but some will generate more.\n",
    "* Remember that a named entity might be more than one token in length.\n",
    "\n",
    "**(4 marks)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract(sentence):\n",
    "    \"\"\"Given a sentence, as a list of tokens, this returns a list of all relations\n",
    "    extracted from the sentence. Each relation is a tuple with three entries:\n",
    "    (named entity one, relation, named entity two)\"\"\"\n",
    "    pos = sentence_pos(sentence)\n",
    "    ner = sentence_ner(sentence, pos)\n",
    "    ret = []\n",
    "    \n",
    "    # **************************************************************** 4 marks\n",
    "    \n",
    "    pattern = r'(VZ?R?([NJRMD]*[IZ])?)+'\n",
    "    \n",
    "    for match in re.finditer(pattern, ''.join(pos)):\n",
    "        # Find and expand the left name...\n",
    "        left = []\n",
    "        for i in range(match.start()-1, -1, -1):\n",
    "            if ner[i]==True:\n",
    "                left = [sentence[i]] + left\n",
    "                \n",
    "            if ner[i]==False and len(left)>0:\n",
    "                break\n",
    "        if len(left)==0:\n",
    "            continue\n",
    "\n",
    "        # Find and expand the right name...\n",
    "        right = []\n",
    "        for i in range(match.end(), len(pos)):\n",
    "            if ner[i]==True:\n",
    "                right.append(sentence[i])\n",
    "                \n",
    "            if ner[i]==False and len(right)>0:\n",
    "                break\n",
    "        if len(right)==0:\n",
    "            continue\n",
    "            \n",
    "        # Generate the relation...\n",
    "        rel = (' '.join(left), ' '.join(sentence[match.start():match.end()]), ' '.join(right))\n",
    "        ret.append(rel)\n",
    "\n",
    "    return ret\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0mLondon is full of pigeons . \n",
      "\u001b[31mN      V  J    I  N       . \n",
      "\u001b[34mT      F  F    F  T       F \n",
      "\u001b[0m\n",
      "  London -- is full of -- pigeons\n",
      "\n",
      "\u001b[0mIn 1781 William Herschel discovered Uranus \n",
      "\u001b[31mI  N    N       N        V          N      \n",
      "\u001b[34mF  T    T       T        F          T      \n",
      "\u001b[0m\n",
      "  1781 William Herschel -- discovered -- Uranus\n",
      "\n",
      "\u001b[0mTrolls really do not like the sun . \n",
      "\u001b[31mN      V      R  R   I    D   N   . \n",
      "\u001b[34mT      F      F  F   F    T   T   F \n",
      "\u001b[0m\n",
      "  Trolls -- really do not like -- the sun\n",
      "\n",
      "\u001b[0mGiant owls would enjoy eatting people . \n",
      "\u001b[31mJ     N    A     V     N       N      . \n",
      "\u001b[34mT     T    F     F     T       T      F \n",
      "\u001b[0m\n",
      "  Giant owls -- enjoy -- eatting people\n",
      "\n",
      "\u001b[0mDragons collect gold , but they do not make microprocessors . \n",
      "\u001b[31mR       V       N    . C   M    R  R   D    N               . \n",
      "\u001b[34mF       F       T    F F   F    F  F   T    T               F \n",
      "\u001b[0m\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.4/dist-packages/ipykernel_launcher.py:80: RuntimeWarning: overflow encountered in exp\n"
     ]
    }
   ],
   "source": [
    "# Small test of the above...\n",
    "tests = ['London is full of pigeons.',\n",
    "         'In 1781 William Herschel discovered Uranus', # 1\n",
    "         \"Trolls really don't like the sun.\",\n",
    "         'Giant owls would enjoy eatting people.',\n",
    "         \"Dragons collect gold, but they don't make microprocessors.\"] # 2\n",
    "\n",
    "# 1. Seems to miss William - misclassified it, at least with the model answer.\n",
    "# 2. Should extract two facts, first sensible, second absurd.\n",
    "\n",
    "for sentence in tests:\n",
    "    tokens = ogonek.Tokenise(sentence)\n",
    "    \n",
    "    pos = sentence_pos(tokens[0])\n",
    "    ner = ['T' if v else 'F' for v in sentence_ner(tokens[0], pos)]\n",
    "    ogonek.aligned_print(tokens[0], pos, ner)\n",
    "    \n",
    "    rels = extract(tokens[0])\n",
    "    for rel in rels:\n",
    "        print('  ' + ' -- '.join(rel))\n",
    "    print()\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 20000 relations under the seas\n",
    "\n",
    "While the above may have tested each step of the system, the below code runs it on the book \"*20,000 leagues under the seas*\" by Jules Verne (widely considered to be the first science fiction book, and full of fairly dubious claims about what goes on underwater). It's no fun to build a system without running it properly! The above does however miss out several important steps (using ML to filter out bad extractions, simplifying relation text, deleting relations that don't appear often enough in a large corpus etc.) so the below is going to make more mistakes than successes. It's actually kinda funny how terrible it is, but it would unfortunately take too much time to polish.\n",
    "\n",
    "It's also an excuse for a terrible play on words. Sorry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for index in range(len(under_the_seas)):\n",
    "    sentence = under_the_seas[index]\n",
    "    rels = extract(sentence)\n",
    "    \n",
    "    if len(rels)>0:\n",
    "        print(' '.join(sentence))\n",
    "        for rel in rels:\n",
    "            print('  ' + ' -- '.join(rel))\n",
    "        print()\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
