{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Exploration\n",
    "\n",
    "The following exercise takes you through using Jupyter to explore some data, and deriving your own algorithm for classifying it. At the end it includes a simple (and largly useless in the real world) machine learning algorithm for you to try out and modify.\n",
    "\n",
    "As an introduction to Jupyter this is a `workbook`, which contains a sequence of `blocks`. This is a `markdown block`, which contains documentation. There are also `code blocks`, which contain code and show the output of running the code below the block. All blocks are editable - you can double click on this to edit it for instance, then hit `run` (icon above with an arrow pointing right into a line) to return it to its current `formatted` state (you don't need to double click on code blocks as they only have one state).\n",
    "\n",
    "Be aware that this workbook requires that two other files exist in the same directory:\n",
    " * Skin_NonSkin.txt\n",
    " * iris_dual.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Marking and Submission\n",
    "\n",
    "These lab exercises are marked, and contribute to your final grade. This lab exercise has 15 marks to earn, equivalent to 15% of your final grade. Every place you have to add code is indicated by\n",
    "\n",
    "`# **************************************************************** 2 marks`\n",
    "\n",
    "with instructions above the code block.\n",
    "\n",
    "Please submit your completed workbook using Moodle before 5pm on Wednesday 17th October 2018. The workbook you submit must be an .ipynb file, which is saved into the directory you're running Jupyter; alternatively you can download it from the menu above using `File -> Download As -> Notebook (.ipynb)`. Remember to save your work regularly (`Save and checkpoint` in the `File` menu, the icon of a floppy disk, or `Ctrl-S`); the version you submit should have all code blocks showing the results (if any) of execution below them. You will normally receive feedback within a week.\n",
    "\n",
    "You must comply with the universities plagiarism guidelines: http://www.bath.ac.uk/library/help/infoguides/plagiarism.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparation\n",
    "\n",
    "The below code block is simply preparation:\n",
    "\n",
    "__%matplotlib inline__: Tells Jupyter to display the graphs generated by matplotlib in the workbook.\n",
    "\n",
    "__import numpy__: Loads 'numerical python', which provides the array objects we store data in.\n",
    "\n",
    "__import matplotlib.pyplot as plt__: Loads an easy to use matplotlib interface, for plotting graphs.\n",
    "\n",
    "Run the below code block by clicking the run icon (an arrow pointing at a line) in the icon line above or typing `ctrl-enter`. The `In[ ]:` next to it will change to `In[1]:` to indicate it was the first code block run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import numpy\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Data\n",
    "\n",
    "Step 1 of any data exploration task is to load the data. \n",
    "\n",
    "For this part of the exercise we will be classifying skin. Specifically, when you take a photograph some pixels are of peoples skin, others are not (e.g. they might be part of a mug of coffee). The machine learning task is to learn a classifier that identifies which pixels are skin and which are not given many exemplars. In a real application a  digital camera may use this as a first step in finding peoples faces, so that is can focus on them or avoid taking photos when people blink.\n",
    "\n",
    "The inputs are: \n",
    " * r - How much red is in the pixel, an integer in [0, 255].\n",
    " * g - How much green is in the pixel, an integer in [0, 255].\n",
    " * b - How much blue is in the pixel, an integer in [0, 255].\n",
    "\n",
    "The output is:\n",
    " * 0 - It is a background pixel.\n",
    " * 1 - It is skin.\n",
    "\n",
    "\n",
    "You need the file `Skin_NonSkin.txt` in the same directory as this workbook for the below code block to work - it should be avaliable on moodle. Alternatively, this data set was obtained from http://archive.ics.uci.edu/ml/datasets/Skin+Segmentation Remember to run it using `ctrl-enter`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of exemplars: 245057\n",
      "Number of background pixels = 194198\n",
      "Number of skin pixels = 50859\n"
     ]
    }
   ],
   "source": [
    "# Open the file for reading...\n",
    "f = open('Skin_NonSkin.txt', 'r')\n",
    "\n",
    "# Each line in the file is an exemplar, unless empty...\n",
    "data = []\n",
    "for line in f.readlines():\n",
    "    parts = [int(v) for v in line.split()]\n",
    "    if len(parts)==4:\n",
    "        data.append(parts)\n",
    "\n",
    "# Close file...\n",
    "f.close()\n",
    "\n",
    "\n",
    "# The data as loaded above is not very conveniant for processing - convert to numpy arrays.\n",
    "# Note that the data in the file uses different conventions to those described above, so\n",
    "# there is some cleanup to make it more conveniant...\n",
    "data = numpy.array(data)\n",
    "\n",
    "in_r = data[:,2]\n",
    "in_g = data[:,1]\n",
    "in_b = data[:,0]\n",
    "\n",
    "out = data[:,3]\n",
    "out[out==2] = 0\n",
    "\n",
    "\n",
    "# Print out how many exemplars were loaded and some very basic statistics...\n",
    "print('Number of exemplars:', out.shape[0])\n",
    "print('Number of background pixels =', (out==0).sum())\n",
    "print('Number of skin pixels =', (out==1).sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train / Test Split\n",
    "\n",
    "For reasons which will be covered in a later lecture you must always have _independent_ test and train data sets - in this case we will have one data set for training, and one for testing. The below code performs this split, putting half of each kind into each set.\n",
    "\n",
    "(Sometimes data sets are provided with a train/test split already built in, which can be good for making sure everyone has comparable results. The 50:50 split is arbitrary - many other choices can be made, as can different choices regarding numbers of training and testing data sets.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This code generates the indices of exemplars for each kind...\n",
    "index_bg = numpy.nonzero(out==0)[0] # bg is short for background.\n",
    "index_skin = numpy.nonzero(out==1)[0]\n",
    "\n",
    "\n",
    "# Split to get indices for each set...\n",
    "half_bg = index_bg.shape[0] // 2\n",
    "half_skin = index_skin.shape[0] // 2\n",
    "\n",
    "index_train = numpy.concatenate((index_bg[:half_bg],index_skin[:half_skin]))\n",
    "index_test = numpy.concatenate((index_bg[half_bg:],index_skin[half_skin:]))\n",
    "\n",
    "\n",
    "# Now use the indices to generate the vectors of data for each case...\n",
    "train_in_r = in_r[index_train]\n",
    "train_in_g = in_g[index_train]\n",
    "train_in_b = in_b[index_train]\n",
    "train_out = out[index_train]\n",
    "\n",
    "test_in_r = in_r[index_test]\n",
    "test_in_g = in_g[index_test]\n",
    "test_in_b = in_b[index_test]\n",
    "test_out = out[index_test]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Statistics\n",
    "\n",
    "Given a new data set the first thing you may want to look at is its statistics, such as mean and standard deviation (for the training set only). The below code prints out the global statistics, your task is to also print out the statistics for background and skin independently. You may get just the members of a vector that correspond to the background by, e.g. `train_in_r[train_out==0]`.\n",
    "\n",
    "__(0.5 marks)__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global:\n",
      "  Red: mean = 114.88769913815618 | sd = 75.18217758110548\n",
      "  Green: mean = 124.14228584486811 | sd = 64.75743226985364\n",
      "  Blue: mean = 119.23729270044397 | sd = 66.73829385809498\n",
      "\n",
      "Background:\n",
      "  Red: mean = 94.66869895673487 | sd = 67.69313702726875\n",
      "  Green: mean = 120.30854076767011 | sd = 68.59341818552605\n",
      "  Blue: mean = 122.28493599316162 | sd = 70.65863369859589\n",
      "\n",
      "Skin:\n",
      "  Red: mean = 192.0926501238743 | sd = 47.07941024001938\n",
      "  Green: mean = 138.78119469896575 | sd = 44.38303251625984\n",
      "  Blue: mean = 107.60006292028785 | sd = 47.18406864719033\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('Global:')\n",
    "print('  Red: mean =', numpy.mean(train_in_r), '| sd =', numpy.std(train_in_r))\n",
    "print('  Green: mean =', numpy.mean(train_in_g), '| sd =', numpy.std(train_in_g))\n",
    "print('  Blue: mean =', numpy.mean(train_in_b), '| sd =', numpy.std(train_in_b))\n",
    "print('')\n",
    "\n",
    "# **************************************************************** 0.5 marks\n",
    "\n",
    "print('Background:')\n",
    "print('  Red: mean =', numpy.mean(train_in_r[train_out==0]), '| sd =', numpy.std(train_in_r[train_out==0]))\n",
    "print('  Green: mean =', numpy.mean(train_in_g[train_out==0]), '| sd =', numpy.std(train_in_g[train_out==0]))\n",
    "print('  Blue: mean =', numpy.mean(train_in_b[train_out==0]), '| sd =', numpy.std(train_in_b[train_out==0]))\n",
    "print('')\n",
    "\n",
    "print('Skin:')\n",
    "print('  Red: mean =', numpy.mean(train_in_r[train_out==1]), '| sd =', numpy.std(train_in_r[train_out==1]))\n",
    "print('  Green: mean =', numpy.mean(train_in_g[train_out==1]), '| sd =', numpy.std(train_in_g[train_out==1]))\n",
    "print('  Blue: mean =', numpy.mean(train_in_b[train_out==1]), '| sd =', numpy.std(train_in_b[train_out==1]))\n",
    "print('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Simplest Classifier\n",
    "\n",
    "Probably the simplest classifier is a single if statement based on a test, e.g.\n",
    "```\n",
    "if feature < 12:\n",
    "    return 1\n",
    "else:\n",
    "    return 0\n",
    "```\n",
    "\n",
    "For continuous features such as colour `less than` or `greater than` tests are a good choice. Fill in the below function with a test based on your intuition from looking at the basic statistics you calculated above (expect to obtain around 70-85% correct).\n",
    "\n",
    "__(1 mark)__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Got 81.88673701735915 % correct\n"
     ]
    }
   ],
   "source": [
    "# Write a classification function...\n",
    "def is_skin1(r, g, b):\n",
    "    # **************************************************************** 1 mark\n",
    "    if r < 140:\n",
    "        return 0\n",
    "    else:\n",
    "        return 1\n",
    "\n",
    "# Code to test classificatuion performance on the test set...\n",
    "estimate = numpy.array([is_skin1(test_in_r[i], test_in_g[i], test_in_b[i]) for i in range(test_in_r.shape[0])])\n",
    "\n",
    "correct = (estimate==test_out).sum()\n",
    "percentage = 100.0 * correct / float(test_in_r.shape[0])\n",
    "\n",
    "print('Got', percentage, '% correct')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualisation for a Better Guess\n",
    "\n",
    "Looking at just the mean and standard deviation is no way to make a guess, even though it does surprisingly well in this case. A better choice is to look at colour coded scatter plots of the data. As you can only look at two dimensions at a time (your screen is flat) and we have three features a good approach is to plot all combinations of two variables, e.g. red vs green, red vs blue and green vs blue. You may want to set the marker parameter of scatter(...) to improve visibility, for instance marker='+' gets you small crosses, marker='.' gets dots.\n",
    "\n",
    "Finish the below code, where red vs green has already been provided.\n",
    "\n",
    "__(0.5 marks)__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Red vs Green:\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 66728 characters]\n",
      "text/plain": [
       "<Figure size 864x864 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Red vs Blue:\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 128708 characters]\n",
      "text/plain": [
       "<Figure size 864x864 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Green vs Blue:\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 119268 characters]\n",
      "text/plain": [
       "<Figure size 864x864 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print('Red vs Green:')\n",
    "plt.figure(figsize=(12,12))\n",
    "plt.scatter(train_in_r[train_out==0], train_in_g[train_out==0], c='b') # c='b' means blue, for background.\n",
    "plt.scatter(train_in_r[train_out==1], train_in_g[train_out==1], c='r') # c='r' means red, for, errr, skin.\n",
    "plt.show()\n",
    "\n",
    "# **************************************************************** 0.5 marks\n",
    "print('Red vs Blue:')\n",
    "plt.figure(figsize=(12,12))\n",
    "plt.scatter(train_in_r[train_out==0], train_in_b[train_out==0], c='b', marker='+')\n",
    "plt.scatter(train_in_r[train_out==1], train_in_b[train_out==1], c='r', marker='.')\n",
    "plt.show()\n",
    "\n",
    "print('Green vs Blue:')\n",
    "plt.figure(figsize=(12,12))\n",
    "plt.scatter(train_in_g[train_out==0], train_in_b[train_out==0], c='b', marker='+')\n",
    "plt.scatter(train_in_g[train_out==1], train_in_b[train_out==1], c='r', marker='.')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Better Classifier\n",
    "\n",
    "Instead of applying the split (`if` statement) to a feature directly we can calculate some function of several features. For instance, we could sum two features\n",
    "\n",
    "```\n",
    "if g+b > 42:\n",
    "    return 1\n",
    "else:\n",
    "    return 0\n",
    "```\n",
    "\n",
    "Additionally, instead of a simple split we could consider distance from an ideal value,\n",
    "\n",
    "```\n",
    "if numpy.fabs((g+b) - 22) < 8:\n",
    "    return 1\n",
    "else:\n",
    "    return 0\n",
    "```\n",
    "\n",
    "In the above `22` is the ideal value and `8` the threshold, so the above considers any pixel where `g+b` is between `22-8` and `22+8` to be skin. Note that `22` and `8` can be derived from the mean and variance of the skin respectively (for some multipliar of the variance). A good choice of function and ideal value/threshold should seperate the two classes.\n",
    "\n",
    "Below you need to write code that combines these two ideas:\n",
    " 1. Look at the graphs and select a function of the original features that will improve seperation.\n",
    " 2. Print out simple statistcs for the new function.\n",
    " 3. Write a function based on the simple statistcs (expect to get 80-95% correct).\n",
    "\n",
    "__(3 marks)__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Background: mean = -25.63984181093523 | sd = 47.994891744319595\n",
      "Skin: mean = 53.311455424908566 | sd = 8.342602224580162\n",
      "\n",
      "Got 93.67904740918476 % correct\n"
     ]
    }
   ],
   "source": [
    "# Write a sensible function of the features and include an explanation of why you choose it (as a comment)...\n",
    "\n",
    "# **************************************************************** 2 marks\n",
    "train_func = train_in_r - train_in_g\n",
    "# In the red vs green graph the skin tones form a thin line at a\n",
    "# 45 degree angle; the above function is rotated 90 degrees to it and\n",
    "# therefore projects all skin tones to a range seperate from the background.\n",
    "\n",
    "\n",
    "# Print out simple statistics for your function of the data...\n",
    "\n",
    "# **************************************************************** 0 marks\n",
    "print('Background: mean =', numpy.mean(train_func[train_out==0]), '| sd =', numpy.std(train_func[train_out==0]))\n",
    "print('Skin: mean =', numpy.mean(train_func[train_out==1]), '| sd =', numpy.std(train_func[train_out==1]))\n",
    "print('')\n",
    "\n",
    "\n",
    "# Write a new and improved classifier...\n",
    "def is_skin2(r, g, b):\n",
    "    # **************************************************************** 1 marks\n",
    "    if numpy.fabs((r - g) - 53.3) < 16.0:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n",
    "\n",
    "\n",
    "# Code to test classification performance on the test set...\n",
    "estimate = numpy.array([is_skin2(test_in_r[i], test_in_g[i], test_in_b[i]) for i in range(test_in_r.shape[0])])\n",
    "\n",
    "correct = (estimate==test_out).sum()\n",
    "percentage = 100.0 * correct / float(test_in_r.shape[0])\n",
    "\n",
    "print('Got', percentage, '% correct')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning\n",
    "\n",
    "You are yet to do any actually machine learning - instead you have been manually proposing and programming classifiers, then measuring their performance. Instead, we can get the computer to try lots of different classifiers and select the best one. This is where the train/test split becomes important. We choose the classifier that gets the best score on the train set, but then verify it's performance on the test set.\n",
    "\n",
    "To do this you need to perform four steps:\n",
    " 1. Code a function that classifies your data based on some parameters.\n",
    " 2. Code a function that scores your test data based on the same parameters.\n",
    " 3. Generate lots of different parameters, and select the set that have the best performance.\n",
    " 4. Score the parameters on the test set.\n",
    "\n",
    "This is what you need to code below.\n",
    "\n",
    "\n",
    "Unlike the above where you coded the function directly, now you need to code a function of some parameters and then find the best parameters. A very common function is a linear sum of the features, specifically `r*sr + g*sg + b*sb` where `sr`, `sg` and `sb` are the parameters to be learned. We can then consider distance from an ideal value, introducing two further parameters, `ideal` for the ideal value and `threshold` to decide how close to the ideal value you have to be to be considered as skin.\n",
    "\n",
    "Be careful how many parameter combinations you try - no effort has been made to be efficient, so it could take a long time to run! You will probably want to print out some kind of progress report. You should be able to obtain over 98%.\n",
    "\n",
    "\n",
    "_Note 1_: Remember the relationship above, you don't want to loop over values for the ideal value, and will want to look over multiples of the standard deviation for the threshold.\n",
    "\n",
    "_Note 2_: You may find the function numpy.linspace(...) useful for generating a selection of parameters to try.\n",
    "\n",
    "_Note 3_: This approach of simply trying every combination is called `brute force` - it works on small problems like this, but does not scale problems of even moderate size. This is not to say it is never used - there are plenty of small problems in the world for which it is the right approach, but most computer scientists would be embarrassed to admit they used it.\n",
    "\n",
    "_Note 4_: You will probably notice that your performance on the training set is better than the test set. This is called overfitting, and discussed in a later lecture.\n",
    "\n",
    "__(4 marks)__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trying sr = 0.0\n",
      "Improved to 34.63506598437921 %\n",
      "Improved to 36.983897689526565 %\n",
      "Improved to 39.1311444637596 %\n",
      "Improved to 48.09392062287295 %\n",
      "Improved to 79.24572958238458 %\n",
      "Trying sr = 0.5\n",
      "Improved to 88.58719160362037 %\n",
      "Improved to 99.02064001175232 %\n",
      "Improved to 99.07695321107656 %\n",
      "Trying sr = 1.0\n",
      "Improved to 99.17896987651903 %\n",
      "Final classifier:\n",
      "  sr = 1.0\n",
      "  sg = -0.5\n",
      "  sb = -0.5\n",
      "  ideal = 68.9020213142475\n",
      "  threshold = 28.15283908726569\n",
      "\n",
      "  Got 94.08140113768985 % correct\n"
     ]
    }
   ],
   "source": [
    "# Code a function to classify an input colour (r,g b) which is driven by the 5 parameters above...\n",
    "def is_skin3(r, g, b, sr, sg, sb, ideal, threshold):\n",
    "    # **************************************************************** 1 marks\n",
    "    if numpy.fabs(r*sr + g*sg + b*sb - ideal) < threshold:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n",
    "\n",
    "\n",
    "# Fill in this function that scores the test data and return the percentage of right values for the train data...\n",
    "def score(sr, sg, sb, ideal, threshold):\n",
    "    # **************************************************************** 1 mark\n",
    "    #estimate = numpy.array([is_skin3(train_in_r[i], train_in_g[i], train_in_b[i], sr, sg, sb, ideal, threshold) for i in range(train_in_r.shape[0])])\n",
    "    estimate = (numpy.fabs(train_in_r*sr + train_in_g*sg + train_in_b*sb - ideal) < threshold).astype(int)\n",
    "    # First version as suggested, second is faster!\n",
    "    correct = (estimate==train_out).sum()\n",
    "    return 100.0 * correct / float(test_in_r.shape[0])\n",
    "\n",
    "\n",
    "# Write a (nested) loop that tries out lots of different parameters.\n",
    "# A sugestion for the outer level of the loop has been provided for you.\n",
    "best_score = -1.0\n",
    "best_sr = None\n",
    "best_sg = None\n",
    "best_sb = None\n",
    "best_ideal = None\n",
    "best_threshold = None\n",
    "\n",
    "for sr in [0.0, 0.5, 1.0]: # Half of range to save computation\n",
    "    print('Trying sr =', sr)\n",
    "    # **************************************************************** 2 marks\n",
    "    for sg in [-1.0, -0.5, 0.0, 0.5, 1.0]:\n",
    "        for sb in [-1.0, -0.5, 0.0, 0.5, 1.0]:\n",
    "            func = sr * train_in_r[train_out==1] + sg * train_in_g[train_out==1] + sb * train_in_b[train_out==1]\n",
    "            ideal = numpy.mean(func)\n",
    "            sd = numpy.std(func)\n",
    "            \n",
    "            for sd_mult in [2.0, 3.0]:\n",
    "                threshold = sd_mult * sd\n",
    "                \n",
    "                s = score(sr, sg, sb, ideal, threshold)\n",
    "                \n",
    "                if s > best_score:\n",
    "                    best_score = s\n",
    "                    best_sr = sr\n",
    "                    best_sg = sg\n",
    "                    best_sb = sb\n",
    "                    best_ideal = ideal\n",
    "                    best_threshold = threshold\n",
    "                    \n",
    "                    print('Improved to', s, '%')\n",
    "\n",
    "\n",
    "# This code will judge the classifier your computer has learned...\n",
    "estimate = numpy.array([is_skin3(test_in_r[i], test_in_g[i], test_in_b[i], best_sr, best_sg, best_sb, best_ideal, best_threshold) for i in range(test_in_r.shape[0])])\n",
    "\n",
    "correct = (estimate==test_out).sum()\n",
    "percentage = 100.0 * correct / float(test_in_r.shape[0])\n",
    "\n",
    "print('Final classifier:')\n",
    "print('  sr =', best_sr)\n",
    "print('  sg =', best_sg)\n",
    "print('  sb =', best_sb)\n",
    "print('  ideal =', best_ideal)\n",
    "print('  threshold =', best_threshold)\n",
    "print('')\n",
    "print('  Got', percentage, '% correct')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Another Data Set\n",
    "\n",
    "Included with this worksheet is the file `synthetic.txt`.  Each row of the file contains 3 numbers, seperated by spaces. The first 2 are the input feature vector, the final value is the class. Your task is to repeat the above for this new data set.\n",
    "\n",
    "__(6 marks)__\n",
    " * 1 mark for loading the data set.\n",
    " * 1 mark for getting it ready.\n",
    " * 1 mark for exploration (printing basic statistics and/or graphs).\n",
    " * 2 marks for a classification algorithm. Your choice how you generate it.\n",
    " * 1 mark for getting above 96% on your test set.\n",
    "\n",
    "We reserve the right to remove marks if your solution is more complicated than it needs to be!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class A:\n",
      "  F0: mean = 0.7361964285714286 | sd = 0.16093010013699127\n",
      "  F1: mean = 0.5960267857142857 | sd = 0.29055908115161083\n",
      "\n",
      "Class B:\n",
      "  F0: mean = 0.2709375 | sd = 0.17545097521775452\n",
      "  F1: mean = 0.4397291666666667 | sd = 0.2764792090523164\n",
      "\n",
      "Function: 0.901*f0 + 0.434*f1 < 0.678\n",
      "Train accuracy: 95.3125 % correct\n",
      "Test accuracy: 97.265625 % correct\n"
     ]
    }
   ],
   "source": [
    "# **************************************************************** 6 marks\n",
    "\n",
    "# Read in data...\n",
    "data = []\n",
    "out = []\n",
    "\n",
    "with open('synthetic.txt', 'r') as fin:\n",
    "    for line in fin.readlines():\n",
    "        parts = line.split()\n",
    "        if len(parts)==3:\n",
    "            data.append([float(parts[0]), float(parts[1])])\n",
    "            out.append(int(parts[2]))\n",
    "\n",
    "data = numpy.array(data)\n",
    "out = numpy.array(out)\n",
    "\n",
    "\n",
    "\n",
    "# This code generates the indices of exemplars for each kind...\n",
    "index_a = numpy.nonzero(out==0)[0]\n",
    "index_b = numpy.nonzero(out==1)[0]\n",
    "\n",
    "\n",
    "\n",
    "# Split to get indices for each set...\n",
    "half_a = index_a.shape[0] // 2\n",
    "half_b = index_b.shape[0] // 2\n",
    "\n",
    "index_train = numpy.concatenate((index_a[:half_a], index_b[:half_b]))\n",
    "index_test = numpy.concatenate((index_a[half_a:],index_b[half_b:]))\n",
    "\n",
    "\n",
    "\n",
    "# Now use the indices to generate the vectors of data for each case...\n",
    "train_data = data[index_train,:]\n",
    "train_out = out[index_train]\n",
    "\n",
    "test_data = data[index_test,:]\n",
    "test_out = out[index_test]\n",
    "\n",
    "\n",
    "\n",
    "# Statistics - just because...\n",
    "print('Class A:')\n",
    "print('  F0: mean =', numpy.mean(train_data[train_out==0,0]), '| sd =', numpy.std(train_data[train_out==0,0]))\n",
    "print('  F1: mean =', numpy.mean(train_data[train_out==0,1]), '| sd =', numpy.std(train_data[train_out==0,1]))\n",
    "print('')\n",
    "\n",
    "print('Class B:')\n",
    "print('  F0: mean =', numpy.mean(train_data[train_out==1,0]), '| sd =', numpy.std(train_data[train_out==1,0]))\n",
    "print('  F1: mean =', numpy.mean(train_data[train_out==1,1]), '| sd =', numpy.std(train_data[train_out==1,1]))\n",
    "print('')\n",
    "\n",
    "\n",
    "\n",
    "# Try various classification functions - being slightly smart...\n",
    "best_angle = -1.0\n",
    "best_split = 0.0\n",
    "best_accuracy = 0.0\n",
    "for angle in numpy.linspace(0.0, numpy.pi, 8):\n",
    "    proj = (train_data * [[numpy.cos(angle), numpy.sin(angle)]]).sum(axis=1)\n",
    "    split = 0.5 * (proj[train_out==0].mean() + proj[train_out==1].mean())\n",
    "    answer = (proj<split).astype(int)\n",
    "    \n",
    "    accuracy = (answer==train_out).sum() / answer.shape[0]\n",
    "    accuracy = max(accuracy, 1.0 - accuracy)\n",
    "    \n",
    "    if accuracy > best_accuracy:\n",
    "        best_angle = angle\n",
    "        best_split = split\n",
    "        best_accuracy = accuracy\n",
    "\n",
    "\n",
    "\n",
    "# Print out function and how good it is...\n",
    "print('Function: {:.3f}*f0 + {:.3f}*f1 < {:.3f}'.format(numpy.cos(best_angle), numpy.sin(best_angle), best_split))\n",
    "print('Train accuracy:', 100*best_accuracy, '% correct')\n",
    "\n",
    "proj = (test_data * [[numpy.cos(best_angle), numpy.sin(best_angle)]]).sum(axis=1)\n",
    "answer = (proj<best_split).astype(int)\n",
    "test_accuracy = (answer==test_out).sum() / answer.shape[0]\n",
    "test_accuracy = max(test_accuracy, 1.0 - test_accuracy)\n",
    "\n",
    "print('Test accuracy:', 100*test_accuracy, '% correct')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
