new_data_csv_book.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook creates an output csv file we will import later as our dataframe. The data we downloaded initially is the entired dataset from the college scorecard website. We selected the columns from the csv data file we wanted and filtered it out, to output a small memory csv. The columns filtered out included ones with a lot of null values, and ones not relevant to our college search algorithm.\n",
    "\n",
    "Initially we import and read the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/tj/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py:2723: DtypeWarning: Columns (7,9,1427,1542,1561,1575,1725,1726,1727,1728) have mixed types. Specify dtype option on import or set low_memory=False.\n",
      "  interactivity=interactivity, compiler=compiler, result=result)\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "#importing train dataset\n",
    "train = pd.read_csv('MERGED2013_PP.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are the categories we want to keep in our final csv file. The categories are divided into four groups. The algorithm categories are general columns used for the algorithm, alg_race is columns used for the algorithm regarding race. Alg_majors holds all the major proportion columns. The disp_categories are columns left in the final csv, not used in the algorithm of college finding, but useful information that is displayed in the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Choosing csv data columns to include in the final csv.\n",
    "\"\"\"\n",
    "\n",
    "# general columns used to filter out final college list \n",
    "alg_categories = ['CONTROL', 'ADM_RATE',\n",
    "                  'SATVR25', 'SATVR75', 'SATMT25', 'SATMT75', 'SATWR25', 'SATWR75',\n",
    "                  'NPT4_PUB', 'NPT4_PRIV']\n",
    "\n",
    "# racial proportions and total population columns used to filter out final college list\n",
    "alg_race = ['UGDS','UGDS_WHITE', 'UGDS_BLACK', 'UGDS_HISP', 'UGDS_ASIAN',\n",
    "            'UGDS_AIAN', 'UGDS_NHPI', 'UGDS_2MOR', 'UGDS_NRA', 'UGDS_UNKN',\n",
    "            'UGDS_WHITENH', 'UGDS_API', 'UGDS_API']\n",
    "\n",
    "# major percentage at each school given in each columns\n",
    "# each column is a specific major, mapped in the original dataset dictionary\n",
    "alg_majors = ['PCIP01', 'PCIP03', 'PCIP04', 'PCIP05', 'PCIP09', 'PCIP10', 'PCIP11',\n",
    "              'PCIP12', 'PCIP13', 'PCIP14', 'PCIP15', \n",
    "              'PCIP16', 'PCIP19', 'PCIP22', 'PCIP23', 'PCIP24', 'PCIP25', 'PCIP26', 'PCIP27',\n",
    "              'PCIP29', 'PCIP30', 'PCIP31', 'PCIP38', 'PCIP39', 'PCIP40', 'PCIP41', 'PCIP42',\n",
    "              'PCIP43', 'PCIP44', 'PCIP45', 'PCIP46',\n",
    "              'PCIP47', 'PCIP48', 'PCIP49', 'PCIP50', 'PCIP51', 'PCIP52', 'PCIP54']\n",
    "\n",
    "# columns kept in final csv, but used for just displaying info after college filter runs\n",
    "disp_categories = ['OPEID','INSTNM','CITY','STABBR', 'INSTURL', 'HIGHDEG', 'LATITUDE', 'LONGITUDE']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We add up all the columns and filter out the unused columns from our current datafram to create a new one. The new dataframe is then export it as a csv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "master_category = alg_categories + alg_majors + alg_race + disp_categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = train[master_category]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.to_csv(\"new_college.csv\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}