{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Herschel Stripe 82 master catalogue\n", "## Preparation of UKIRT Infrared Deep Sky Survey / Large Area Survey (UKIDSS/LAS)\n", "\n", "Information about UKIDSS can be found at http://www.ukidss.org/surveys/surveys.html\n", "\n", "The catalogue comes from `dmu0_UKIDSS-LAS`.\n", "\n", "In the catalogue, we keep:\n", "\n", "- The identifier (it's unique in the catalogue);\n", "- The position;\n", "- The stellarity;\n", "- The magnitude for each band in aperture 3 (2 arcsec).\n", "- The hall magnitude is described as the total magnitude.\n", "\n", "J band magnitudes are available in two eopchs. We take the first arbitrarily.\n", "\n", "The magnitudes are “*Vega like*”. The AB offsets are given by Hewett *et al.* (2016):\n", "\n", "| Band | AB offset |\n", "|------|-----------|\n", "| Y | 0.634 |\n", "| J | 0.938 |\n", "| H | 1.379 |\n", "| K | 1.900 |\n", "\n", "Each source is associated with an epoch. These range between 2005 and 2007. We take 2006 for the epoch." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "This notebook was run with herschelhelp_internal version: \n", "33f5ec7 (Wed Dec 6 16:56:17 2017 +0000)\n" ] } ], "source": [ "from herschelhelp_internal import git_version\n", "print(\"This notebook was run with herschelhelp_internal version: \\n{}\".format(git_version()))" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "#%config InlineBackend.figure_format = 'svg'\n", "\n", "import matplotlib.pyplot as plt\n", "plt.rc('figure', figsize=(10, 6))\n", "\n", "from collections import OrderedDict\n", "import os\n", "\n", "from astropy import units as u\n", "from astropy.coordinates import SkyCoord\n", "from astropy.table import Column, Table\n", "import numpy as np\n", "\n", "from herschelhelp_internal.flagging import gaia_flag_column\n", "from herschelhelp_internal.masterlist import nb_astcor_diag_plot, remove_duplicates\n", "from herschelhelp_internal.utils import astrometric_correction, mag_to_flux" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true }, "outputs": [], "source": [ "OUT_DIR = os.environ.get('TMP_DIR', \"./data_tmp\")\n", "try:\n", " os.makedirs(OUT_DIR)\n", "except FileExistsError:\n", " pass\n", "\n", "RA_COL = \"las_ra\"\n", "DEC_COL = \"las_dec\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## I - Column selection" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING: UnitsWarning: 'RADIANS' did not parse as fits unit: At col 0, Unit 'RADIANS' not supported by the FITS standard. [astropy.units.core]\n" ] } ], "source": [ "#Is the following standard (different names for radec vs mag)?\n", "imported_columns = OrderedDict({\n", " 'SOURCEID': 'las_id',\n", " 'RA': 'las_ra',\n", " 'Dec': 'las_dec',\n", " 'YHALLMAG': 'm_ukidss_y',\n", " 'YHALLMAGERR': 'merr_ukidss_y',\n", " 'YAPERMAG3': 'm_ap_ukidss_y',\n", " 'YAPERMAG3ERR': 'merr_ap_ukidss_y',\n", " 'J_1HALLMAG': 'm_ukidss_j',\n", " 'J_1HALLMAGERR': 'merr_ukidss_j',\n", " 'J_1APERMAG3': 'm_ap_ukidss_j',\n", " 'J_1APERMAG3ERR': 'merr_ap_ukidss_j',\n", " 'HAPERMAG3': 'm_ap_ukidss_h',\n", " 'HAPERMAG3ERR': 'merr_ap_ukidss_h',\n", " 'HHALLMAG': 'm_ukidss_h',\n", " 'HHALLMAGERR': 'merr_ukidss_h',\n", " 'KAPERMAG3': 'm_ap_ukidss_k',\n", " 'KAPERMAG3ERR': 'merr_ap_ukidss_k',\n", " 'KHALLMAG': 'm_ukidss_k',\n", " 'KHALLMAGERR': 'merr_ukidss_k',\n", " 'PSTAR': 'las_stellarity'\n", " })\n", "\n", "catalogue = Table.read(\n", " \"../../dmu0/dmu0_UKIDSS-LAS/data/UKIDSS-LAS_Herschel-Stripe-82.fits\")[list(imported_columns)]\n", "for column in imported_columns:\n", " catalogue[column].name = imported_columns[column]\n", "\n", "#Epochs between 2005 and 2007. Rough average:\n", "epoch = 2006\n", "\n", "# Clean table metadata\n", "catalogue.meta = None" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/opt/anaconda3/envs/herschelhelp_internal/lib/python3.6/site-packages/astropy/table/column.py:1096: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.\n", "Check the NumPy 1.11 release notes for more information.\n", " ma.MaskedArray.__setitem__(self, index, value)\n" ] } ], "source": [ "# Adding flux and band-flag columns\n", "for col in catalogue.colnames:\n", " if col.startswith('m_'):\n", " \n", " errcol = \"merr{}\".format(col[1:])\n", " \n", " # LAS uses a huge negative number for missing values\n", " catalogue[col][catalogue[col] < -100] = np.nan\n", " catalogue[errcol][catalogue[errcol] < -100] = np.nan \n", "\n", " # Vega to AB correction\n", " if col.endswith('y'):\n", " catalogue[col] += 0.634\n", " elif col.endswith('j'):\n", " catalogue[col] += 0.938\n", " elif col.endswith('h'):\n", " catalogue[col] += 1.379\n", " elif col.endswith('k'):\n", " catalogue[col] += 1.900\n", " else:\n", " print(\"{} column has wrong band...\".format(col))\n", " \n", " flux, error = mag_to_flux(np.array(catalogue[col]), np.array(catalogue[errcol]))\n", " \n", " # Fluxes are added in µJy\n", " catalogue.add_column(Column(flux * 1.e6, name=\"f{}\".format(col[1:])))\n", " catalogue.add_column(Column(error * 1.e6, name=\"f{}\".format(errcol[1:])))\n", " \n", " # Band-flag column\n", " if \"ap\" not in col:\n", " catalogue.add_column(Column(np.zeros(len(catalogue), dtype=bool), name=\"flag{}\".format(col[1:])))\n", " \n", "# TODO: Set to True the flag columns for fluxes that should not be used for SED fitting." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "<Table masked=True length=10>\n", "
idx | las_id | las_ra | las_dec | m_ukidss_y | merr_ukidss_y | m_ap_ukidss_y | merr_ap_ukidss_y | m_ukidss_j | merr_ukidss_j | m_ap_ukidss_j | merr_ap_ukidss_j | m_ap_ukidss_h | merr_ap_ukidss_h | m_ukidss_h | merr_ukidss_h | m_ap_ukidss_k | merr_ap_ukidss_k | m_ukidss_k | merr_ukidss_k | las_stellarity | f_ukidss_y | ferr_ukidss_y | flag_ukidss_y | f_ap_ukidss_y | ferr_ap_ukidss_y | f_ukidss_j | ferr_ukidss_j | flag_ukidss_j | f_ap_ukidss_j | ferr_ap_ukidss_j | f_ap_ukidss_h | ferr_ap_ukidss_h | f_ukidss_h | ferr_ukidss_h | flag_ukidss_h | f_ap_ukidss_k | ferr_ap_ukidss_k | f_ukidss_k | ferr_ukidss_k | flag_ukidss_k |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 433862346878 | 10.780595913 | -0.000129223693811 | 20.2362 | 0.296362 | 20.3432 | 0.115142 | nan | nan | nan | nan | 20.0619 | 0.147741 | 20.1174 | 0.349096 | 19.8185 | 0.152875 | 19.5448 | 0.283097 | 0.000171409 | 29.2101 | 7.97319 | False | 26.4681 | 2.80695 | nan | nan | False | nan | nan | 34.2946 | 4.66662 | 32.5864 | 10.4775 | False | 42.9128 | 6.04226 | 55.2198 | 14.3981 | False |
1 | 433862346879 | 10.8333696177 | -6.23643295622e-05 | 20.7321 | 0.3192 | 20.5294 | 0.135316 | nan | nan | nan | nan | 20.5888 | 0.239027 | 21.1071 | 0.793732 | nan | nan | nan | nan | 0.00306749 | 18.4988 | 5.43854 | False | 22.2958 | 2.77874 | nan | nan | False | nan | nan | 21.1091 | 4.6472 | 13.0964 | 9.57417 | False | nan | nan | nan | nan | False |
2 | 433862346881 | 10.852724721 | -0.000610532571668 | 19.9908 | 0.18699 | 20.542 | 0.136606 | nan | nan | nan | nan | 20.2763 | 0.179777 | 19.5385 | 0.205607 | nan | nan | nan | nan | 0.00306749 | 36.6159 | 6.30616 | False | 22.0398 | 2.77302 | nan | nan | False | nan | nan | 28.151 | 4.66127 | 55.5399 | 10.5176 | False | nan | nan | nan | nan | False |
3 | 433862346882 | 10.8515730307 | -0.000438797882296 | 19.5506 | 0.138169 | 20.1427 | 0.0959221 | 19.2315 | 0.114457 | 19.7289 | 0.09569 | 19.4244 | 0.082911 | 18.9036 | 0.126451 | 19.0777 | 0.0785753 | 18.7677 | 0.143056 | 9.52581e-06 | 54.926 | 6.9898 | False | 31.8364 | 2.81267 | 73.6885 | 7.76817 | False | 46.6072 | 4.10767 | 61.6908 | 4.71095 | 99.6701 | 11.6081 | False | 84.9016 | 6.14437 | 112.963 | 14.8839 | False |
4 | 433862346885 | 10.8131228064 | -0.00167437926533 | 20.6052 | 0.337929 | 20.7477 | 0.164823 | nan | nan | nan | nan | 19.9673 | 0.135656 | 19.6182 | 0.285299 | 19.7819 | 0.148376 | 19.2456 | 0.251229 | 0.000171409 | 20.793 | 6.4717 | False | 18.2351 | 2.76822 | nan | nan | False | nan | nan | 37.418 | 4.67514 | 51.607 | 13.5608 | False | 44.3851 | 6.06563 | 72.7391 | 16.8311 | False |
5 | 433862346886 | 10.7488126007 | -0.00228587354144 | 20.1795 | 0.214321 | 20.8601 | 0.182412 | 20.8221 | 0.339144 | 20.4306 | 0.181247 | 20.2561 | 0.175744 | 19.8524 | 0.249454 | 20.1742 | 0.210643 | 20.1374 | 0.377733 | 0.000479863 | 30.7738 | 6.07463 | False | 16.4421 | 2.76239 | 17.0279 | 5.31888 | False | 24.4198 | 4.07652 | 28.6781 | 4.64202 | 41.596 | 9.55691 | False | 30.9252 | 5.99976 | 31.9932 | 11.1306 | False |
6 | 433862346887 | 10.7830579735 | -0.00225177569112 | 20.3407 | 0.264272 | 20.7747 | 0.169201 | 19.9259 | 0.198652 | 20.1462 | 0.140169 | 19.8417 | 0.121016 | 19.303 | 0.196935 | 19.4242 | 0.106809 | 18.7035 | 0.167878 | 9.52581e-06 | 26.529 | 6.45725 | False | 17.7883 | 2.77213 | 38.8726 | 7.11234 | False | 31.7344 | 4.09694 | 42.0062 | 4.682 | 68.9904 | 12.5137 | False | 61.7045 | 6.07014 | 119.841 | 18.53 | False |
7 | 433862346888 | 10.7842626274 | -0.00211730354719 | 19.0507 | 0.0953579 | 19.881 | 0.0767292 | 18.8711 | 0.0960997 | 19.3171 | 0.0662912 | 18.7057 | 0.0435461 | 18.3449 | 0.0747749 | 18.332 | 0.0399497 | 17.8692 | 0.0629031 | 9.52581e-06 | 87.0372 | 7.64429 | False | 40.5128 | 2.86305 | 102.702 | 9.09022 | False | 68.1042 | 4.1582 | 119.6 | 4.79686 | 166.746 | 11.4838 | False | 168.735 | 6.20861 | 258.425 | 14.9721 | False |
8 | 433862346890 | 10.8480454023 | -0.00270797269773 | 21.6444 | 0.758328 | 20.8715 | 0.183663 | 20.4897 | 0.249714 | 20.4527 | 0.18471 | 20.1997 | 0.167693 | 20.0458 | 0.353997 | 19.7492 | 0.144575 | 20.2567 | 0.673979 | 9.52581e-06 | 7.98444 | 5.5767 | False | 16.2701 | 2.75225 | 23.126 | 5.31886 | False | 23.9293 | 4.07096 | 30.2083 | 4.6657 | 34.8066 | 11.3485 | False | 45.7434 | 6.09112 | 28.662 | 17.7921 | False |
9 | 433862346893 | 10.7474988819 | -0.00330219454216 | 15.7817 | 0.0044827 | 15.7288 | 0.00410445 | 15.6423 | 0.00408131 | 15.5806 | 0.00345701 | 15.4067 | 0.00325391 | 15.4431 | 0.00432236 | 15.7567 | 0.00480228 | 15.8111 | 0.00624661 | 0.49923 | 1767.37 | 7.29699 | False | 1855.54 | 7.01456 | 2009.53 | 7.55385 | False | 2126.94 | 6.77222 | 2496.34 | 7.48144 | 2414.19 | 9.61099 | False | 1808.55 | 7.99933 | 1720.06 | 9.89608 | False |