{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook prepare the catalogues that will be analysed by CIGALE for SED fitting and physical parameter estimation." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import numpy as np\n", "import os\n", "os.environ['LOG_LEVEL'] = 'INFO'\n", "\n", "from astropy.table import Table\n", "\n", "from herschelhelp.filters import correct_galactic_extinction\n", "from herschelhelp.external import convert_table_for_cigale" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [], "source": [ "SUFFIX = '20171016'\n", "master_catalogue = Table.read(\"../../dmu32/dmu32_ELAIS-N1/data/ELAIS-N1_{}_cigale.fits\".format(SUFFIX))" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "4026292" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(master_catalogue)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Best sources\n", "\n", "Define a good far-IR measurement as:\n", "- an existing flux in the band;\n", "- the flag from XID+ must not be set;\n", "- the signal to noise ratio must be over 2." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": true }, "outputs": [], "source": [ "good = {}\n", "for band in ['pacs_green', 'pacs_red', 'spire_250', 'spire_350', 'spire_500']:\n", " good[band] = (~np.isnan(master_catalogue['f_{}'.format(band)]) & \n", " ~master_catalogue['flag_{}'.format(band)])\n", " good[band][good[band]] &= (master_catalogue[good[band]]['f_{}'.format(band)] /\n", " master_catalogue[good[band]]['ferr_{}'.format(band)] >= 2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will keep only sources with at leat 2 good far-IR measurements (we may actually use less sources are not all may have a redshift)." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": true }, "outputs": [], "source": [ "combined_good = np.sum(list(good.values()), axis=0) >= 2" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of good sources: 51153\n" ] } ], "source": [ "print(\"Number of good sources: {}\".format(np.sum(combined_good)))" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Only sources with at least two optical and at least two near infrared detections\n", "optnir = ((master_catalogue['flag_optnir_det'] == 3) \n", " | (master_catalogue['flag_optnir_det'] == 7))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Main catalogue for CIGALE" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": true }, "outputs": [], "source": [ "best_catalogue = master_catalogue[combined_good].copy()" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Correction for galactic extinction\n", "best_catalogue = correct_galactic_extinction(best_catalogue, inplace=True)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "scrolled": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:herschelhelp.external:For 2572 sources, the band wfc_u should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 1745 sources, the band megacam_u should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 924 sources, the band suprime_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 909 sources, the band megacam_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 929 sources, the band gpc1_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 878 sources, the band wfc_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 490 sources, the band suprime_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 446 sources, the band gpc1_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 340 sources, the band wfc_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 357 sources, the band megacam_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 108 sources, the band gpc1_i should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 106 sources, the band suprime_i should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 109 sources, the band wfc_i should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n" ] } ], "source": [ "# Convertion to CIGALE format\n", "best_catalogue = convert_table_for_cigale(best_catalogue, inplace=True, remove_zerofluxes=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Band selection\n", "\n", "We want to use only one filter for similar bands. We define an order of preference and set to NaN the flux in the lower prefered bands when a prefered band is available. Some band may have a 0 flux, we set there values to NaN." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": true }, "outputs": [], "source": [ "g_bands = ['suprime_g', 'megacam_g', 'wfc_g', 'gpc1_g']\n", "u_bands = ['megacam_u', 'wfc_u']\n", "r_bands = ['suprime_r', 'megacam_r', 'wfc_r', 'gpc1_r']\n", "i_bands = ['suprime_i', 'wfc_i']\n", "z_bands = ['suprime_z', 'megacam_z', 'wfc_z']\n", "y_bands = ['suprime_y', 'gpc1_y']\n", "\n", "def remove_unneeded_fluxes(list_of_bands):\n", " for band_idx, band in enumerate(list_of_bands[:-1]):\n", " mask = ~np.isnan(best_catalogue[band])\n", " for lower_band in list_of_bands[band_idx+1:]:\n", " best_catalogue[lower_band][mask] = np.nan\n", " best_catalogue[\"{}_err\".format(lower_band)][mask] = np.nan" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": true }, "outputs": [], "source": [ "remove_unneeded_fluxes(g_bands)\n", "remove_unneeded_fluxes(u_bands)\n", "remove_unneeded_fluxes(r_bands)\n", "remove_unneeded_fluxes(i_bands)\n", "remove_unneeded_fluxes(z_bands)\n", "remove_unneeded_fluxes(y_bands)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": true }, "outputs": [], "source": [ "best_catalogue.write(\"data_tmp/ELAIS-N1_cigale_best_extcor_{}.fits\".format(SUFFIX), overwrite=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# All optnir objects with photoz" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": true }, "outputs": [], "source": [ "best_catalogue = master_catalogue[optnir].copy()" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": true }, "outputs": [], "source": [ "best_catalogue = best_catalogue[~np.isnan(best_catalogue[\"redshift\"])]" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of optnir sources with z-phot: 457324\n" ] } ], "source": [ "print(\"Number of optnir sources with z-phot: {}\".format(len(best_catalogue)))" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Correction for galactic extinction\n", "best_catalogue = correct_galactic_extinction(best_catalogue, inplace=True)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:herschelhelp.external:For 60786 sources, the band wfc_u should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 58081 sources, the band megacam_u should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 43238 sources, the band suprime_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 43092 sources, the band megacam_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 43310 sources, the band gpc1_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 42694 sources, the band wfc_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 24486 sources, the band suprime_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 22447 sources, the band gpc1_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 17763 sources, the band wfc_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 18618 sources, the band megacam_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 2363 sources, the band gpc1_i should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 2354 sources, the band suprime_i should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 2375 sources, the band wfc_i should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n" ] } ], "source": [ "# Convertion to CIGALE format\n", "os.environ['LOG_LEVEL'] = 'INFO'\n", "best_catalogue = convert_table_for_cigale(best_catalogue, inplace=True, remove_zerofluxes=True)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": true }, "outputs": [], "source": [ "remove_unneeded_fluxes(g_bands)\n", "remove_unneeded_fluxes(u_bands)\n", "remove_unneeded_fluxes(r_bands)\n", "remove_unneeded_fluxes(i_bands)\n", "remove_unneeded_fluxes(z_bands)\n", "remove_unneeded_fluxes(y_bands)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": true }, "outputs": [], "source": [ "best_catalogue.write(\"data_tmp/ELAIS-N1_cigale_optnir_extcor_{}.fits\".format(SUFFIX), overwrite=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## All XID+ objects to understand selection " ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "xid = {}\n", "for band in ['pacs_green', 'pacs_red', 'spire_250', 'spire_350', 'spire_500']:\n", " xid[band] = (~np.isnan(master_catalogue['f_{}'.format(band)]) & \n", " ~master_catalogue['flag_{}'.format(band)])\n", "xid = np.sum(list(xid.values()), axis=0) >= 1" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "best_catalogue = master_catalogue[xid].copy()" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "best_catalogue = best_catalogue[~np.isnan(best_catalogue[\"redshift\"])]" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of optnir sources with z-phot: 203612\n" ] } ], "source": [ "print(\"Number of optnir sources with z-phot: {}\".format(len(best_catalogue)))" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "# Correction for galactic extinction\n", "best_catalogue = correct_galactic_extinction(best_catalogue, inplace=True)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "for col in best_catalogue.colnames:\n", " if col.startswith('f_'):\n", " err_col = 'ferr' + col[1:]\n", " if col not in ['f_pacs_green', 'f_pacs_red', 'f_spire_250', 'f_spire_350', 'f_spire_500']:\n", " best_catalogue[col] = np.nan\n", " best_catalogue[col] = np.nan" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:herschelhelp.external:For 11158 sources, the band wfc_u should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 8315 sources, the band megacam_u should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 4985 sources, the band suprime_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 4942 sources, the band megacam_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 5006 sources, the band gpc1_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 4809 sources, the band wfc_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 3199 sources, the band suprime_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 2894 sources, the band gpc1_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 1986 sources, the band wfc_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 2142 sources, the band megacam_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 537 sources, the band gpc1_i should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 533 sources, the band suprime_i should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 542 sources, the band wfc_i should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n" ] } ], "source": [ "# Convertion to CIGALE format\n", "os.environ['LOG_LEVEL'] = 'INFO'\n", "best_catalogue = convert_table_for_cigale(best_catalogue, inplace=True, remove_zerofluxes=True)" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "best_catalogue.write(\"data_tmp/ELAIS-N1_cigale_allxid_extcor_{}.fits\".format(SUFFIX), overwrite=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Catalogue using spectroscopic redshift" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": true }, "outputs": [], "source": [ "best_catalogue = master_catalogue[optnir].copy()" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": true }, "outputs": [], "source": [ "best_catalogue.remove_column(\"redshift\")\n", "best_catalogue[\"zspec\"].name = \"redshift\"" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "collapsed": true }, "outputs": [], "source": [ "best_catalogue = best_catalogue[~np.isnan(best_catalogue[\"redshift\"])]" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of sources with z-spec: 3787\n" ] } ], "source": [ "print(\"Number of sources with z-spec: {}\".format(len(best_catalogue)))" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Correction for galactic extinction\n", "best_catalogue = correct_galactic_extinction(best_catalogue, inplace=True)" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "INFO:herschelhelp.external:For 168 sources, the band wfc_u should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 116 sources, the band megacam_u should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 67 sources, the band suprime_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 66 sources, the band megacam_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 68 sources, the band gpc1_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 65 sources, the band wfc_g should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 53 sources, the band suprime_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 53 sources, the band gpc1_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 53 sources, the band wfc_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 53 sources, the band megacam_r should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 4 sources, the band gpc1_i should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 4 sources, the band suprime_i should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n", "INFO:herschelhelp.external:For 5 sources, the band wfc_i should not be used because it overlaps or is below the Lyman limit at the redshift of these sources. These fluxes were set to NaN.\n" ] } ], "source": [ "# Convertion to CIGALE format\n", "os.environ['LOG_LEVEL'] = 'INFO'\n", "best_catalogue = convert_table_for_cigale(best_catalogue, inplace=True, remove_zerofluxes=True)" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "collapsed": true }, "outputs": [], "source": [ "remove_unneeded_fluxes(g_bands)\n", "remove_unneeded_fluxes(u_bands)\n", "remove_unneeded_fluxes(r_bands)\n", "remove_unneeded_fluxes(i_bands)\n", "remove_unneeded_fluxes(z_bands)\n", "remove_unneeded_fluxes(y_bands)" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "collapsed": true }, "outputs": [], "source": [ "best_catalogue.write(\"data_tmp/ELAIS-N1_cigale_optnir_extcor_zspec_{}.fits\".format(SUFFIX), overwrite=True)" ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python (herschelhelp_internal)", "language": "python", "name": "helpint" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.2" } }, "nbformat": 4, "nbformat_minor": 2 }