XMM-LSS master catalogue¶
Preparation of Hyper Suprime-Cam Subaru Strategic Program Catalogues (HSC-SSP) ultradeep data¶
This catalogue comes from dmu0_HSC. We only have n921 and n816 photometry on the ultradeep field.
In the catalogue, we keep:
- The
object_idas unique object identifier; - The position;
- The g, r, i, z, y, n921, n816 aperture magnitude in 2” that we aperture correct;
- The g, r, i, z, y, n921, n816 kron fluxes and magnitudes.
- The extended flag that we convert to a stellariy.
TODO: Check that the magnitudes are AB.
We don't know when the maps have been observed. We will use the year of the reference paper.
In [1]:
from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))
In [2]:
%matplotlib inline
#%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))
plt.style.use('ggplot')
from collections import OrderedDict
import os
from astropy import units as u
from astropy import visualization as vis
from astropy.coordinates import SkyCoord
from astropy.table import Column, Table
import numpy as np
from herschelhelp_internal.flagging import gaia_flag_column
from herschelhelp_internal.masterlist import nb_astcor_diag_plot, nb_plot_mag_ap_evol, \
nb_plot_mag_vs_apcor, remove_duplicates
from herschelhelp_internal.utils import astrometric_correction, mag_to_flux, aperture_correction
In [3]:
OUT_DIR = os.environ.get('TMP_DIR', "./data_tmp")
try:
os.makedirs(OUT_DIR)
except FileExistsError:
pass
RA_COL = "hsc-udeep_ra"
DEC_COL = "hsc-udeep_dec"
In [4]:
# Pritine HSC catalogue
orig_hsc = Table.read("../../dmu0/dmu0_HSC/data/HSC-PDR1_uDeep_XMM-LSS.fits")
I - Aperture correction¶
To compute aperture correction we need to dertermine two parametres: the target aperture and the range of magnitudes for the stars that will be used to compute the correction.
Target aperture: To determine the target aperture, we simulate a curve of growth using the provided apertures and draw two figures:
- The evolution of the magnitudes of the objects by plotting on the same plot aperture number vs the mean magnitude.
- The mean gain (loss when negative) of magnitude is each aperture compared to the previous (except for the first of course).
As target aperture, we should use the smallest (i.e. less noisy) aperture for which most of the flux is captures.
Magnitude range: To know what limits in aperture to use when doing the aperture correction, we plot for each magnitude bin the correction that is computed and its RMS. We should then use the wide limits (to use more stars) where the correction is stable and with few dispersion.
In [5]:
bands = ["g", "r", "i", "z", "y", "n921", "n816"]
apertures = ["10", "15", "20", "30", "40", "57", "84", "118", "168", "235"] #Removed "40" and "235" because they lack errors
magnitudes = {}
magnitude_errors ={}
stellarities = {}
for band in bands:
magnitudes[band] = np.array(
[orig_hsc["{}mag_aperture{}".format(band, aperture)] for aperture in apertures]
)
magnitude_errors[band] = np.array(
[orig_hsc["{}mag_aperture{}_err".format(band, aperture)] for aperture in apertures]
)
stellarities[band] = 1 - np.array(orig_hsc["{}classification_extendedness".format(band)])
# Some sources have an infinite magnitude
mask = np.isinf(magnitudes[band])
magnitudes[band][mask] = np.nan
magnitude_errors[band][mask] = np.nan
mag_corr = {}
I.a - g band¶
In [6]:
nb_plot_mag_ap_evol(magnitudes['g'], stellarities['g'], labels=apertures)
We will use aperture 40 as target.
In [7]:
nb_plot_mag_vs_apcor(orig_hsc['gmag_aperture20'], orig_hsc['gmag_aperture40'], stellarities['g'])
We will use magnitudes between 20.0 and 21.5
In [8]:
# Aperture correction
mag_corr['g'], num, std = aperture_correction(
orig_hsc['gmag_aperture20'], orig_hsc['gmag_aperture40'],
stellarities['g'],
mag_min=20.0, mag_max=21.5)
print("Aperture correction for g band:")
print("Correction: {}".format(mag_corr['g']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
I.b - r band¶
In [9]:
nb_plot_mag_ap_evol(magnitudes['r'], stellarities['r'], labels=apertures)
We will use aperture 40 as target.
In [10]:
nb_plot_mag_vs_apcor(orig_hsc['rmag_aperture20'], orig_hsc['rmag_aperture40'], stellarities['r'])
We use magnitudes between 19.0 and 20.5.
In [11]:
# Aperture correction
mag_corr['r'], num, std = aperture_correction(
orig_hsc['rmag_aperture20'], orig_hsc['rmag_aperture40'],
stellarities['r'],
mag_min=19.0, mag_max=20.5)
print("Aperture correction for r band:")
print("Correction: {}".format(mag_corr['r']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
I.c - i band¶
In [12]:
nb_plot_mag_ap_evol(magnitudes['i'], stellarities['i'], labels=apertures)
We will use aperture 40 as target.
In [13]:
nb_plot_mag_vs_apcor(orig_hsc['imag_aperture20'], orig_hsc['imag_aperture40'], stellarities['i'])
We use magnitudes between 18.5 and 19.8.
In [14]:
# Aperture correction
mag_corr['i'], num, std = aperture_correction(
orig_hsc['imag_aperture20'], orig_hsc['imag_aperture40'],
stellarities['i'],
mag_min=18.5, mag_max=19.8)
print("Aperture correction for i band:")
print("Correction: {}".format(mag_corr['i']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
I.d - z band¶
In [15]:
nb_plot_mag_ap_evol(magnitudes['z'], stellarities['z'], labels=apertures)
We will use aperture 40 as target.
In [16]:
nb_plot_mag_vs_apcor(orig_hsc['zmag_aperture20'], orig_hsc['zmag_aperture40'], stellarities['z'])
We use magnitudes between 18.5 and 19.8.
In [17]:
# Aperture correction
mag_corr['z'], num, std = aperture_correction(
orig_hsc['zmag_aperture20'], orig_hsc['zmag_aperture40'],
stellarities['z'],
mag_min=18.5, mag_max=19.8)
print("Aperture correction for z band:")
print("Correction: {}".format(mag_corr['z']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
I.e - y band¶
In [18]:
nb_plot_mag_ap_evol(magnitudes['y'], stellarities['y'], labels=apertures)
We will use aperture 40 as target.
In [19]:
nb_plot_mag_vs_apcor(orig_hsc['ymag_aperture20'], orig_hsc['ymag_aperture40'], stellarities['y'])
We use magnitudes between 18.0 and 19.5.
In [20]:
# Aperture correction
mag_corr['y'], num, std = aperture_correction(
orig_hsc['ymag_aperture20'], orig_hsc['ymag_aperture40'],
stellarities['y'],
mag_min=18.0, mag_max=19.5)
print("Aperture correction for y band:")
print("Correction: {}".format(mag_corr['y']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
I.f - n921 band¶
In [21]:
nb_plot_mag_ap_evol(magnitudes['n921'], stellarities['n921'], labels=apertures)
We will use aperture 40 as target.
In [22]:
nb_plot_mag_vs_apcor(orig_hsc['n921mag_aperture20'], orig_hsc['n921mag_aperture40'], stellarities['n921'])
We use magnitudes between 18.0 and 19.5.
In [23]:
# Aperture correction
mag_corr['n921'], num, std = aperture_correction(
orig_hsc['n921mag_aperture20'], orig_hsc['n921mag_aperture40'],
stellarities['n921'],
mag_min=18.0, mag_max=19.5)
print("Aperture correction for n921 band:")
print("Correction: {}".format(mag_corr['n921']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
I.g - n816 band¶
In [24]:
nb_plot_mag_ap_evol(magnitudes['n816'], stellarities['n816'], labels=apertures)
We will use aperture 40 as target.
In [25]:
nb_plot_mag_vs_apcor(orig_hsc['n816mag_aperture20'], orig_hsc['n816mag_aperture40'], stellarities['n816'])
We use magnitudes between 18.0 and 19.5.
In [26]:
# Aperture correction
mag_corr['n816'], num, std = aperture_correction(
orig_hsc['n816mag_aperture20'], orig_hsc['n816mag_aperture40'],
stellarities['n816'],
mag_min=18.0, mag_max=19.5)
print("Aperture correction for n816 band:")
print("Correction: {}".format(mag_corr['y']))
print("Number of source used: {}".format(num))
print("RMS: {}".format(std))
II - Stellarity¶
HSC does not provide a 0 to 1 stellarity value but a 0/1 extended flag in each band. We are using the same method as UKIDSS (cf this page) to compute a stellarity based on the class in each band:
\begin{equation*} P(star) = \frac{ \prod_{i} P(star)_i }{ \prod_{i} P(star)_i + \prod_{i} P(galaxy)_i } \end{equation*}where $i$ is the band, and with using the same probabilities as UKDISS:
| HSC flag | UKIDSS flag | Meaning | P(star) | P(galaxy) | P(noise) | P(saturated) |
|---|---|---|---|---|---|---|
| -9 | Saturated | 0.0 | 0.0 | 5.0 | 95.0 | |
| -3 | Probable galaxy | 25.0 | 70.0 | 5.0 | 0.0 | |
| -2 | Probable star | 70.0 | 25.0 | 5.0 | 0.0 | |
| 0 | -1 | Star | 90.0 | 5.0 | 5.0 | 0.0 |
| 0 | Noise | 5.0 | 5.0 | 90.0 | 0.0 | |
| 1 | +1 | Galaxy | 5.0 | 90.0 | 5.0 | 0.0 |
In [27]:
# We are creating an array containing the extended flag in all band.
# Some sources have no flag in some band, there will be NaN in the array.
hsc_ext_flag = np.array([
orig_hsc[colname] for colname in
['gclassification_extendedness',
'rclassification_extendedness',
'iclassification_extendedness',
'zclassification_extendedness',
'yclassification_extendedness',
'n921classification_extendedness',
'n816classification_extendedness']
])
In [28]:
hsc_pstar = 0.9 * (hsc_ext_flag == 0) + 0.05 * (hsc_ext_flag == 1)
hsc_pgal = 0.05 * (hsc_ext_flag == 0) + 0.9 * (hsc_ext_flag == 1)
# We put back the NaN values
hsc_pstar[np.isnan(hsc_ext_flag)] = np.nan
hsc_pgal[np.isnan(hsc_ext_flag)] = np.nan
In [29]:
stellarity = np.nanprod(hsc_pstar, axis=0) / np.nansum(
[np.nanprod(hsc_pgal, axis=0), np.nanprod(hsc_pstar, axis=0)], axis=0)
stellarity = np.round(stellarity, 3)
In [30]:
vis.hist(stellarity, bins='scott');
In [31]:
orig_hsc.add_column(Column(data=stellarity, name="stellarity"))
II - Column selection¶
In [32]:
imported_columns = OrderedDict({
"object_id": "hsc-udeep_id",
"ra": "hsc-udeep_ra",
"dec": "hsc-udeep_dec",
"gmag_aperture20": "m_ap_hsc-udeep_g",
"gmag_aperture20_err": "merr_ap_hsc-udeep_g",
"gmag_kron": "m_hsc-udeep_g",
"gmag_kron_err": "merr_hsc-udeep_g",
"rmag_aperture20": "m_ap_hsc-udeep_r",
"rmag_aperture20_err": "merr_ap_hsc-udeep_r",
"rmag_kron": "m_hsc-udeep_r",
"rmag_kron_err": "merr_hsc-udeep_r",
"imag_aperture20": "m_ap_hsc-udeep_i",
"imag_aperture20_err": "merr_ap_hsc-udeep_i",
"imag_kron": "m_hsc-udeep_i",
"imag_kron_err": "merr_hsc-udeep_i",
"zmag_aperture20": "m_ap_hsc-udeep_z",
"zmag_aperture20_err": "merr_ap_hsc-udeep_z",
"zmag_kron": "m_hsc-udeep_z",
"zmag_kron_err": "merr_hsc-udeep_z",
"ymag_aperture20": "m_ap_hsc-udeep_y",
"ymag_aperture20_err": "merr_ap_hsc-udeep_y",
"ymag_kron": "m_hsc-udeep_y",
"ymag_kron_err": "merr_hsc-udeep_y",
"n921mag_aperture20": "m_ap_hsc-udeep_n921",
"n921mag_aperture20_err": "merr_ap_hsc-udeep_n921",
"n921mag_kron": "m_hsc-udeep_n921",
"n921mag_kron_err": "merr_hsc-udeep_n921",
"n816mag_aperture20": "m_ap_hsc-udeep_n816",
"n816mag_aperture20_err": "merr_ap_hsc-udeep_n816",
"n816mag_kron": "m_hsc-udeep_n816",
"n816mag_kron_err": "merr_hsc-udeep_n816",
"stellarity": "hsc-udeep_stellarity"
})
catalogue = orig_hsc[list(imported_columns)]
for column in imported_columns:
catalogue[column].name = imported_columns[column]
epoch = 2017
# Clean table metadata
catalogue.meta = None
In [33]:
# Aperture correction
for band in bands:
catalogue["m_ap_hsc-udeep_{}".format(band)] += mag_corr[band]
In [34]:
# Adding flux and band-flag columns
for col in catalogue.colnames:
if col.startswith('m_'):
errcol = "merr{}".format(col[1:])
flux, error = mag_to_flux(np.array(catalogue[col]), np.array(catalogue[errcol]))
# Fluxes are added in µJy
catalogue.add_column(Column(flux * 1.e6, name="f{}".format(col[1:])))
catalogue.add_column(Column(error * 1.e6, name="f{}".format(errcol[1:])))
# Band-flag column
if 'ap' not in col:
catalogue.add_column(Column(np.zeros(len(catalogue), dtype=bool), name="flag{}".format(col[1:])))
# TODO: Set to True the flag columns for fluxes that should not be used for SED fitting.
In [35]:
catalogue[:10].show_in_notebook()
Out[35]:
III - Removal of duplicated sources¶
We remove duplicated objects from the input catalogues.
In [36]:
SORT_COLS = [
'merr_ap_hsc-udeep_i', 'merr_ap_hsc-udeep_r', 'merr_ap_hsc-udeep_z',
'merr_ap_hsc-udeep_y', 'merr_ap_hsc-udeep_g']
FLAG_NAME = 'hsc-udeep_flag_cleaned'
nb_orig_sources = len(catalogue)
catalogue = remove_duplicates(
catalogue, RA_COL, DEC_COL,
sort_col= SORT_COLS,
flag_name=FLAG_NAME)
nb_sources = len(catalogue)
print("The initial catalogue had {} sources.".format(nb_orig_sources))
print("The cleaned catalogue has {} sources ({} removed).".format(nb_sources, nb_orig_sources - nb_sources))
print("The cleaned catalogue has {} sources flagged as having been cleaned".format(np.sum(catalogue[FLAG_NAME])))
III - Astrometry correction¶
We match the astrometry to the Gaia one. We limit the Gaia catalogue to sources with a g band flux between the 30th and the 70th percentile. Some quick tests show that this give the lower dispersion in the results.
In [37]:
gaia = Table.read("../../dmu0/dmu0_GAIA/data/GAIA_XMM-LSS.fits")
gaia_coords = SkyCoord(gaia['ra'], gaia['dec'])
In [38]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL],
gaia_coords.ra, gaia_coords.dec)
In [39]:
delta_ra, delta_dec = astrometric_correction(
SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]),
gaia_coords
)
print("RA correction: {}".format(delta_ra))
print("Dec correction: {}".format(delta_dec))
In [40]:
catalogue[RA_COL] += delta_ra.to(u.deg)
catalogue[DEC_COL] += delta_dec.to(u.deg)
In [41]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL],
gaia_coords.ra, gaia_coords.dec)
IV - Flagging Gaia objects¶
In [42]:
catalogue.add_column(
gaia_flag_column(SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]), epoch, gaia)
)
In [43]:
GAIA_FLAG_NAME = "hsc-udeep_flag_gaia"
catalogue['flag_gaia'].name = GAIA_FLAG_NAME
print("{} sources flagged.".format(np.sum(catalogue[GAIA_FLAG_NAME] > 0)))
V - Saving to disk¶
In [44]:
catalogue.write("{}/HSC-UDEEP.fits".format(OUT_DIR), overwrite=True)