xFLS master catalogue

Preparation of KPNO-FLS data

Isaac Newton Telescope / Wide Field Camera (INT/WFC) catalogue: the catalogue comes from dmu0_KPNO-FLS.

In the catalogue, we keep:

  • The identifier (it's unique in the catalogue);
  • The position;
  • The stellarity;
  • The aperture magnitude;
  • The total magnitude.

We don't know when the maps have been observed. We will use the year of the reference paper.

In [1]:
from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))

This notebook was run with herschelhelp_internal version: 
255270d (Fri Nov 24 10:35:51 2017 +0000)
In [2]:
%matplotlib inline
#%config InlineBackend.figure_format = 'svg'

import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))

from collections import OrderedDict
import os

from astropy import units as u
from astropy.coordinates import SkyCoord
from astropy.table import Column, Table
import numpy as np

from herschelhelp_internal.flagging import  gaia_flag_column
from herschelhelp_internal.masterlist import nb_astcor_diag_plot, remove_duplicates
from herschelhelp_internal.utils import astrometric_correction, mag_to_flux
In [3]:
OUT_DIR =  os.environ.get('TMP_DIR', "./data_tmp")
try:
    os.makedirs(OUT_DIR)
except FileExistsError:
    pass

RA_COL = "kpno_ra"
DEC_COL = "kpno_dec"

I - Column selection

In [4]:
imported_columns = OrderedDict({
        'internal_id': "kpno_intid",
        'RAJ2000': "kpno_ra",
        'DEJ2000': "kpno_dec",
        'Class':  "kpno_stellarity",
        'RcmagAp': "m_ap_kpno_r", 
        'e_RcmagAp': "merr_ap_kpno_r", 
        'RcmagTot': "m_kpno_r", 
        'e_RcmagTot': "merr_kpno_r",
    })


catalogue = Table.read("../../dmu0/dmu0_KPNO-FLS/data/KPNO-FLS_xFLS.fits")[list(imported_columns)]
for column in imported_columns:
    catalogue[column].name = imported_columns[column]

epoch = 2011

# Clean table metadata
catalogue.meta = None
In [5]:
# Adding flux and band-flag columns
for col in catalogue.colnames:
    if col.startswith('m_'):
        
        errcol = "merr{}".format(col[1:]) 

        flux, error = mag_to_flux(np.array(catalogue[col]), np.array(catalogue[errcol]))
        
        # Fluxes are added in µJy
        catalogue.add_column(Column(flux * 1.e6, name="f{}".format(col[1:])))
        catalogue.add_column(Column(error * 1.e6, name="f{}".format(errcol[1:])))
        
        # Band-flag column
        if "ap" not in col:
            catalogue.add_column(Column(np.zeros(len(catalogue), dtype=bool), name="flag{}".format(col[1:])))
        
# TODO: Set to True the flag columns for fluxes that should not be used for SED fitting.
In [6]:
catalogue[:10].show_in_notebook()
Out[6]:
<Table length=10>
idxkpno_intidkpno_rakpno_deckpno_stellaritym_ap_kpno_rmerr_ap_kpno_rm_kpno_rmerr_kpno_rf_ap_kpno_rferr_ap_kpno_rf_kpno_rferr_kpno_rflag_kpno_r
degdegmagmagmagmag
01258.01879583359.63627777780.36823.710.1123.120.131.191240.1206892.051160.245594False
12258.018359.63584722220.30723.430.0822.470.111.54170.1135973.73250.378154False
23258.02019583359.63564722220.05521.880.0221.490.036.426880.1183879.204490.254329False
34258.050759.63690833330.85424.170.1623.580.190.779830.114921.342760.234979False
45257.901259.63560833330.6324.60.2524.560.170.5248070.1208410.5445030.085256False
56258.066759.63688888890.98222.820.0422.80.042.703960.09961752.754230.10147False
67257.964259.63542777780.02921.410.0120.760.029.908320.09125918.03020.332128False
78258.069259.63770833330.91124.20.0824.20.080.7585770.0558940.7585770.055894False
89258.10559.63805833330.923.760.1123.680.191.137630.1152571.224620.214303False
910257.94759.63592777780.62423.20.0723.030.081.905460.122852.228430.164197False

II - Removal of duplicated sources

We remove duplicated objects from the input catalogues.

In [7]:
SORT_COLS = ['merr_ap_kpno_r']
FLAG_NAME = 'kpno_flag_cleaned'

nb_orig_sources = len(catalogue)

catalogue = remove_duplicates(catalogue, RA_COL, DEC_COL, sort_col=SORT_COLS,flag_name=FLAG_NAME)

nb_sources = len(catalogue)

print("The initial catalogue had {} sources.".format(nb_orig_sources))
print("The cleaned catalogue has {} sources ({} removed).".format(nb_sources, nb_orig_sources - nb_sources))
print("The cleaned catalogue has {} sources flagged as having been cleaned".format(np.sum(catalogue[FLAG_NAME])))

The initial catalogue had 722525 sources.
The cleaned catalogue has 670833 sources (51692 removed).
The cleaned catalogue has 50497 sources flagged as having been cleaned

III - Astrometry correction

We match the astrometry to the Gaia one. We limit the Gaia catalogue to sources with a g band flux between the 30th and the 70th percentile. Some quick tests show that this give the lower dispersion in the results.

In [8]:
gaia = Table.read("../../dmu0/dmu0_GAIA/data/GAIA_xFLS.fits")
gaia_coords = SkyCoord(gaia['ra'], gaia['dec'])
In [9]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec)
In [10]:
delta_ra, delta_dec =  astrometric_correction(
    SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]),
    gaia_coords
)

print("RA correction: {}".format(delta_ra))
print("Dec correction: {}".format(delta_dec))

RA correction: -0.3095527144182597 arcsec
Dec correction: -0.12726026545522018 arcsec
In [11]:
catalogue[RA_COL] =  catalogue[RA_COL] + delta_ra.to(u.deg)
catalogue[DEC_COL] = catalogue[DEC_COL]  + delta_dec.to(u.deg)
In [12]:
nb_astcor_diag_plot(catalogue[RA_COL], catalogue[DEC_COL], 
                    gaia_coords.ra, gaia_coords.dec)

IV - Flagging Gaia objects

In [13]:
catalogue.add_column(
    gaia_flag_column(SkyCoord(catalogue[RA_COL], catalogue[DEC_COL]), epoch, gaia)
)
In [14]:
GAIA_FLAG_NAME = "kpno_flag_gaia"

catalogue['flag_gaia'].name = GAIA_FLAG_NAME
print("{} sources flagged.".format(np.sum(catalogue[GAIA_FLAG_NAME] > 0)))

24932 sources flagged.

V - Saving to disk

In [15]:
catalogue.write("{}/KPNO.fits".format(OUT_DIR), overwrite=True)