{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Legacy MIPS map homogenisation\n",
"\n",
"\n",
"The original MIPS maps have been produced by IPAC. This code takes the individual image files and adds them into a single multi-layered fits files to have the homogeneous HELP format. \n",
"\n",
"## Caveats \n",
"* I've run this on the MIPS 24/70/160 micron maps\n",
"* only puts in image, error and coverage files (and adds blank PSF extension)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Guide
\n",
"Steps:
\n",
"* Obtain list of fields from sub-folders\n",
"for each field:\n",
"* read in coverage, image and noise\n",
"* attach meta data\n",
"* create PSF extension\n",
"* Save out\n",
"\n",
"Things you may wish to edit
\n",
"* You'll need to change the folder path in the 2nd code block to point you where the files are in your system\n",
"* Add in a specifc field (or switch to wild chracter) in the path to pick field(s)\n",
"* read in more of the available files and add as additional extensions?????\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Importing libraries"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"from datetime import datetime\n",
"from itertools import product\n",
"import glob\n",
"\n",
"import numpy as np\n",
"import astropy as astropy\n",
"\n",
"from astropy.io import fits\n",
"from astropy.table import Table\n",
"from astropy.table import Column\n",
"from astropy import wcs\n",
"\n",
"import shutil\n",
"\n",
"VERSION = \"0.9\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# This is where you set up the field and folders to work on\n",
"the notebook needs to be run for each field in turn - commenting in/out as appropriate"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"#field_specified='HDF-N' \n",
"#field_specified='XMM-LSS' \n",
"#field_specified='AKARI-NEP' \n",
"#field_specified='AKARI-SEP' \n",
"#field_specified='Bootes' \n",
"#field_specified='CDFS-SWIRE' \n",
"#field_specified='COSMOS' \n",
"#field_specified='EGS' \n",
"#field_specified='ELAIS-N1' \n",
"field_specified='ELAIS-N2' \n",
"#field_specified='ELAIS-S1' \n",
"#field_specified='GAMA-09' \n",
"#field_specified='GAMA-12' \n",
"#field_specified='GAMA-15' \n",
"#field_specified='NGP' \n",
"#field_specified='SGP' \n",
"#field_specified='Lockman-SWIRE' \n",
"#field_specified='SA13' \n",
"#field_specified='SPIRE-NEP' \n",
"#field_specified='SSDF' \n",
"#field_specified='xFLS' \n",
"#field_specified='XMM-13hr' \n",
"#field_specified='Herschel-Stripe-82' \n",
"\n",
"\n",
"mainfolder='../../dmu17/dmu17_Legacy_maps/'\n",
"moc_dir= '../../dmu2/dmu2_field_coverages/'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# finding all the Legacy mosaic files in the folder"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Number of files found: 3\n"
]
}
],
"source": [
"# finding all the Legacy mosaic files (other files should be there if the mosaic file is)\n",
"filelist= glob.glob(f\"{mainfolder}/{field_specified}/data/*mosaic.fits\", recursive=True)\n",
"print(\"Number of files found:\", len(filelist))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Performing the homogenisation\n",
"i.e. merging component files into a single multi-extension file and adding some keywords\n",
"traps errors so that it can continue, e.g. if a file is missing\n",
"skips over files if they've already been logged as \"downloaded\" (this could be improved to be an option"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"----------------\n",
"0 ../../dmu17/dmu17_Legacy_maps//ELAIS-N2/data/swire_EN2_M3_v4_mosaic.fits\n",
"ELAIS-N2\n",
"../../dmu17/dmu17_Legacy_maps//ELAIS-N2/data/swire_EN2_M3_v4_mosaic.fits / processed...\n",
"----------------\n",
"1 ../../dmu17/dmu17_Legacy_maps//ELAIS-N2/data/swire_EN2_M1_v4_mosaic.fits\n",
"ELAIS-N2\n",
"../../dmu17/dmu17_Legacy_maps//ELAIS-N2/data/swire_EN2_M1_v4_mosaic.fits / processed...\n",
"----------------\n",
"2 ../../dmu17/dmu17_Legacy_maps//ELAIS-N2/data/swire_EN2_M2_v4_mosaic.fits\n",
"ELAIS-N2\n",
"../../dmu17/dmu17_Legacy_maps//ELAIS-N2/data/swire_EN2_M2_v4_mosaic.fits / processed...\n"
]
}
],
"source": [
"for i, file in enumerate(filelist):\n",
" print('----------------')\n",
" print(i,file)\n",
" folders=file.split('/')\n",
" print(folders[len(folders)-3])\n",
" field=folders[len(folders)-3]\n",
" filename=folders[len(folders)-1]\n",
" ext=0\n",
" \n",
" # Load in image map\n",
" image_map=fits.open(file)\n",
" image_hdu = fits.ImageHDU(header=image_map[ext].header,\n",
" data=image_map[ext].data)\n",
" image_hdu.header['EXTNAME'] = \"IMAGE\"\n",
" \n",
" # Load in uncertainty map\n",
" noise_map=fits.open(file.replace('mosaic','mosaic_unc'))\n",
" noise_hdu = fits.ImageHDU(header=noise_map[ext].header,\n",
" data=noise_map[ext].data)\n",
" noise_hdu.header['EXTNAME'] = \"ERROR\"\n",
" \n",
" \n",
" # Load in coverage map\n",
" cov_map=fits.open(file.replace('mosaic','mosaic_cov'))\n",
" cov_hdu = fits.ImageHDU(header=cov_map[ext].header,\n",
" data=cov_map[ext].data)\n",
" cov_hdu.header['EXTNAME'] = \"COVERAGE\"\n",
"\n",
" \n",
" #Create primary header\n",
" \n",
" CHNLNUM=image_hdu.header['CHNLNUM'] \n",
" primary_hdu = fits.PrimaryHDU()\n",
" primary_hdu.header.append((\n",
" \"CREATOR\", \"Herschel Extragalactic Legacy Project\"\n",
" ))\n",
" primary_hdu.header.append((\n",
" \"TIMESYS\", \"UTC\", \"All dates are in UTC time\"\n",
" ))\n",
" primary_hdu.header.append((\n",
" \"DATE\", datetime.now().replace(microsecond=0).isoformat(),\n",
" \"Date of file creation\"\n",
" ))\n",
" primary_hdu.header.append((\n",
" \"VERSION\", VERSION, \"HELP product version\"\n",
" ))\n",
" primary_hdu.header.append((\n",
" \"TELESCOP\", \"Spitzer\", \"Name of the telescope\"\n",
" ))\n",
" primary_hdu.header.append((\n",
" \"INSTRUME\", \"MIPS\", \"Name of the instrument\"\n",
" ))\n",
" primary_hdu.header.append((\n",
" \"FILTER\", CHNLNUM, \"1=24um,2=70um,3=160um\"\n",
" ))\n",
" primary_hdu.header.append((\n",
" \"FIELD\", field, \"Name of the HELP field\"\n",
" ))\n",
"\n",
"\n",
" #Add empty extension for PSF\n",
" psf_hdu = fits.ImageHDU()\n",
" psf_hdu.header['EXTNAME'] = \"PSF\"\n",
" psf_hdu.header.add_comment(\"The PSF is not available.\")\n",
"\n",
" hdu_list = fits.HDUList([primary_hdu, image_hdu, noise_hdu, cov_hdu, psf_hdu])\n",
" hdu_list.writeto(file.replace('mosaic','help'),\n",
" checksum=True, overwrite=True)\n",
" print(f\"{file} / processed...\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Moving the files\n",
"Once we run this code, we obtain the Help Legacy Maps, in the folder: /dmu17_Legacy_maps/'. We need to put them in /dmu17_HELP_Legacy_Maps/"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"dest = f\"./{field_specified}/data/\"\n",
"files = glob.glob(f\"{mainfolder}/{field_specified}/data/*_help.fits\", recursive=True)\n",
"\n",
"for f in files:\n",
" shutil.move(f, dest)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"-------------------------------------------------------------------------------\n",
"\n",
"\n",
"\n",
"**Authors**: This code was adapted from code used for SPIRE homogenisaton (Yannick Roehlly) and in turn adapted for PACS data (Peter Hurley). [Seb Oliver](http://www.sussex.ac.uk/profiles/91548)\n",
"\n",
" \n",
"For a full description of the database and how it is organised in to `dmu_products` please the top level [readme](../readme.md).\n",
" \n",
"The Herschel Extragalactic Legacy Project, ([HELP](http://herschel.sussex.ac.uk/)), is a [European Commission Research Executive Agency](https://ec.europa.eu/info/departments/research-executive-agency_en)\n",
"funded project under the SP1-Cooperation, Collaborative project, Small or medium-scale focused research project, FP7-SPACE-2013-1 scheme, Grant Agreement\n",
"Number 607254.\n",
"\n",
"[Acknowledgements](http://herschel.sussex.ac.uk/acknowledgements)\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.6"
}
},
"nbformat": 4,
"nbformat_minor": 2
}