The p3d logo. a general data-reduction tool for fiber-fed integral-field spectrographs
About p3d




Documentation & Tutorials



Authors & Thanks

Sourceforge project page


The software can be downloaded in two ways – either download a stable release version or a development version from the project Subversion tree. Either version contains all source files, required data files, and pre-compiled binary files (without these binary files p3d cannot be used without an IDL license).

Stable release
Development version

Information on how p3d is installed on the system is described on the readme page.

Note on the installation procedure on Windows platforms:
Although p3d can be used under Windows, we have not yet been able to write detailed installation instructions for this platform; we do not use Windows machines in our own research. We would be most grateful if you could help us out with such instructions. The installation procedure is in principle the same as for the other two platforms; i.e. Linux and Mac OSX.

Release History
Change log
Here is a list of changes between the releases (the text is a copy of the NEWS file).
$Id: NEWS 4665 2017-06-28 11:04:48Z christersandin $

2.5 (2017-06-28; Revision 4664) - Heart of Gold
  o Simulation tool (p3d_csim):
        Multiple important bug fixes.
     - Correct the written simulated raw files so that they can be reduced
       with the p3d data reduction tools.

  o Tutorials and associated scripts (in doc/):
     - Add images to the description of the dispersion-mask creation part in
       all three data-reduction tutorials (MRS1, PMAS-Larr, and PMAS-PPak).

     - Add the feature to all scripts that the environment variable "rootpath"
       can be set to change the path; this way it isn't necessary anymore to
       edit any of the scripts before using them. Adjust the line numbers in
       all tutorial scripts to apply to the adjusted scripts.

  o All tools (in ${p3d_path}):
        Important bug fixes and ajustments.
     - Instruments that use the NGC CCD controller and provide combined raw
       images that include prescan and overscan regions in the image (such as
       ERA2) are now correctly sliced up again (i.e., the prescan and overscan
       regions are correctly removed). The bug was introduced with revision
       4330 and fixed with revision(s 4620 and) 4630.

     - Set the default polynomial order used when fitting extracted continuum
       spectra with a linear polynomial to 0 [MRS1 instrument]. The spectrum
       is difficult to fit with lower order polynomials, and doing so anyways
       introduces strange features in flat-fielded spectra.

     - Correct all examples that are run from the command line using a shell -
       remove all white space before and after equal signs (that I introduced
       earlier to make the source code easier to read).

  o A number of minor bug fixes and adjustments to the language of comments
    throughout the code.

2.4.3 (2017-06-06; Revision 4608) - Millenium Falcon
  o Simulation tool (p3d_csim):
        New tool.
     - Add a tool to add spectra to simulated "raw" CCD images. The tool
       p3d_csim can be used with any setup that contains the required
       header keywords preset file. The first added setup is to model
       Mosaic-like detectors, it is called mosms. Here is a list of features
       that are included in the simulations:

       + Select the instrument configuration, and thereby lock information
         about how many spectra to model, their separation, their geometrical
	 placement on the IFU, and the wavelength range.

       + Select the spectra to work with.
         o The object spectrum can currently be that of a point source, which
	   can be provided as a two-column plain-text file or a binary-table
	   fits file. The point source is spread out using a defined seeing and
	   a modeled two-dimensional Gaussian profile.
	 o A continuum spectrum is used to create traces for all spectra;
	   either provide a spectrum in a file or use a constant value
	 o An arc spectrum is created using a list of wavelengths and
	   intensities, provided as a two-column plain-text file or a arrays.

       + For each kind of image, except the bias images, the spectra are
         centered on the spatial axis on the simulated CCD image. The spectra
	 are convolved separately on the dispersion and cross-dispersion
	 (spatial) axes:

         o One kernel or a set of kernels spread out on the dispersion axis
	   describe how the dispersion varies on the dispersion axis. It is
	   possible to use the same kernel(s) for all spectra, but one can also
	   use different kernels. The resulting kernel is interpolated for
	   pixels intermediate to the provided wavelengths. The kernel can be
	   subsampled to any depth.

           Variations of the dispersion are accounted for by offsetting the
	   dispersion kernel using an optional dispersion mask.

           The spectrum is at first convolved with the kernel on the
	   dispersion axis.

         o Another kernel describes how the spectrum is spread out on the
	   cross-dispersion axis. It is specified just as the dispersion-axis
	   kernel described above.

           Spectra that are curved on the CCD image are account for by
	   offsetting the cross-dispersion kernel using a trace fit for each
	   spectrum, provided as a set of polynomial parameters.

           The spectrum is then multiplied with the normalized cross-dispersion
	   axis profile for each pixel on the dispersion axis. The resulting
	   profile is added to the output image.

         o After both convolutions, the output image is rebinned to remove the

       + Each image is split into blocks, according to the definition in the
         instrument parameter files. Prescan and overscan regions are added to
	 the image and it is flipped according to how the corresponding
	 instrument CCD image would appear.

         Poisson noise and readout noise as well as a bias level is added to
	 the image before it is converted to the unsigned integer type
	 (16 bits per pixel) and is written to a FITS file using multiple

       + The idea is then that the resulting bias, continuum, arc, and object
         images are fed back to the reduction tools of p3d to reduce the files
	 to allow an analysis of what results.

2.4.2 (2017-04-13; Revision 4551) - Millenium Falcon
  o Tutorials (in doc/):
        New data-reduction and visualization tutorials.
     - Add data-reduction and visualization tutorials for three instrument
       setups: MRS1, PMAS/Larr, and PMAS/PPak.

       + The data-reduction tutorials describe all steps necessary to convert
         raw input images to extracted spectra; this includes the five steps of
	 creating a master bias, tracing the spectra, creating a dispersion
	 mask, extracting spectra for normalization of extracted spectra, and
	 extracting object spectra, using all previous products. The PMAS/Larr
	 tutorial is a bit more extensive than the other two and also includes
	 the creation of a sensitivity function, flux calibration using the
	 sensitivity function, correcting for effects of DAR, and converting
	 the resulting RSS image to a data cube. The tutorials come with both
	 reduction scripts - written in Python, bash, and IDL - and data files
	 that can be used to follow all steps of the reductions. The tutorial
	 data are available in a file that can be downloaded through a link on
	 the p3d web page. The reduction scripts are available both in the p3d
	 distribution as well as in the tutorial data files.
       + The visualization tutorials describe how one can make use of the
         functionality in the p3d spectrum viewer. Examples of covered issues
	   o Starting the spectrum viewer from the command line.
           o Using the spectrum image: changing colors, selecting wavelength
	     bins, panning in wavelength and scrolling across spectra.
	   o Using the spatial map: panning and zooming, creating regions,
	     adding background regions, and blinking between spatial-map
	     images of different wavelength bins.
	   o Using the spectrum plot: panning and zooming, showing emission
	     lines, fitting emission lines and setting the redshift.
         As with the data-reduction tutorials, the visualization tutorials come
	 with both launch scripts - written in Python, bash, and IDL - and data
	 files that can be used to follow the tutorial. The tutorial data are
	 available in a file that can be downloaded through a link on the p3d
	 web page. The launch scripts are available both in the p3d
	 distribution and in the tutorial data files.

  o The science-object extraction routine (p3d_cobjex):
        Add a new feature.
     - The keyword cosmic-ray cleaning keyword CRCLEAN can now be specified
       for each input image individually.

  o The tracing procedure (p3d_ctrace):
        Add new functionality.
     - Add a new keyword, IMCOMBINE, which allows the tool to either combine
       all input images before tracing spectra (as before) or trace spectra in
       each image individually.

  o Correction for differential atmospheric refraction (p3d_darc):
        Minor adjustments.
     - Correct a few minor bugs to make the tool work anew with instruments
       such as the PMAS/Larr.
     - In the properties popup GUI: add a label on the first line to indicate
       which file the properties are shown for.

  o All tools (in ${p3d_path}):
        Bug fixes and minor adjustments.
     - Correct the handling of pre-scan and over-scan regions so that they
       can again be used instead of master-bias images, if necessary.
     - Correct the creation of landscape PostScript images, they are anew
       created as intended.
     - Correct the handling of decimal-type keywords when the tools are called
       using Python.
     - Capitalize widget labels to increase the appeal.

  o Numerous minor bug fixes and adjustments to comments throughout the code.

2.4.1 (2017-03-10; Revision 4452) - Millenium Falcon
  o Tutorials (in doc/):
        New tutorial (which is an all-new feature).
     - Add an instrument configuration tutorial for MRS1. The tutorial
       describes all steps necessary to setup p3d for the instrument MRS1 (and
       this tutorial can be used to setup similar instruments as well). The
       tutorial comes with a small data set that can be used to have a look at
       the exact format of the raw files that p3d expects for this instrument.
       The tutorial data are available through a link on the p3d web page.

  o All tools (in ${p3d_path}):
        Instrument setups.
     - Adjust the setup of MRS1.

  o The use of prescan and overscan regions (p3d_tool_pobias):
        Important bug fix.
     - Corrected a serious bug, where two keywords had ambiguous names;
       resulting in the feature of using prescan and overscan regions
       being unavailable.
     - Correct the output to the PostScript file of the value of

  o Line fitting (p3d_ifsfit, p3d_ifsfit_mosaic):
        Minor (bug) fixes.
     - Fix output maps to conform to FITS specifications.
     - Exclude pixels with non-positive errors from both binning and single
       spectra; this allows the C-based routine cmpfit to always converge.
     - Fix a type to convert MUSE variances to sigmas.

  o Some additional minor bug fixes and adjustments to comments.

2.4 (2016-09-10; Revision 4387) - Millenium Falcon

  o Spectrum Viewer (p3d_sv):
        New functionality.
     - Increase the number of definable regions to 1024, which may all overlap.
       A region may use any (one) other region as a background (except itself),
       which co-added and scaled spectrum is subtracted from the co-added main
       spectrum. The spectrum plot now shows either the unsubtracted spectrum,
       the background-subtracted spectrum (this is the default), the
       background-subtracted spectrum and the background spectrum, or all three
     - Regions and background regions are selected either by entering the
       required index or by clicking in the spatial map - using the Ctrl and
       Ctrl+Shift modifier keys simultaneously.
     - Regions configurations can be saved as well as loaded to and from a
       plain-text file. Such region files can be loaded already upon launchning
       the spectrum viewer through use of the keyword REGIONS.
     - It is also possible to load DS9 region files. The following seven
       DS9-types of regions are supported: circle, annulus, ellipse (as well as
       ellipse ... & !ellipse ...), box (as well as box ... & !box ...),
       polygon, and text (which is actually a property and not a region). All
       regions can have local properties associated with them; specifically,
       the following properties are recognized: text, width, color, and
       textangle. All colors in the file /usr/share/X11/rgb.txt are recognized.
     - Multiple DS9 region files are loaded either by selecting the load option
       multiple times or by selecting several files simultaneously.
     - Additional checkbox buttons allow viewing all defined regions
       simultaneously in the spatial-map panel, together with the DS9
       properties if such are defined.
     - An additional option in the Spatial-map menu allows all spatial elements
       in all regions to be unselected with one click; a dialog GUI is shown to
       confirm the action.
     - When multiple groups are chosen for the same region in the regions tool
       (XROI), they are now xor:ed when calculating the spatial elements mask.
       For example, as a result, by creating two circles with the same origin
       using different radii, the resulting region is an annulus.
     - Add coordinate information for the spectrum image and the spectrum
       plot. The information includes the wavelength pixel, the wavelength
       or the wavenumber, the flux and its error (when available) at the
       position of the mouse cursor. The background flux (and its error) is
       shown for the spectrum plot when a background region is defined. The
       spectrum number is shown when the mouse pointer hovers above the
       spectrum image.
     - The flux is indicated in the spectrum panel with a diamond symbol. When
       the actual value is higher or smaller than what is seen in the spectrum
       plot, a plus (minus) symbol is shown above (below) the diamond.
     - Add the option to use physical coordinates of the spatial elements
       that are flipped on the East-West axis. The flipped coordinates are the
       same that one would see in the tool ds9.
     - Add the option to fit a Gaussian emission line in the spectrum plot;
       this feature is used by clicking on the respective outer background
       points and then at the position of the emission line. The fitted
       parameters are shown in the coordinate field until either a new fit is
       made or the "Clean Fit" button is selected.
     - Add emission line list files that are used to show the locations of
       emission lines. The redshift used is either the default system redshift
       (specified with the REDSHIFT_Z and REDSHIFT_VEL keywords) or the
       redshift of the line fit. Additional widgets allow the system redshift
       to be altered inside the tool.
     - Data cubes that use the FITS header keyword BUNIT (such as data cubes of
       the MUSE pipeline) now make use of this constant when it is in the form
       '10**(xxx)*erg/s/cm**2/Angstrom' to scale the read data with the factor
       indicated with xxx.

  o Line fitting (p3d_ifsfit):
        Major new functionality and bug fixes.
     - Now also handles binned data in three different approaches:
        o A bin map can be provided as a plain-text file or a binary-table fits
          file. The format of the bin map includes x and y positions for each
          spatial element as well as a bin index. The data can be contained in
          the fits files with a freely chosen extension name (defaults are
          BIN_TABLE and VORONOI_TABLE) and column names (defaults are BINIDX or
          BINNUM for the bin index and X and Y for the //x// and //y//
          positions, respectively).
        o The second approach is to let p3d_ifsfit calculate a bin map using
          the tool VORONOI_2D_BINNING (Cappellari & Copin 2003), where it is
          possible to define the bin range to use and the target
          signal-to-noise value.
        o The final approach is to use a cube image that was already binned,
          and where the bin map is provided either as a separate extension or
          in a separate file. This setup requires quite a number of fulfilled
          conditions to work - please see the documentation of p3d_ifsfit for
          more details.
     - Both binned and unbinned data use the C-based DLM p3d_cifsfit, which
       means fast and parallelized fits.
     - Now also writes spatial map image files in addition to the binary-table
       fits files. The image files contain images of all fitted parameters and
       their errors in separate extensions. While the binary-table fits files
       are meant to be used with p3d_ifsfit_mosaic, the image files can be used
       by any tool reading fits images.

2.3.1 (2016-02-08; Revision 3993) - Bounty

  o All tools:
        Bug fixes.
     - Correct the size of written postscript and PDF documents.
     - Correct all tools so that they can be used with the updated version of
       the IDL astro-lib routine READCOL (the affected version is of Oct 2015).

  o Line fitting (p3d_ifsfit):
        Minor new functionality and bug fixes.
     - Add the *real* redshift as a separate keyword, Z (which could be used
       instead of the wavelength shift of the keyword REDSHIFT). Alternatively,
       the redshift can be specified as a velocity using the new keyword VEL_Z.
     - Correct the C-version of the line-fitting tool.
     - Correct the use of the tool when LOGLEVEL is not used.

  o Graphical presentation of line fits (p3d_ifsfit_mosaic):
        Bug fixes.
     - Correct the filename use when writing to a PDF file.
     - Correct the color bar for non-log plots.

  o Create a dispersion mask (p3d_dmask):
        Bug fix.
     - Correct the tool so that it can be used again.

  o Line fitting (p3d_sv):
        Minor bug fixes.
     - Correct the standard-star tab calibration-file widgets.
     - The fine shifts of the wavelength range in the spectrum-plot window
       (holding the Ctrl or Ctrl+Shift keys when using the mouse wheel) now
       works with a minimum shift to higher wavelengths of 1 pixel (could be 0
       pixel until now).

2.3 (2015-12-16; Revision 3951) - Bounty

  o All tools (in ${p3d_path}):
        Improved the documentation, minor and important new functionality.
     - Entries in log files are now prefixed with a time stamp.
     - Add the option to avoid using the p3d queue system; this is accomplished
       by setting the -s or --noqueue flag when calling the bash scripts or Use this functionality to avoid having to use the p3d
       queue system that requires some effort to learn and understand.
     - Made improvements to the p3d queue system to make it more stable. The
       queue system now requires Python version >= 2.7 to run.
       (Only one hurdle remains to have it work on Windows-based platforms;
        the POSIX tool "xwininfo" is used to check for the existence of the IDL
        popup window on the screen.)
     - Add a p3d-specific IDL distribution that removes the necessity to
       register with EXELISVIS to download IDL; this distribtion is used
       whenever IDL is not installed on the system. IDL is assumed to be free
       of IDL when the environment variable IDL_DIR is unset. Please e-mail us
       to get a download link for this distribution file; use the e-mail
       address in the README file.
     - The spectrum viewer, the dispersion-mask creation GUI, and the new tool
       p3d_ifsfit_mosaic now include the ColorBrewer color tables; also when
       the IDL version < 8.3. These color tables are available as values
       higher than the regular IDL tables. To specify the Spectral ColorBrewer
       color table in an IDL-independent way, it is suggested to use use either
       the command:
         "COLORTABLE='CB25'" (on the IDL command line) or
         "COLORTABLE=CB25" (in a shell or in Python, outside IDL)
       or the equivalent command:
         "COLORTABLE='25,brewer.tbl'" (on the IDL command line) or
         "COLORTABLE=25,brewer.tbl" (in a shell or in Python, outside IDL)
     - All PostScript and PDF printouts can now be made using one of the
       following paper sizes: ISO A2, A3, [A4], A5, US Letter, Legal, or
       Tabloid (A4 is the default when nothing else is specified). The paper
       size is set either with the keywords A2, A3, A5, LETTER, LEGAL, or
       TABLOID or with the user-parameters 'a2', 'a3', 'a5', 'letter', 'legal',
       or 'tabloid'.
     - Add support for the new instrument Wise/WIFU, according to the wishes of
       A. Levi.
     - E3D files are now written with the full header of the original data
     - The p3d main GUI has been updated; mostly on the inside, but also on the
         o Data can now be flux calibrated from the GUI.
         o The IFS/IFU selection GUI is rewritten to use several tabs.
         o The code is rewritten for increased flexibility (it would now be
	   possible to add language files, if the community would have such an
     - Fiber position-table files are now read before they are used, and can be
       individual to an IFU for the same instrument setup.
     - Add rudimentary support for IMACS. Some work is still required with this
       rather complex instrument.

  o The spectrum viewer (p3d_sv):
        Major new functionality
     - The spectrum viewer has been renamed from p3d_rss to p3d_sv, where sv
       stands for spectrum viewer (rss stands for row-stacked spectra).
     - The spectrum viewer can now be used with reduced data cubes of the MUSE
       instrument. The amount of available RAM on the computer sets the only
       limit to what size can be loaded. The keyword NO_ERRORS can be set to
       avoid loading the error or variance image and thereby save on loading
       time and RAM.
     - Spatial maps of square-shaped spatial elements are now drawn with
       regular images, while circular and hexagon-shaped spatial elements keep
       using polygons. Using regular images, the spectrum image and the
       spatial map are recalculated every time they change; this is made so to
       avoid using more memory than the exiting window sizes (enabling the
       use of huge MUSE datacubes).
     - Add coordinate labels that show the world coordinates (RA, DEC), the
       spatial-element id and image row, spatial-element coordinates, and
       window coordinates on the status line.
     - All draw areas on the first two tabs (not the sensitivity-function tab)
       can now be controlled using the scroll wheel - to pan and zoom. Hover
       the mouse pointer above the question mark below the respective draw area
       on the right-hand side to see help on how to use the mouse.
     - Added two reference wavelength bins in the spectrum image that can be
       used with the new blinking functionality to blink between the regular
       wavelength bin and either one or both of the reference bins. The blink
       time can also be configured.
     - All draw widgets are now redrawn by p3d instead of being handled by IDL.
     - The tool can now be resized; the relative height of the spatial-map area
       can be set with a widget slider in the new 'Extra' control panel.
     - The code has been recoded for speed and is split into several source
       files for increased visibility.

  o Line fitting (p3d_ifsfit, p3d_ifsfit_mosaic):
        Major new functionality
     - Add the tool p3d_ifsfit, which can be configured to fit any number of
       emission and absorption lines in IFU data, using any (possible)
       wavelength range and combination of separate and blended lines. For
       example, both a Gaussian-profile emission line (HII regions) and a
       Lorentzian-profile absorption line (stellar population) of Halpha can be
       fitted simultaneously.
     - Add the tool p3d_ifsfit_mosaic that creates output for presentation from
       data of p3d_ifsfit; output to (encapsulated) postscript, screen, or
       bitmap files. Options exist to control sizes, number of panels, colors,
       axes, fonts, properties, color-bar appearance, etc.

  o Numerous minor bug fixes and adjustments to comments throughout the code.

2.2.6 (2014-09-22; Revision 3223) - Serenity

  o All tools (in ${p3d_path}):
        Improved the documentation, minor and important new functionality.
     - Add support for the new instrument ERASMUS-F. This instrument is
       configured in a similar way to how VIRUS is done. The instrument uses
       six bundles that are split on two detectors, left and right. The
       reductions need to use the correct detector, and only one bundle with
       19 fibers each is reduced at any one time.
     - Add support for some of the IFUs of VIRUS (i.e. not VIRUS-P)...
       additional support is added as the IFUs become available (and are used).
     - Line-profile plots are now drawn on multiple pages, for instruments
       that use many pixels.
     - Add the color table "cubehelix" (all its options are available through
       keywords in p3d, and p3d_rss):

  o The spectrum viewer (p3d_rss):
        Minor new functionality
     - The spatial map can now be saved as a bitmap image or to a postscript
       file, the output includes a color bar. Formats include TIFF, JPEG, PNG,
       PS, and EPS. The size of the image is specified in advance with
       droplist widgets in the menu. The output is optionally annotated with
       the spatial-element ID and row number.
     - Initial support to draw the x-axis with the unit wavenumber or
       nanometer, instead of the default unit Ångström.

  o The tracing procedure (p3d_ctrace):
        Add new functionality.
     - Add the new profile function "TophatGaussian" to complement the already
       existing types that are used when calculating cross-dispersion profiles
       ("Gaussian", "Lorentzian", "DoubleGaussian", "Lorentzian/Gaussian"; note
       that the name strings differ slightly compared to before, but the older
       strings are still acccepted, such as "Lorentz/Gauss"). The new profile
       function is now used by default with the VIRUS-P, the VIRUS-W, and the
       VIRUS setups.

       Note! This is not an optimal profile for these instruments, but it
             appears to be slightly better than a Gaussian profile.

  o Compilation of the IDL routines for the p3d server (p3d_tool_makevm):
        Convenience fix.
     - The compilation is now correctly reported to fail if the required IDL
       routines are not available in the path.

  o Various minor bug fixes and adjustments to comments throughout the code. (2013-12-19; Revision 3053) - Serenity

  o Cosmic-ray cleaning (p3d_cr):
        Fixed a bug.
     - The C-based cosmic-ray cleaning routine is reactivated after the
       problematic bug that is mentioned for release 2.2.5 was fixed. The fix
       was applied with revision 3025.

  o Correct a bug in the C routine p3d_cssl_median.c. Switching from using
       IDL_LONG to size_t, in an iterator, I had not checked that the value was
       compared with a negative value. The result was that the code crashed.
       The bug could be circumvented by not using the scattered-light
       subtraction (by using the following statement in the user-parameter
       file: "subtractscatteredlight no"). This bug was introduced in revision
       2989, and was fixed with revision 3045.

2.2.5 (2013-11-06; Revision 3018) - Serenity

  o All tools (in ${p3d_path}):
        New instrument setups.
     - Added support for the AAT/AAOMega/SAMI instrument. The setup should work
       more or less starting with revision 2851 (adjusted the spectrum
       extraction with in revision 2912).

     - Added support for the VIRUS instrument. This new setup is only intended
       to be used as a testing tool for lab data. The setup has initially been
       configured for IFUs 11, 18, and 8 (as well as partly 3, 4, and 5), but
       support for all 65-80 IFUs is easily added.

  o All tools that combine raw-data images:
        New functionality.
     - The C-based module that combines raw images with the min/max filter
       method, p3d_cimcom.c, now works as expected also when there are more
       than 20 images. This is how it works from now on.

       When the user selects 10 images or fewer - this value is set with the
       user parameter imcomb_minmaxgroupn - the images are stacked and the two
       pixels with the lowest and the highest values are discarded. Another set
       with the lowest and the highest values are discarded if the number of
       images is higher than 10, but lower than or equal to 20. The total
       number of discarded pixels are as follows (assuming that
       imcomb_minmaxgroupn=10), as function of the number of combined images n:
       n<11, 2; 10=2780.

  o C routines:
     - Optimized the C routines p3d_cpixex.c and p3d_cmpd.c.

2.2.2 (2013-05-29; Revision 2771) - Serenity

  o All tools (in ${p3d_path}):
        Major new functionality.
     - All tools in p3d were so far accompanied by their individual binary file
       for use with the IDL Virtual Machine. This setup has now been completely
       changed. From now on there is only one binary file that contains all
       tools - "p3d_tool_server.sav". The binary file is launched as a server
       on any number of threads on the computer where p3d is used (default is
       all threads minus one).

       Thereafter, a new Python-based p3d queue handler manages all p3d IDL
       servers that were launched by the user, by submitting jobs to free
       servers. Jobs are dispatched by the new tool, which is
       also called by the now obsolete bash scripts in ${p3d_path}/vm/ (those
       scripts are still kept to allow already written scripts to be executed
       without any changes).

       An important feature is that jobs are submitted using either all threads
       or just one thread (the queue manages different jobs transparently). It
       is up to the user to select an appropriate number of threads. Any tool
       that uses multiple threads internally, such as profile fitting or
       spectrum extraction, should be launched using one thread. The older bash
       scripts have been modified such that,,
       and require all threads.

       Simultaneously the p3d compiling script, as well as the additional tools
       download script are replaced with Python versions of the same tools: is replaced with, and is replaced with

     - Activated the new parallelized C routine that cleans single images of
       cosmic-ray hits. For a test with one VIMOS image and 8 threads the
       C based cleaning took 16.6s, and with 1 thread 45.8s. The corresponding
       time using the IDL-based routine was 41.6s. The routine  - p3d_ccr.c -
       was mainly written by F. Tabataba-Vakili.

     - Activated the new parallelized C routine that combines images using the
       min/max method. For a test with three VIMOS images the IDL took 14.13s,
       the C routine with one thread 0.335s, and with four threads 0.115s, the
       corresponding speedup is thereby 40 and 120! The routine
       - p3d_cimcom.c - was written by F. Tabataba-Vakili.

     - Activated the new parallelized C routine that calculates pixel-based
       cross-dispersion profiles for the optimal extraction when using
       p3d_extract_optimal_mox and for viewing the profiles in
       p3d_extract_optimal (the entry in the remaining routine p3d_optimal_mpd
       was converted when that routine was made a C routine). The routine
       - p3d_cpixex.c - was written by F. Tabataba-Vakili.

        Minor bug fix.
     - All tools in p3d can be used with the new p3d queue server system. Up to
       now there was a bug when parsing filenames of keywords. This has been
       fixed with revision 2680, but it is still impossible to use filenames
       that contain only one starting slash; such as "/file.fits".

     - The p3d main GUI tool has been corrected to reload multi-extension
       dispersion-mask images properly (when creating a new dispersion-mask
       image). The tool now also behaves properly also when the external
       program ds9 is not in the path. Thanks to Pieter Gruyters for pointing
       at the issues that these two bugs led to. This was fixed with revision

     - The latest VIMOS setup, which is used since 2012.03.15 at the telescope,
       has been fixed (bvimos_hr.prm). The cropped detector-section that is
       used with HR-blue was used with all grisms, which is in error. This has
       been fixed so that HR-blue uses a cropped version, while the other
       grisms use the uncropped section. This was fixed with revision 2711.
       This issue was addressed by P. Gruyters.

  o The RSS to cube conversion routine (p3d_rss2cube):
        Fixed a minor bug.
     - The spatial element scale could not be specified using the keyword
       SPAXELSCALE, but instead had to be provided in the user-parameter file.
       This was corrected with revision 2667.

  o Dispersion-mask creation routine (p3d_cdmask):
        Fixed a minor bug.
     - In p3d_wavecal_dispmask_gui there is a statement that compares an
       integer with a string. Apparently this works great in IDL 7.1 x86_64,
       but not at all in IDL 7.1.1 x86_64. This has been corrected with
       revision 2685. This bug was spotted by C. Herenz.

     - p3d_wavecal_dispmask_add_spec has had a bug since revision 2402 (this
       was when the scaled polynomial fit of the wavelength array was added)
       that prevents calibration-fiber data from being added for instruments
       that use such; in this case FLAMES/Argus/IFU1/IFU2. This has been
       corrected with revision 2729. This bug was found by S. Kamann.

  o Spectrum extraction (p3d_cdmask, p3d_cflatf, p3d_cobjex):
     - Reducing VIMOS data and not having /NOOBCHECK set there was a bug that
       prevented the OB id number to be parsed in dispersion-mask images that
       use the new extension-based format. This was corrected with revision
       2761. This bug was found by A. Castrillon.

  o Flux calibration (p3d_fluxsens, p3d_fluxcal, p3d_rss):
 	Fixed a minor/major bug.
     - A sensitivity-function file can be saved using either p3d_fluxsens or
       the sensititivity-function tab in p3d_rss. In the latter case the
       wavelength-array keywords CRVAL, CDELT, and CRPIX, were so far not
       written for the first dimension (as they have to be for p3d_fluxcal to
       work). (Note! This only concerns instruments where the second axis is
       the dispersion axis.) This has been corrected with revision 2678.
       This bug was spotted by P. Gruyters.

  o Updated support for some instruments:
     - Integral (/INTEGRAL, /OLDINTEGRAL):
       On a request of M. de Bont and K. Exter, the instrument-keywords were
       adjusted so that keyword parameters containg a dash are read properly.
       This makes possible the creation of a sensitivity function for this
       instrument (the spectrum must still be corrected for flux lost between
       fibers, but it is a start).
       The old-Integral instrument-keywords file was updated in revision 2719,
       previous revisions (since a long time back) are unable to extract
       airmass values properly.
     - VIRUS-P (/VIRUS):
       After a request from E. Freeland (Stockholm Univ.) a dead-fibers file
       was added for fiber bundle 2. Note that the fiber status have evolved
       from the initial state (where everything worked) to now. Hence, it may
       be necessary to edit a copy of the new dead-fibers file for intermediate

  o The p3d main GUI (p3d_tool_gui):
         Bug fix!
     - Switching to the new extension-based format for dispersion-mask images,
       I had forgotten to fix the checking of which quadrant is used with
       VIMOS files. This was fixed with revision 2621.

  o Various minor bug fixes and adjustments to comments throughout the code.

2.2.1 (2013-01-21; Revision 2611) - Serenity

  o All tools (in ${p3d_path}):
        New instrument setup supported.
     - Added support for the latest VIMOS setup, which is used since 2012.03.15
       at the telescope. The new setup is used with the BVIMOS keyword, and the
       bvimos_hr.prm instrument-parameter file. Only half of the CCD is used
       with the new HR-blue configuration, compared to the older configuration.
       This support was added with revision 2599.

  o The dispersion-mask creation (p3d_wavecal_dispmask_gui):
        Corrected a major bug.
     - In previous updates to use fits-file extensions also with the
       dispersion-mask file a zero-extension header was forgotten in the
       first-extension header. It is impossible to save a dispersion mask with
       this bug. This has been corrected with revision 2605, and is the cause
       for this release.

  o Various minor bug fixes and adjustments to comments throughout the code.

2.2 (2012-12-13; Revision 2578) - Serenity
  o Flux calibration (p3d_fluxsens, p3d_fluxcal, p3d_rss):
        Now ready to be used.

     - With this release all flux-calibration routines, as well as related
       parts in the master user-parameter file, are thoroughly documented and
       tested. The flux calibration procedure is intended to be the following:
       at first, a sensitivity function is created using observed spectra of a
       standard star with the tool p3d_fluxsens. Thereafter extracted science
       spectra are flux calibrated using p3d_fluxcal.

       p3d_fluxsens is meant to be an automatic tool that avoids any user
       interaction, an interactive version is instead available in the spectrum
       viewer; all options of p3d_fluxsens are available through various
       widgets under the sensitivity-function tab. To mention some features,
       the sensitivity function is fitted to an observed (and sky subtracted)
       spectrum using one of eight kinds of interpolation: regular polynomial,
       Chebyshev polynomial, Legendre polynomial, three varieties of (cubic)
       splines, and two varieties of quadratic piecewise interpolation; the
       polynomial order can be chosen in the range 0-20.

       One feature is only available in the spectrum viewer, the masking of
       unwanted calibration standard-star data band-passes/entries. These
       entries are easily rejected by clicking on them one by one with the left
       mouse button, or dragging out a box with the left mouse button in one of
       the plot panels. Edited data can be saved to a plain-text file for
       continued use with other data or with p3d_fluxsens. Masked entries are
       unmasked in the reverse order they were masked by pressing the right
       mouse button repeatedly.

       The polynomial fits were implemented and checked to work with revision

  o The spectrum viewer (p3d_rss):
        Interesting new functionality
     - The spectrum viewer is able to read both RSS images and data cubes that
       were created using the CALIFA project data-reduction pipeline. To view
       such data files it is necessary to start the spectrum viewer from the
       command line. Here is an example:
 CALIFA_cube.fits.gz parfile=ppak4k.prm

       Data cubes are converted to an RSS image, by the p3d spectrum viewer,
       before the graphical user interface is launched.

  o The science-object extraction routine (p3d_cobjex):
        Corrected a major bug.

     - Corrected a serious bug that affects some reductions using re-alignment
       of the dispersion solution with telluric lines. Specifically, the bug
       affects anyone using a dispersion mask that is poorly determined for the
       lowermost pixels, which might be the case with data where there are no
       calibration arc lines in the lowermost part of the wavelength range. The
       correction was made to the routine p3d_wavecal_dispersion_correction.

       This bug fix involves two issues. Since very long it was forgotten to
       update the arrays with relative pixel sizes after offsets were applied.
       The second issue is the following. The resampling wavelength array was
       always calculated starting with the first pixel in the output array. If
       the dispersion mask was poorly determined at the lower wavelength range
       it was often impossible to create an accurate wavelength calibration, in
       particular when the dispersion solution was re-aligned using telluric
       lines. An example of data where this might happen are VIMOS/HR-orange
       data, where there are no arc lines shortwards of about 5500Å. Wavelength
       calibrated data using a dispersion mask that is poorly determined for
       lower wavelengths would result in the wavelength array being shifted one
       pixel at all wavelengths, of affected spectra.

       The resampling wavelength array is now calculated starting from the
       middle output pixel, by default. The starting pixel can be chosen with
       the keyword RESAMPLE_STARTPX or the user parameter 'resample_startpx'.
       This issue was completely corrected with revision 2349.

  o Accuracy of dispersion masks (p3d_cdmask):
        Major modification - significantly improved the accuracy.

     - Polynomial fits have been drastically improved, by rescaling abscissa
       arrays to always use the range [-1,1]. This way it is possible to use
       high-order regular polynomials with good results; when the order is
       about <15. For dispersion masks this adjustment has required saving of
       additional data, with the lower and the upper pixel values that were
       used in the scaling, in the dispersion-mask file. All tools that read
       such dispersion-mask files automatically recognize file formats using
       both the new format and the older format (which is unable to perform the
       rescaling due to the missing data). The polynomial fits were implemented
       and checked to work with revision 2410. The spectrum-viewer buttons in
       the main GAUI panel were updated to work with revision 2478.

  o All tools (in ${p3d_path}):
        Major modifications.

     - Added support for the instruments VIRUS-W and Integral (new CCD, all
       four IFUs: SB1-SB3 and equalized). These instruments use files where
       data of several blocks are contained in the same file.

     - Cosmic-ray cleaning: Added the keyword GAUSSKERNELSIZE and the user
       parameter 'crgausskernelsize' to the tools p3d_ctrace, p3d_cdmask,
       p3d_cflatf, p3d_cobjex, and p3d_cr.

       Also added some instructions on how to make the cosmic-ray cleaning
       mimic the results achieved in Husemann et al. (2012; PyCosmic).
       Basically, one must set the keyword GROWRADIUS to zero.

     - Repaired the support for the FLAMES mini IFUs (IFU1 and IFU2). Due to an
       improvement in revision 2095 to handle dead fibers correctly the routine
       was unable to trace spectra at all. The "improvement" has been commented
       out in revision 2508, and the routine works as it should again.

     - Up to now (nearly) all output data were written to separate files. Now
       the keyword ALLINONE, as well as the user parameter "allinone", has been
       added to all tools. The default value of this keyword and user parameter
       is that allinone is set. This means that (nearly) all output data are
       written to the same file, using consecutive extensions. To have the same
       behavior as before it is necessary to unset this keyword.

       All tools of p3d handle files using both approaches, in any mixture.
       For example, if parts of the files where created with ALLINONE unset, it
       is possible to continue with ALLINONE set.

       The names of the different extensions are explained in the master
       user-parameter file.

       The purpose of this change was to allow the resulting data cube files to
       be used with the CASA viewer (which requires additional header
       keywords), but also have a smaller number of resulting output files. All
       written fits files are now (meant to) conform to the FITS standard.

       Multi-extension fits files are easily examined using the program ds9
       using commands such as the following examples:
         ds9 file.fits[DATA] ::
           To view data in the main extension.
         ds9 file.fits.gz[ERROR] ::
           To view data in the error extension.
         ds9 file.fits.gz[DATA] file.fits.gz[MASK] ::
           To open the main data and the mask image in separate frames.

       This new mode was confirmed to work fully with revision 2478.

  o The use of prescan and overscan regions (p3d_c?????):
        Corrected a serious bug (that should affect very few)
     - Prescan regions of PMAS/4kx4k data were never normalized to use the mean
       gain, as the rest of the data are, including overscan data. Corrected
       this bug by adding a correction factor to the block combined image, all
       reductions making use of the prescan regions must be redone. This was
       fixed with revision 2465.

     - Added the possibility to specify bias values explicitly using the
       BIASCONSTANT keyword, or the 'biasconstant' user parameter. This support
       was added and checked to work with revision 2559.

  o The master-bias tool (p3d_cmbias):
        Minor new functionality
     - Added the keyword NODIALOG, and the new user parameter 'biasnodialog',
       which avoids the widget dialog that asks if a single image should be
       smoothed. By default NODIALOG is set, and no smoothing is performed.
       This support was included with revision 2462.

  o Bias-subtracted images (p3d_ctrace, p3d_cdmask, p3d_cflatf, p3d_cobjex):
        Minor new functionality
     - Added the keyword SAVEBIASSUB, and the new user parameter 'savebiassub',
       which save the (optionally combined) bias-subtracted raw-data image to a
       separate file. This could be a useful option if one wants to check the
       quality of the bias subtraction, in particular when using a bias image
       that was created using prescan and overscan data. This support was
       included with revision 2482.

  o The RSS to cube conversion routine (p3d_rss2cube):
        Fixed a minor bug.
     - Corrected a bug that affects output saturated-pixels data cubes. So far
       such data cubes were never created, instead the bad-pixels mask image
       was written again, if a bad-pixels mask image was present.

  o The p3d development version repository.
        Changed location
     - Until 7.12.2012 the p3d repository address was:
       After 7.12.2012 the repository address was changed due to software
       upgrade at The new addresses are:
       Directories pointing at the old directory can be used for as long as
       there is no need or wish to update p3d. The new repository is
       incompatible with the old repository, which is why the old one must be
       replaced with a copy using one of the two new repository addresses
       before p3d can be updated anew.

       p3d developers have two choices to access the repository: either using
       the svn+ssh protocol or the https protocol. The exact command lines are
       provided at the p3d web site when one is logged in (with repository
       write access).

  o Various minor bug fixes and adjustments to comments throughout the code.

2.1.4 (2012-09-17; Revision 2250) - The Warbird
  o The spectrum viewer (p3d_rss):
        Major new functionality
     - Spatial elements used to be selectable (deselectable) only by clicking
       on them one by one with the left (right) mouse button, or selecting
       either all or no elements. With version 2.1.2 it became possible to also
       select spectra of dead, low-transmission, and sky fibers.

       Now it is in addition possible to select spectra (spatial elements)
       using a widget tool that defines one or several regions of interest
       (ROIs). These ROIs are defined interactively using either rectangles,
       ellipses, polygons, or freehand drawn curves. The spatial elements that
       fall roughly within all setup regions are selected upon closing the ROI
       tool. The same regions are used anew whenever the ROI tool is opened
       again in the same session. It is also possible to save the ROIs inside
       the ROI tool, and later load then from the spectrum viewer for reuse.

       Additionally, it is possible to save and load created spatial-element
       bin map files. These map files are plain-text files that contain as many
       bins as required; each bin is assigned a positive integer, which is
       greater than zero. The spectrum viewer allows viewing and creating one
       such bin at a time. The bin number is changed using a droplist widget.

  o Regarding the instrument support for PMAS/4kx4k CCD (both PPAK and Larr):
     - When files are downloaded from the Calar-Alto archive they have
       filenames with a date and time string in them. This time string does not
       work properly with previous versions of p3d, since p3d cannot determine
       the names of all files of all blocks automatically. This has now been
       corrected with revision 2196, which uses the original filename that is
       available in the header-keyword keyword ORIG_FILENAME FILENAME
       (of PMAS/4kx4k CCD files).

  o The dispersion-mask creation (p3d_wavecal_dispmask_gui):
     - At some point the feature was deactivated, which displays a help for
       each button in the tool. It has now been reactivated again, with
       revision 2185.

  o The use of prescan and overscan regions (p3d_cr):
     - There has been a small bug that would prevent prescan and overscan
       bias images from being used with images that are to be cleaned of
       cosmic rays. This was fixed with revision 2230.

  o Combining multiple files with extracted data (p3d_cexposure):
        Corrected a serious bug.
     - An error was included with the new feature using bad-pixels mask images
       of revision 2126 (which is included in release 2.1.3). The error of the
       first file was never squared, which results in final errors that should
       be squared to be more correct. This has been corrected with revision

  o Various minor bug fixes and adjustments to comments throughout the code.

2.1.3 (2012-06-28; Revision 2134) - The Warbird

  o The spectrum extraction
    (p3d_cdmask, p3d_cflatf, p3d_cobjex, and p3d_cexposure):
        New functionality
     - An output bad-pixels mask image is written if a bad-pixels mask image or
       a cosmic-ray mask image are provided, or if the cosmic-ray mask image is
       calculated. Each spectrum element in the output mask contains the total
       number of masked pixels in the bad-pixels mask image. The output mask
       images are used by p3d_cexposure when extracted images are combined.
     - The resampling of extracted spectra from pixel to wavelength units is
       now done more carefully to completely avoid Not-a-Numbers in the output
       spectrum image.

  o Bias subtraction:
        Minor new functionality
     - New-generation-controller instruments (ERA2) can now also use prescan or
       overscan regions to calculate a bias image.

  o Saving data in the e3d format:
        New functionality
     - All routines writing science-object row-stacked spectrum (RSS) files now
       also write e3d-formatted files. This option is activated either using
       the keyword SAVEE3D or the user parameter savee3d. The following four
       routines can now do this, in addition to p3d_cobjex: p3d_fluxcal,
       p3d_cvimos_combine, p3d_darc, and p3d_cexposure.

  o Removing of cosmic-ray hits in single raw-data images (p3d_cr):
        Minor new functionality
     - Added the new keyword and user parameter GROWRADIUS, which allows a
       definition of the width of the frame of pixels around each pixel that is
       masked as a cosmic-ray hit. Additional cosmic-ray hits are identified in
       these neighbor pixels. In this case the first criterion is used as
       S'>sigfrac*sigclip. The growing radius has previously always been set to
       1, but can now also be set to 0 and 2. The default value is 1.

  o The spectrum viewer (p3d_rss):
        Minor new functionality
     - The (approximate) spectrum-viewer window size can now be specified using
       the keyword WINXYSIZE. Allowed values are:
       The p3d main GUI provides a set of pre-defined values in the Options
       menu. Only those options are available that are smaller than or equal to
       the screen size.
     - Since revision 1982 it is no longer necessary to specify the IFU angle
       header keyword (KWANGLE) when the PARFILE keyword is set. The correct
       parameter name is now read automatically from the instrument-specific
       keywords file. Note that this new feature only affects sessions that are
       not based on use of the p3d main GUI.
       Note! For instruments that do not use this option it is necessary to
         update to revision 2116, since previous revisions contain a bug that
         does not allow the spectrum viewer to be opened.

  o The fiber position tables for FLAMES/ARGUS, FLAMES/IFU1, and FLAMES/IFU2
    now also contain positions for the sky fibers. These sky fibers can be
    viewed in the spectrum viewer when the keyword ANDSKY is set, or the GUI
    menu option "Spectrum viewer - view sky fibers" is set to "yes". The sky
    fiber positions do not correspond to the actual position on the sky.

  o Various minor bug fixes and adjustments to comments throughout the code.

2.1.2 (2012-04-23; Revision 1948) - The Warbird
  o All tools (in ${p3d_path}):
        Improved the documentation, minor and important new functionality.

     - Added support for the new instrument ERA2. This instrument uses files
       where data of several blocks are contained in the same file. It has
       been necesary to adjust p3d quite significantly to support this format.
       Hence, some problems might occur elsewhere.

     - Added support to use prescan and overscan regions to create a bias
       when no master-bias image is specified. The region to use is simply
       specified using the following parameters:

       biaspx :: uses the x-axis prescan region. Specifically, a median value
                 is calculated for each pixel on the y axis in the x-axis
                 prescan region. The same array is used with all y-axis columns
                 of the data that are bias subtracted.

       biaspy :: uses the y-axis prescan region. Specifically, a median value
                 is calculated for each pixel on the x axis in the y-axis
                 prescan region. The same array is used with all x-axis rows
                 of the data that are bias subtracted.

       biasox :: uses the x-axis overscan region. Specifically, a median value
                 is calculated for each pixel on the y axis in the x-axis
                 overscan region. The same array is used with all y-axis
                 columns of the data that are bias subtracted.

       biasoy :: uses the y-axis overscan region. Specifically, a median value
                 is calculated for each pixel on the x axis in the y-axis
                 overscan region. The same array is used with all x-axis rows
                 of the data that are bias subtracted.

       biasconstant :: Set this keyword to use a constant value as a bias
                 value; otherwise an array is used. All the regions that are
                 specified using the keywords - or user parameters - biaspx,
                 biaspy, biasox, and biasoy can be used when this keyword,
                 or user parameter, is set; otherwise only one of them.

       Default values are specified in the instrument-parameter files of the
       following instruments: PMAS/Larr, PMAS/PPAK, VIMOS, FLAMES, GMOS. For
       as long as the raw data contains either region any region can be used.
       The use is specified either using a keyword in p3d_cr, p3d_ctrace,
       p3d_cdmask, p3d_cflatf, and p3d_cobjex, or using the user-parameter

       Note! These regions are only used when no master-bias image is

       This functionality is so far preliminary, and not fully documented. Full
       support will be included starting release 2.2.

     - The routine documentation has been significantly revised for clarity.
       p3d_cobjex: improved the documentation so that it more accurately
         describes how the science-object spectrum extraction works, including
         information on how to use the instrument-flexure correction, as well
         as scattered-light subtraction features.
       p3d_cflatf: improved the documentation so that it more accurately
         describes how flat-field data are treated, including a section on how
         to combine a lamp flat-field image with a twilight flat-field image.
       p3d_rss: Cleaned up the documentation, removing a part belonging to

     - The instrument-parameter file PARFILE can now be specified without the
       full path as a prefix. I.e., instead of
       it is now possible to write
       the file is searched for automatically in the p3d data subdirectory.

  o Flux calibration (p3d_fluxsens, p3d_fluxcal):
        New functionality.
     - Added first versions of routines that allow flux calibration. These
       routines are only preliminary in this release. More complete
       functionality, and documentation, will be available in the future
       release 2.2.

  o The science-object extraction routine (p3d_cobjex):
        Added some new functionality.

     - Added the option to save the final normalized flat-field image. This
       image is now stored when then keyword SAVEFINALFLAT is set, or the
       user parameter ff_save_final_image is set to 'yes'.

     - The flat-field transmission file of an extracted twilight flat-field
       image is now properly used to normalize data for the fiber-to-fiber
       transmission. To use this feature: first set the TRANSMISSION keyword
       before running the p3d_cflatf tool (also see the new keywords PCUTLOW
       and PCUTHIGH), and then extract the data using p3d_cobjex, setting the
       NF2F keyword.

     - Added the option that the wavelength arrays of individual spectra can be
       corrected individually, when using one or several sky-emission lines.
       Previously all wavelength arrays of the dispersion mask were always
       offset using one median value calculated for all used sky-emission lines
       and spectra. This approach has been found to be particularly poorly
       suited with the multi-detector instrument VIMOS.

       Note that the default method now is to use an individual correction.

       A linear fit is now also made of the pixel offset between two or more
       sky-emission lines. It is up to the user to select appropriately spaced

       Additionally, the wavelength offset is weighted with the pixel size,
       since the offset due to flexure takes place in pixel space, and not in
       wavelength space. This treatment allows a more correct offset. See the
       keyword ONESKYOFFSET, as well as the user parameter 'oneskyoffset', for
       more information.

     - During the dispersion correction the errors are now ensured to be
       non-zero - there are no more NaNs in the resulting output images.

  o The flat-field preparation routine (p3d_cflatf):
        minor new functionality.

     - Improved the documentation so that it more accurately describes how
       flat-field data are treated, including a section on how to combine a
       lamp flat-field image with a twilight flat-field image.

     - Updated the handling of the fiber-to-fiber transmission file, so that it
       will work to correct data using, for example, both a lamp flat and a
       twilight flat.

     - Added the keywords PCUTLOW and PCUTHIGH which can be set to use only a
       part of the CCD before calculating the fiber-to-fiber transmission
       array. It might be useful to change one of these values if the CCD
       sensitivity is lower on one side. For example, with the VIMOS HR-blue
       grating PCUTLOW should be set to something like PCUTLOW=32.0.

  o The dispersion-mask creation tool (p3d_cdmask):
        New functionality.
     - Added the keyword LINELISTLIMIT, as well as the user parameter
       'linelistlimit_dm', which allows an automatic selection of only those
       entries that are brighter than the selected value. This option is only
       available if the arc-line list file contains at least two columns, where
       a relative intensity is specified in the second column (as a decimal
       value). All entries are used, which do not contain a second column.

  o The spectrum viewer (p3d_rss):
        New functionality.
     - Added the option to save either the currently selected spectrum (as
       before) or both the currently selected spectrum and the subtracted
       spectrum to the same file. Both options are available in the File menu.
     - Added the option to divide all spectra with the selected spectrum, to
       the previous option of subtracting the selected image from all spectra.

     - Hexagon-shaped elements are now shown and selected properly in the
       spatial map. Such elements are used with the GMOS-S and the GMOS-N IFUs.

     - If the new keyword PARFILE is specified it is no more necessary to
       specify the argument POSFILE, or the dispersion axis DAXIS.

     - In addition to selecting " all" and "none" of the spatial elements in
       the spatial map view, it is now possible to also select "a- t" (all but
       low-transmission elements), "   t" (low-transmission elements), "a-dt"
       (all but dead and low-transmission elements), "a-d " (all but dead
       elements), "  d " (dead elements), "  dL" (dead and low-transmission
       elements), "a-s " (all but sky elements), and " sky" (all sky elements).
       Of course, these changes only apply to the instruments which have such
       low-transmission, dead, or sky fibers. It is, additionally required to
       specify the PARFILE argument when starting p3d_rss.

  o Various minor bug fixes and adjustments to comments throughout the code.

2.1.1 (2012-02-06; Revision 1615) - The Warbird
  o Combining multiple files with extracted data (p3d_cexposure):
        New functionality.

     - Added a new tool that combines mutliple extracted exposures that are
       taken illuminating the exact same field on the sky. The supplied data
       are at first checked for consistency in image size and equal values on
       the wavelength calibration parameters CRVAL and CDELT. Thereafter the
       files are either averaged, weighting with the exposure time of the
       individual exposures. Alternatively, setting the /SATURATED keyword it
       is expected that all files have a different exposure time. In this case
       the longest exposure is used as reference. Portions of individual
       spectra, where a mask indicates that a pixel was saturated, are then
       replaced with scaled data of the same spectrum and portion of a file
       using a shorter exposure time. The routine accepts any number of files.
       The routine produces one output file with merged data, and a file with 
       corresponding errors if all the error files of the input images are

  o Handling of saturated pixels in the raw data:
        New functionality.

     - From now on a new mask file is written along with extracted data
       whenever the keyword SATMASK is set in the routines p3d_cdmask,
       p3d_cflatf, and p3d_cobjex. This mask file indicates, for each element
       in the spectra, if any of the used raw-data pixels were saturated. Only
       the center part of the cross-dispersion profile is considered for
       optimally extracted spectra. This functionality will be used to combine
       extracted spectra that use different exposure times.

  o The science-object extraction procedure (p3d_cobjex):
        Minor bugs affecting the functionality.

     - A cosmic-ray mask file that was provided as an input keyword (CRMASK)
       was never identified.

     - If a run would contain several groups, where some would require the
       combination of images, and some others not, while /crclean was set, then
       the combined images would not be properly reduced.

  o The dispersion-mask creation procedure (p3d_cdmask):
        Major bug affecting PMAS 4kx4k reductions.

     - A new method to handle arc-line entries in vignetted regions of the
       PMAS 4kx4k CCD, both Larr and PPAK, was introduced in version 2.1.
       However, the initial version is completely filled with bugs. This has
       been corrected with revision 1557, also the log-file output has been
       improved. Additionally, the vignetted arc-line entries (wavelengths) are
       indicated in the "Check the final fits" tab, for the affected spectra.

       Note! To continue to use version 2.0, when creating a dispersion-mask,
             it is necessary to comment the "vignettedfile_dm" entries in the
             instrument-parameter files larr4k.prm and ppak4k.prm.

        Minor issue affecting any data with calibration spectra.

     - Corrected the display of the affected spectrum (number) in the
       "Check the final fit" tab. The number is now showing the correct
       spectrum also for data that contain calibration spectra.

  o Compatibility with PINGSoft:
        Added functionality

     - Added PINGSoft-formatted fiber-position table files (to
       ${p3d_path}/data/tables/pingsoft_postables/), which can be used with the
       reduced outcome of p3d to analyze the data using the IFS analysis tool
       These new tables were introduced with revision 1602.

  o Reduction work with GMOS:
        Minor issue affecting cosmic-ray cleaning with GMOS data

     - Fixed the cosmic-ray cleaning routine so that it works with the
       multi-extension data of GMOS. It did not until now. This was fixed with
       revision 1577.

  o Reduction work with MPFS:
        Major issue affecting any accurate spectrum extraction

     - Updated the instrument parameters so that it is possible to calculate
       accurate cross-dispersion line profiles for optimal extraction by
       default. This was made with revision 1584.

  o Reduction work with VIRUS-P:
        Major bug affecting VIRUS-P reductions.

     - Fixed a bug that concerned how the parsing of the CCD binning parameter
       CCDSUM. The bug caused all VIRUS-P reductions to not work, since quite a
       while. This was fixed with revision 1572.

  o The (automatic) feature to compile the C-language source files...
        Minor bug affecting those with a src/ directory unavailable for writing

     - Corrected the minor bug (a forgotten path separator) so that it works

  o Various minor bug fixes and adjustments to comments throughout the code.

2.1 (2012-01-16; Revision 1529) - The Warbird
  o Correction for differential atmospheric refraction (p3d_darc):

     - Added a new tool that corrects extracted science images for differential
       atmospheric refraction. The new tool works out-of-the-box with all the
       IFUs that use square-shaped spatial elements (VIMOS, FLAMES, PMAS/Larr,
       MPFS, and SPIRAL). In some cases the required physical properties are
       unavailable in the data header. In that case they must be entered by

     - The routine allows the user to choose between three implemented
       approaches to calculate the offsets. The results of all three approaches
       are collected in a plot, which allows the user to choose the most
       accurate method as he or she believes is more correct.

     - The offsets are calculated using the wavelength-dependent
       refractive-index equation of Ciddor (1996; as well as Edlén 1966).

     - A second approach uses a height-integrating approach to calculate the
       refractive-index that comes from the Starlink project. The routine
       integrates the refractive index through the trophosphere and the

     - A third method uses an empirical fit of a two-dimensional Gaussian
       profile to the data itself to calculate the DAR-caused offsets on both
       the X and the Y axis.

     - Most of the features of the DAR-correction routine of J.Walsh (ESO) have
       been included with his permission.

     - Please note that the calculated errors are off as correlated errors are
       completely ignored (at the moment).

  o Removing of cosmic-ray hits in single raw-data images (p3d_cr):
     - Added a new tool that cleans single raw-data images of cosmic-ray hits.
       The new tool works with all raw data that are used with p3d.

     - The tool uses an IFS-adapted version of the L.A. Cosmic IDL routine of
       J.Bloom (Berkeley); this adaptation has been made with the permission of
       both P.van Dokkum (Yale) and J.Bloom. The modifications have been
       implemented partly according to the suggestions of B.Husemann (AIP).

     - By default the routine only produces a cleaned image and a cosmic-ray
       hit mask. However, if the keyword WRITEALL is set then intermediate
       products are also written to the disk at each iteration; these products
       allow a close inspection of the quality of the cleaning procedure, and
       may also require huge amounts of disk space.

     - The cosmic-ray mask is used as a bad-pixel mask during (optimal)
       spectrum extraction. Thereby the, so-called, weight of masked pixels are
       set to zero - i.e. they are excluded from the calculations.

     - The routine can be called separately, but is also available from
       p3d_ctrace, p3d_cdmask, p3d_cflatf, and p3d_cobjex, when the new keyword
       /CRCLEAN is set.

  o The spectrum extraction:
     - The resampling of the extracted spectra from pixel to wavelength units
       can now be done using either a one-dimensional drizzling algorithm, or
       the original linear-interpolation approach. The error calculation of the
       original linear-interpolation method has been improved. The older less
       accurate errors are still available for comparison; through the use of
       the ORIGINALERRORS keyword.
     - The multi-profile deconvolution method that uses a band-diagonal matrix
       has been fully ported to C for fast and parallelized execution. The new
       routine, which has been written by F. Tabataba-Vakili at AIP, compiles
       and executes automatically; together with the other C routines.

  o The RSS to cube converter (p3d_rss2cube):
     - Added a new tool that converts the extracted images that p3d writes into
       cubes, which may be easier to work with. This routine is only available
       as a stand-alone tool, and also only works with the IFUs that use
       square-shaped elements, i.e. PMAS/Larr, SPIRAL, VIMOS, FLAMES, and MPFS.

  o The scattered-light subtraction:
     - The scattered-light subtraction can now be switched on and off using a
       keyword to the call of p3d_cflatf and p3d_cobjex.

  o Regarding the instrument support for VIMOS and FLAMES:
     - Due to an extremely serious bug in the routine that reads the gain and
       the readout noise values from the data header, all errors calculated for
       these two instruments so far have been wrong - they have been too small.

  o Regarding the instrument support for VIMOS:
     - Corrected a bug that would prevent a proper extraction of data that were
       sampled using the medium resolution setup (the DETSEC parameter was
       improperly read from the instrument-parameter file).      

  o Regarding the instrument support for PMAS/4kx4k CCD (both PPAK and Larr):
     - The new PMAS 4kx4k CCD has a region in all four corners of the CCD that
       are vignetted. When creating a dispersion-mask image, any arc line
       inside of these vignetted regions are now automatically excluded from
       the chosen selection of line-list entries before the arc-line positions
       are fitted with a polynomial. This way it is thought that the wavelength
       calibration can be made more accurate in affected spectra. Note that a
       exact definition of the vignetted region is difficult to make. If you
       think that you have a better solution on how to define the extent of
       the vignetted regions, then please, have a look at the two files
       ppak4k_vignet.dat and larr4k_vignet.dat.

  o The spectrum viewer (p3d_rss):
     - The spatial maps are now saved as two- or three-dimensional image arrays
       instead as of an unordered one-dimensional array (which is how has been
       until now). In the new file each pixel corresponds to a spatial element.
       This feature is now only available on instruments with square-shaped
       spatial elements, i.e. PMAS/Larr, SPIRAL, VIMOS, FLAMES, and MPFS.

  o The instrument- and user-parameter file parser (p3d_misc_read_params):
     - The routine did not return the correct flag if a parameter was only
       found in the user-parameter file and not in the instrument-parameter
       file. For example, if crval is only set in the user-parameter file then
       it would always be reported as not found. This has been corrected with
       revision 1414.

  o Added the (automatic) feature to compile the C-language source files in a
    temporary directory on the local computer. Up to now the compilation was
    always done in ${p3d_path}/src/, which would not work on a central

  o Various minor bug fixes and adjustments to comments throughout the code.

2.0.3 (2011-10-13; Revision 1181) - The Swashbuckler
  o Corrected a problem that occasionally would prevent the spectra
    from being properly traced. The story is:
      The first part of the tracing procedure locates peaks in the
      data. The routine (p3d_tracing_findspec0) tries to find a
      resonable value on what is a minimum value in non-peak areas.
      It could fail if the bias level varies across the image, as
      it does with the PMAS 4kx4k CCD. Consequently the master-bias
      image is now subtracted from the continuum image, before the
      latter one is used to trace the spectra. This solution seems
      to work.
      This is a problem that is more likely to occur with a
      multi-file instrument such as PMAS using the 4kx4k CCD;
      because, the bias level differs across the files.
    The problem was reported on Oct 13 2011, by Manu Nuñez.

  o The VIMOS extracted-images combination routine; p3d_cvimos_combine:
      Corrected a serious bug that would prevent the correct use of an input
      renormalization correction-array file (the keyword CORRFFILE). Also
      replaced the calculation of a mean telluric-line flux with a median.
      Thereafter the fluxes are normalized again using the maximum
      median-value quadrant.

2.0.2 (2011-09-28; Revision 1150) - The Swashbuckler
  o Regarding the instrument support for VIMOS:
     - Added a new feature to the combination routine (p3d_cvimos_combine).
       Flux-calibrated data are now re-normalized using a telluric line. This
       final normalization corrects for smaller inaccuracies in the data that
       are left after the flux calibration (which is performed using, e.g.,
       IRAF). This new re-normalization can be switched off, and the telluric
       line that is used can be chosen freely (or automatically). The
       re-normalization can be done both per spectrum and per quadrant.
     - The telluric-line fits are now illustrated in a plot that includes
       the raw data and the fit. One plot is created for each spectrum and
       each detector. The file is compressed automatically to save space.

  o The science-object extraction (p3d_cobjex):
     - The telluric-line fits of the sky alignment procedure, which is applied
       with the application of the dispersion solution, are now illustrated in
       a plot that includes the raw data and the respective fit. One plot is
       created for each spectrum and each detector. The file is compressed
       automatically to save space.

  o The spectrum viewer (p3d_rss):
     - Added a colorbar for the spectrum image and the spatial image.
     - Corrected a format mix-up when saving a mean or total spectrum. Also
       corrected a serious bug affecting the error calculations of the
       "sky"-subtracted spectrum.
     - Corrected the annotation of the "viewed" wavelength in the spectrum
     - The wavelength-range zoom value and the wavelength-range shift value
       are now reset to the initial values when the 'Reset' button is pressed.

  o Corrected a bug in the object-extraction routine p3d_cobjex, which could
       appear if the GROUP keyword was used together with flat fielding. The
       result could be seen as (kind of) randomly rearranged spatial maps
       (fixed with revision 1054).

  o Corrected the World Coordinate System header entries for the
       cross-dispersion axis. Now the data are properly labeled in ds9. /PW

  o Corrected the e3d output; due to an (old) -1 image-row offset the fiber
       position table inthe e3d file was always wrong (corrected with
       revision 1048).

  o Cleaned up all routines of the trace/disp.mask/object shift reference
       files. These were never used, and are not required anymore
       (fixed with revision 1056).

  o Various minor bug fixes and adjustments to comments throughout the code.

2.0.1 (2011-08-18; Revision 1028) - The Swashbuckler
  o After creating a trace mask, fitting line profiles (also the trace-mask
       creation routine), or extracting spectra (using any of the optimal
       extraction methods), p3d now -always- creates a postscript printout.
       This printout can be used to check if the (respective) fit was
       satisfactory. The printout paper-size (US letter/a4) and the number of
       wavelength bins that are shown can be set in the user parameter file.
       The output file is compressed using bzip2, if this command exists in the
       path; if not then the file is left as it is.

  o Corrected a bug in the C routine p3d_cssl_median.c. The third dimension of
       the input arrays arr and data was improperly accessed. This bug was
       spotted by F. Tabataba-Vakili.

  o Corrected an issue in, which could, eventually, result in
       an all-zeros flat-fielded object spectrum image.

2.0 (2011-06-17; Revision 1011) - The Swashbuckler
  o Regarding the instrument support:
       Corrected the tracing procedure for the mini-IFUs. Francesco D'Eugenio
       informed us of this inconsistency. Fibers 192, 229, and 278 (IFU1) and
       192 and 303 (IFU2) are dead, as well as the last (off-chip) fibers. This
       was corrected with revision 877 (tracings using all previous revisions
       are consequently wrong).

  o Recentering the calculated cross-dispersion line profiles on object data:
     - Due to flexure data of most instruments are shifted on the CCD as the
       time progresses (as a function on where on the sky the telescope is
       pointing). Starting with Revision 897 p3d now attempts to correct for
       such offsets in the cross-dispersion direction, whenever optimal
       extraction is used and the parameter recenterprofiles is set to yes.
       These offsets are normally very small, but even an offset of 0.10 pixel
       can have a strong influence on the extracted intensities; the influence
       becomes stronger the closer the spectra are arranged on the CCD. There
       are three methods available to calculate these offsets: either the data
       are smoothed (median-filtered) on the dispersion axis before the line
       profiles are fitted anew (for one or a few pixels on the dispersion
       axis), or the routine first searches for the brightest telluric line in
       the data, before that line is fitted (this approach is only available if
       a dispersion-mask image is provided as input to p3d_cobjex), or an
       offset is provided explicitly. The recentering can be configured either
       through the user parameter file (all settings), or through the RECENTER
       keyword to p3d_cobjex (only the basic setup - which normally is enough).

  o The fiber-flat field normalization (p3d_flatfield).
     - Starting with revision 966 any dead, unused, or low-transmission fibers
       are ignored when calculating a mean spectrum, which is used to
       normalize the flat-field data using /NDISP.

  o The dispersion-mask creation (p3d_cdmask and p3d_wavecal_dispmask_gui):
     - Up to now it was always necessary to create a dispersion-mask image from
       scratch interactively, which could be a quite time-consuming process if
       there are many dispersion masks to create. Starting with revision 892 it
       is now possible to specify a starting dispersion-mask image as an input.
       Using such a starting image, which -must- use the same observing
       conditions as the new dispersion-mask image, the creation process is
       very fast!
     - An arc-line list file can now be specified in the p3d main GUI, before
       it launches p3d_cdmask.
     - Starting with revision 1004 the used arc-line list is stored in the
       header of the dispersion-mask image. Because of this it is not
       necessary, anymore, to (first store and then remember to) provide
       p3d_cdmask with the same arc-line list file that was used to create the
       DISPMASKIN dispersion-mask image - the data is loaded automatically from
       the header.

  o The vimos extracted spectra combination program:
     - The previously very preliminary version of the VIMOS extracted-objects
       image combination program is now debugged, fully documented, it writes
       log messages to both the status line and the log file. The routine,
       p3d_cvimos_combine, can be called from both the IDL command line, the
       shell prompt (using, and from the GUI (using
       the lowermost righthand-side tab in the VIMOS configurations).

  o The spectrum viewer:
     - Added error handling to all (mean) spectrum-writing commands. The mean
       spectrum file now also contains the extended header keyword, to make the
       file fully consistent with the FITS format.

  o The log files:
     - Starting with revision 909, added the file revision number to the
       routine-specific identification string "rname". This addition makes it
       possible to see which routine version was used to produced the outcome.

  o The scattered-light subtraction:
     - Starting with revision 963 the calculation of a median-filtered image,
       using all parts of the image where there are no spectra, is now carried
       out by a C routine. The C routine is orders of magnitude faster than the
       IDL routine, and also makes use of multi-processor platforms (through
       OpenMP). The routine is compiled automatically by p3d.

  o Corrected numerous smaller bugs.

2.0b (2011-05-03; Revision 873) - The Swashbuckler
  o Modified the implementation:
       Data using the high (medium) resolution grism (HR [MR]) are now selected
       using the /VIMOS and /NVIMOS (/MVIMOS and /MNVIMOS) keywords. Data from
       before (after) the instrument reburbish in May-August 2010 must now use
       the /VIMOS and /MVIMOS (/NVIMOS and /MNVIMOS) keywords.

  o The IDL VM scripts
    Made some minor corrections to how the code is executed using the IDL VM.
    The routine scripts (p3d_c* would have returned an error regarding
    a missing environmental variable "path", which is never set, or used, in
    these routines. Removed the statement "cd $path". The script ""
    has prevented a correct use of the data output directory. The script no
    more changes directory to ${p3d_path} before launching the p3d GUI; this
    allows a correct use of the /CWD keyword.

     - These scripts contain a large number of additional improvements.

  o The spectrum extraction
    Added the option to subtract a scattered light component, which could just
    as well be a dark current component. This mode is activated by setting the
    darkscattsubtract parameter to yes in the user parameter file. The
    subtraction works by fitting a two-dimensional image to the
    bias-subtracted image where the spectra should be extracted from. The fit
    is only made on the unmasked pixels; all pixels nearby to spectra are
    masked. All properties of the fit are controlled with the options in the
    (master) user parameter file.

  o The cross-dispersion profile fitting routine
    Has now been ported to C. Instead of the IDL version of MPFIT the new
    routine uses the C version, which is named cmpfit. The necessary source
    code is retrieved automatically with the script
    The C source code is compiled when it is used, completely transparently
    to the user (using the IDL routine MAKE_DLL). If you see that all your
    cores, on your multi-core computer, are not used, then check if your
    compiler supports OpenMP. The routine has been found to compile
    correctly on linux and macosx, but should also work as it is on solaris-
    based platforms. We still need to set up the proper compiler options for
    windows-based platforms.

    If the compilation fails, for whatever reason, then the original IDL
    routine is used instead.

  o The dispersion-mask creation GUI
     - Dead or unused fibers are no more fitted in the final step, when the
       individual arc-line pixel positions are fitted to the raw data. This way
       the probability that any arc-line in a spectrum after, or before, the
       current spectrum will be incorrectly fitted is minimized.
     - Corrected the behavior when reading a arc-line file using the File menu
       option. Using the same approach as when the routine first starts, the
       read line list is now keeping arc-line entries outside the current
       wavelength range.

  o Various smaller improvements:
     - The default plot colors are now black lines on a white background. It
       used to be the opposite.
     - Also the spectrum-gap files (gapfile) can be put in the directory of the
       user parameter file now.

  o Corrected various bugs:
     - Exporting the extracted science spectra in Euro3D format is now working.

     - Corrected a bug in the spectrum viewer, where the summed spectrum was
       saved instead of the average spectrum, even though an average spectrum
       was selected.

2.0a (2011-01-27; Revision 661) - The Swashbuckler
  o Documentation
    All main data-reduction routines are now documented; this includes the six
    files:,,,,, and Also the spectrum viewer,,
    has been documented.

    The documentation includes the following information:
     - A short description of what the routine does.
     - The main GUI routine ( and the dispersion-mask routine
       (p3d_cdmask) both contain a complete description of how they are used.
     - A section with information on where additional (detailed) information
       can be found on how the routine works, and where the parameter files can
       be found, which are required to run the routine.
     - A section about the format of the input and the output files.
     - Additional sections with more detailed information on specific tasks.
     - A description of all input and output parameters/keywords.
     - An example section with information on how to use the routine; both by
       itself from the IDL command prompt, or using shell scripts together with
       the IDL Virtual Machine.

  o Added support for six additional instruments:
       p3d has been configured to work with data of all four IFUs of INTEGRAL,
       i.e.: SB1, SB2, SB3, and the Equalized IFU. In the standard setup the
       data must come from the newer 2-detector configuration. If you have data
       of the older one-detector configuration you must use the old detector
       configuration (/OLDINTEGRAL).

       As is the case with the IFU of MPFS all spectra of INTEGRAL are placed
       on the CCD without any gaps. Consequently the spectrum profile fitting
       becomes very slow, as all spectra are fitted simultaneously.

       p3d has been configured to work with all available configurations of
       FLAMES/ARGUS and the mini IFUs FLAMES/IFU1 and FLAMES/IFU2.

     - VLT/UT3/VIMOS (/VIMOS):
       Specifically, p3d is now capable of working with data using the medium-
       and high-resolution grisms (i.e. HR Blue, HR Orange, HR Red, and MR).
       No support for the low-resolution grisms is planned, since these modes
       produce data that is very difficult to reduce accurately.
       Thanks to an improved tracing procedure it should not be necessary to
       modify any parameters when tracing the spectra. But as always, please
       check the outcome to see if p3d could indeed identify all spectra
       correctly. VIMOS is a complex IFU to trace since there are so many
       unused and dead fibers; actually, several spectra simply fall off the
       CCDs. Creating a dispersion solution p3d starts off with a linear
       dispersion solution, which has been calibrated by hand for the HR Blue
       and HR Orange grisms. The remaining two grisms, i.e. HR Red and MR, have
       not yet been calibrated, with respect to the wavelength ranges. So it
       will be more tricky to create a dispersion solution for these grims
       if you are uncertain on which arc lines you can see.

       VIMOS data is provided for four separate detectors, referred to as
       quadrants. The data must be reduced for each quadrant separately,
       before they are combined into one data set. p3d checks all raw data
       files for the header keyword that indicates which detector was used, it
       is not possible to combine images originating in different detectors.

       Starting October 2010 VIMOS has been upgraded with aligned CCDs. The
       current configuration will unlikely work directly for data of this
       upgraded IFU. However, the required changes should be limited to
       providing new dead fiber files. 
     - Gemini/GMOS (GMOS-S and GMOS-N) (/GMOSS,/GMOSN):
       Currently p3d supports the two one-slit modes; i.e. the red and the blue
       slits. The two-slit mode is more complex to reduce, and is not (yet)
       supported. Since we do not have data for GMOS-N the data files are not
       completely configured for this IFU yet (in principle it should only be
       necessary to modify the dead fibers file). Additionally, note that the
       wavelength axis of data of these IFUs cover more than 6000 pixels. The
       initial guess of the linear dispersion solution works poorly across all
       the detectors, which is why it is necessary to make a manual dispersion
       solution, shifting lines at the opposite end by as much as several
       hundred pixels.

     - MPFS (/MPFS3K):
       p3d supports the newer 3kx2k CCD. The spectrum separation is nearly the
       same as the FWHM; hence, the spectrum-to-spectrum cross talk is most
       significant. The strong overlap currently prevents good fits of the
       profiles. Because the spectra are so tightly packed it seems necessary
       to limit the permitted FWHM range during the fitting process. The same
       profile width does not seem to work for both the blue and the red ends.
       A more complete solution will likely have to account for a varying
       FWHM across the dispersion axis. Note that there is no gap between the
       fibers of this IFU (O.K. a gap due to a dead fiber is used as a gap to
       have two spectrum groups). The spectrum profile fitting procedure
       becomes very slow since all spectra are fitted simultaneously.

     - VIRUS-P (/VIRUS):
       Both the setup of fiber bundles 1 and 2 are now available using the
       /VIRUS keyword; the keyword /OLDVIRUS for fiber bundle 1 has been
       removed from the routine

     - and PMAS:
       Through a minor change in the gratings file the new V500 grating is now
       fully supported (the only modification was to replace U300 with V500).

  o Cross-dispersion line profile fitting procedure:
    Before the signal of different spectra can be deconvolved during the
    spectrum extraction it is necessary to calculate a line profile for each
    spectrum. p3d now allows the background linear function to be controlled:
     - By default the constant value and the slope are fitted together with
       all the other profile parameters of each group of spectra.
     - The slope can now be set to zero explicitly by setting the parameter
       "lprofnobgslope" to "yes".
     - The constant value can be set to any positive value (including 0.0),
       by setting the parameter "lprofbgzlevel" to that value. If the parameter
       is instead set to -1 the default fitting is achieved, while a parameter
       value of -2 results in using the lowest value outside the fiber regions.
       This last option is useful whenever scattered light seems to be present.

  o The tracing procedure:
     - When fitting profiles it is now also possible to fit the width of each
       spectrum individually; by setting the 'lprofwidth' parameter to yes.
       Please note, however, that this procedure currently seems to work well
       only with Gaussian profiles. The other profile functions seem to need to
       be more constrained when also the spectrum width is fitted. Since this
       mode makes the profile fitting even slower the default is lprofwidth=no.
     - It is now possible to use a list of spectrum separations for the second
       tracing step. Such a list must have SPNUM-1 entries, where SPNUM is the
       number of spectra. It is important for this tracing mode that the first
       spectrum is actually found in the data. If it is not then it is
       absolutely compulsory that the first spectrum/spectra are indicated in
       the dead fibers file as being dead, unused, or possibly unused.
       Currently this mode is used with the FLAMES/ARGUS IFU, where the
       calibration fiber positions are difficult to fit with a constant
     - Dead, unused, and possibly unused fibers are not used anymore in the
       third tracing step; this is where the spectrum pattern is matched with
       the spectra found in the second tracing step. This improvement was
       necessary to trace spectra of IFUs with many unused or dead fibers,
       such as VIMOS.
     - Dead, unused, and possibly unused fibers are now indicated in the trace
       inspector GUI with dark green symbols and lines.

  o The flat fielding procedure:
    Has been modified to include a better fiber-to-fiber transmission
    normalization, as well as a division by the average spectrum. The new
    modes are available through options in the user parameter file, as is the
    old approach (where all spectra where divided with the average spectrum).

    The fiber-to-fiber transmission function can now either be taken from the
    current or a separate dataset. This approach is useful if you want to use
    twilight flats to perform the fiber-to-fiber transmission correction, while
    you at the same want to use a continuum flat to first smooth and replace
    the dataset with a polynomial fit, before the full spectra are normalized.

  o Using p3d with the IDL VM - i.e., without an IDL license:
    This was previously a bit tricky as all configuration had to be done using
    the GUI. This has now all become much easier.

     - All commands that are required to start p3d (i.e. those of
       and - actually, nearly all commands) can now be entered at
       the shell prompt when starting p3d using the VM script
       For example, to start p3d for the VIMOS IFS instrument, with a log file
       dred.log, and writing all output data to the current directory, you
       would give the following command (this, currently, works on all the
       UN*X platforms):
         ${p3d_path}/vm/ /vimos logfile=dred.log /cwd

     - It is now also possible to call the individual routines using the IDL
       VM. Five new shell scripts are provided in the vm/ directory, one for
       each data-reduction task:,,,, and These scripts
       are most easily called by writing shell scripts that define all the
       required input parameters of the respective task. Each IDL routine
       and has been modified to read command line parameters.
       How the parameters are passed to the shell script is described in the
       respective IDL routine. Here is an example on how to call
       from the terminal (outside IDL):

 tracefile_1.fits.gz,tracefilefile_2.fits.gz \
              $parfile masterbias=$masterbias userparfile=$userparfile \
              opath=$opath logfile=$opath/dred.log /verbose

       Note that it is necessary to click on the splash screen that pops up
       each time the IDL VM is started. A trick on how to automatically click
       on this splash screen (to actually start IDL) is given in README.

     - The spectrum viewer can also be called using the IDL VM. A new shell
       script is provided in the vm/ directory "". In comparison
       to when this routine is called from inside IDL the first argument is
       the filename of the spectrum fits-file. Inside IDL the first argument
       would be the already loaded image variable.

  o The fiber position tables for PMAS/PPAK/4kx4kCCD and PMAS/PPAK/2kx4k CCD
    now also contain the proper positions of the sky fibers. These sky fibers
    are consequently shown properly when viewing them in the spatial map of the
    spectrum viewer.

  o Corrected numerous bugs related to configurations where the dispersion axis
    is in the second dimension (i.e. on the Y axis).

1.1.1 (2010-04-21; Revision 182)
  o Corrected a problem that has affected all optimally extracted spectra of
    all revisions and releases so far. The problem has been that it has been
    necessary to apply a 0.5pixel offset to the profile spectrum positions
    before the spectrum extraction. Previously this offset was applied in an
    improper way which has not solved the problem. From now on all profile
    positions are offset by 0.5 pixel before the data is saved to the *_cdlp*
    file. In order to achieve a better accuracy in you extracted spectra it is
    necessary to either recalculate the profiles or apply the 0.5 pixel offset
    by hand, before extracting the spectra anew. If you want to know more about
    how important it is with accurate profile center positions for the accuracy
    of your extracted spectra please have a look in Sections 4.1 and 4.4 in
    the p3d paper.
    Note! Most instruments are affected by flexure, which has the potential to
    cause an additional offset between the profiles calculated using a
    continuum or sky flat exposure and the object exposure. p3d does not -yet-
    correct for such offsets (although support might be added soon).
  o The image viewer p3d_imv now also looks for raw data in the directory of
    the main input file (FILENAME) and in the parent directory of FILENAME.
    This makes it easier to inspect the data if the reductions are carried out
    on one machine, and are thereafter transferred to another computer (where
    the path is different).

1.1 (2010-04-16; Revision 162) - Upperkuchen
  o Much improved support for use with the PMAS 4kx4k CCD.
  o Implemented a much more strict checking of the available parameters in the
    instrument keywords file(s).
  o Included support for flipping the raw data on the dispersion axis. The
    flipping is performed when the DFLIP parameter is set (either in the
    instrument parameter file or in the user parameter ditto).
  o File suffixes, and the string that is used with combined images, can now be
    set in the user parameter file.
  o All parameters in the master user parameter file
    (data/master_userparfile.dat) are now fully described.
  o Added transparent support for gzip-compressed input data files.
    Z-compressed files (which were compressed using the UN*X-command compress)
    and bzip2-compressed files are also supported on UN*X platforms - see the
    READFITS routine of astro-lib for more information. Output data files are
    compressed using gzip (all platforms!) if the user parameter compress is
    set to yes (default==no).
  o All references to PASP have been changed to A&A, which is where the paper
    on p3d is published (yes, we changed our mind).
  o Updated PPAK tracing parameters to work more generally.
  o Document headers have been adjusted to work better with the IDL->html
    translation tools of O. Streicher.
  o Fixed numerous smaller bugs.

  o GUI
    - Added several different filters in the file dialogs, supporting
      compressed files as well as non-compressed ones.
    - Only the first-block files of multi-block instruments are now shown by
      default in the file dialogs.
    - The default width of file dialogs is now set to 500px when using the VM
      (see the Xdefaults file in vm/).

  o Image combination
    - All image-combination-related operations have been moved to a separate
      routine (routines/ to comply with the
      "Don't Repeat Yourself" coding principle.
    - Added a slow combination method since the fast MIN() function of IDL
      gives strange results (with PMAS at least). This slow method is now also
      the default.

  o Spectrum extraction
    - The optimal extraction routines MOX and MPD have been moved to their own
    - The default extraction method for MPD is now 'banddiagonal' and not 'svd'
      as previously ('svd' is painfully slow).
    - Information about what is going on is now also printed on the console
      when verbose==1.

  o Master Bias
    - It is now possible to pick only one file and then let p3d smooth it
      before it is used as a master bias. This now also works with the
      PMAS 4kx4k CCD.

  o Wavelength calibration / Dispersion Mask creation
    - Now works properly with two input files for all supported instruments.
      Two files (ideally of different exposure time) can be used if arc lines
      on one side of the detector are much fainter/brighter than on the other
    - After entries in the line list were matched with the data a more precise
      Gaussian fitting (or weighting) was previously performed in the
      "Create dispersion mask" step. In order to allow a checking of the more
      precise fitting this fitting has now been given its own function in the
      GUI ("Fit centroids"). This function (button) must now be used after the
      "Match centroids" step, and before the "Create dispersion mask" step.
    - The default behavior for p3d is now to not rebin the data on the
      dispersion axis before starting the GUI. This allows big data sets to be
      used on very small screens (DBIN=1). If the user still wants to have the
      data rebinned then she/he can select another mode in the Options menu,
      or simply set the DBIN parameter in the user parameter file.

  o Object extraction
    - Now accepts a cosmic ray mask (although this method does not work well at
    - p3d can now also save output spectra in the E3D-format (finally; as we
      have promised in the paper on p3d).

1.0 (2010-02-02; Revision 30) - Destiny
  o Initial version.
page last updated on: Wednesday June 28, 2017 11:06:27; retrieved on: Thursday June 29, 2017 03:41:23