The p3d logo. a general data-reduction tool for fiber-fed integral-field spectrographs
About p3d

Readme

Download

References

Documentation & Tutorials

Links

Screenshots

Authors & Thanks

Sourceforge project page

README

p3d README

Version information: This is the subversion trunk version of p3d, which contains all the latest bleeding edge features; although, beware that some of the functionality may have been broken during the addition of other new features. If you want a version where few modifications are made, use one of the release packages instead (see: http://sourceforge.net/projects/p3d/files/).

$Id: README 4609 2017-06-06 13:35:59Z christersandin $

Table of Contents

1.

Introduction

  • 1.1 About p3d

  • 1.2 Installing p3d

  • 1.3 Use p3d with the IDL Virtual Machine, without a license

  • 1.4 Use p3d from the IDL command line, with an IDL license

  • 1.5 Additional details about the auxiliary software

2.

Contact

  • 2.1 Funding acknowledgments

  • 2.2 Collaborators

  • 2.3 Reporting Bugs

  • 2.4 Feature Requests

3.

Software Platform

  • 3.1 Coding Philosophy

4.

Supported Instruments

5.

Recompiling the binary meta-files for the IDL Virtual Machine

6.

IDL Terms of use

1) Introduction

Section 1.1 contains a very brief introduction to p3d while the installation procedure is discussed in Section 1.2. Sections 1.3 and 1.4 describe how p3d is used from outside of IDL with the IDL Virtual Machine or from the IDL command line using a license, respectively. Section 1.5 discusses details of the required auxiliary software for users with a specific interest.

1.1) About p3d

p3d is a data-reduction tool for fiber-fed integral-field spectrographs. Please see the project web site for more details: http://p3d.sourceforge.net.

1.2) Installing and using p3d

The number of required steps to install p3d is very small for basic use. The only requirement besides p3d itself is a working version of Python (version >= 2.7) and IDL (version >= 6.2). In case that IDL is unavailable on the system, there is the option to ask us for a p3d-specific IDL distribution that works on all platform supported by IDL; IDL is assumed unavailable when the environment variable IDL_DIR is unset. To get a download link to the p3d-specific IDL distribution, please send an e-mail to the address given in the contact section.

The minimal installation consists of retrieving p3d, IDL, as well as the additional (auxiliary) software that p3d uses. p3d can download the additional software by itself when it is launched from outside IDL (with one exception, see Section 1.5); otherwise, it is only necessary to launch the software download tool p3d_acquire_xtools.py in the p3d vm directory. There are two approaches to retrieving a copy of IDL, see sections i and ii. Section iii contains information on how to acquire p3d itself (including this file).

Attention: While p3d can be run using IDL versions >= 6.2, there seems to be a bug with IDL versions < 8.4 that appears when subtracting background regions in the spectrum viewer. Therefore, consider using IDL version 8.4 or higher.

i) Use the p3d-specific IDL distribution

Due to licensing issues of IDL, we cannot put the p3d IDL distribution on the internet for free download, but just send us an e-mail and receive a download link in return. The p3d IDL distribution is about 260MB large and works on all platforms supported by IDL version 8.6.0; this distribution is only meant to be used with the pre-compiled binary save files of p3d (regardless of if an IDL license is available or not) and not from the IDL command prompt. The reason is that it does not contain all of IDL, but only the parts that p3d use.

The distribution is a zip archive file that must be placed so as to have the idlrte8.6.0 directory in archive in the p3d contrib directory (see section iii below). The environment variable IDL_DIR, which configuration is described below, is set automatically to the correct path when it is unset and the p3d-specific distribution is identified.

ii) Install a full copy of IDL

A complete version of IDL can otherwise be downloaded from the following web page (it is necessary to register to be able to download the software): http://www.exelisvis.com/Downloads/ProductDownloads.aspx See the relevant IDL documents on how to install IDL (i.e. extract or copy the files to a proper location on your computer) and on how to acquire and setup an IDL license (if one is required). After IDL is installed, it is necessary to setup the system environment to tell it where IDL is found; this is unnecessary if the p3d-specific distribution is used (see i) or if your system administrator(s) has already done this. The environment variable IDL_DIR must be set to the name of the directory where IDL is installed. Check if it is already set by entering the following command (on *NIX-type platforms):

printenv IDL_DIR

If the output of the command points to the path of the IDL version to use, there is no need to set the path anew.

If IDL_DIR is unset and the p3d-specific distribution is not used, or if there is a wish to use a personal downloaded copy of IDL, the following line can, depending on the used system shell, be added to the bash terminal setup file in the home directory (~/.bashrc):

export IDL_DIR=/path/to/IDL/

or if the csh is used (~/.cshrc):

setenv IDL_DIR /path/to/IDL/

where the tilde character (~) denotes the user home directory.

It is possible to use the p3d-specific IDL distribution instead of an installed version of IDL (if the files are placed in the required directory) by simply unsetting the IDL_DIR environment variable as follows (on *NIX-type platforms):

unset IDL_DIR

Attention: If p3d is to be used on a Windows-based platform, the IDL_DIR environment variable is defined using Windows GUI tools. The exact procedure to set this variable varies depending on the used version of Windows. The variable should also not in this case be set when using the p3d-specific IDL distribution.

Check that the IDL executable is available by entering the following command (this step is not required on Windows-based platforms):

which idl

If there is no output, a symbolic link could be created that points at the IDL binary; this can be done using the following example (assuming there is a bin/ directory in the home directory and that ~/bin exists in the PATH environment variable:

cd ~/bin
ln -s ${IDL_DIR}/bin/idl

When "idl" is typed at the prompt IDL should start. If there is no IDL license available, there will be a lot of information about this fact – but there is no need for concern about these particular messages if the intention is to run p3d using the IDL Virtual Machine.

iii) Get the p3d software

Either use the p3d development version through the subversion tool, or download one of the prepared releases from the program files area (https://sourceforge.net/projects/p3d/files/). The former approach is preferred when new functionality is required, which was not yet included in a release version; although, there is a slight chance that the development code contains new bugs (that are corrected as soon as they are found and time permits, or when we are told of their existence).

To download p3d from the subversion repository at sourceforge.net, open up a terminal and change the directory to where the software is to be placed. Thereafter, enter (or paste) the following command:

svn checkout svn://svn.code.sf.net/p/p3d/code/trunk p3d

whereby all p3d-related files are placed in the directory named p3d (any directory name can be used). It is also possible to use the http protocol instead of svn by replacing "svn:" with "http:".

To update p3d when using the development version, change to the p3d directory in a terminal (or whatever name is used for this directory) and type the command 'svn update'. For any additional functionality of subversion, including a complete and very extensive manual, please consult its web page: http://subversion.apache.org/.

The required auxiliary programs of p3d are downloaded automatically to the p3d contrib and src directories if they are not detected when a p3d tool is launched from the shell, with one exception being the Voronoi binning tool of Cappellari & Copin (2003). The files of the following sources are required:

  • Astro-lib (http://idlastro.gsfc.nasa.gov/homepage.html)

    • All files (because it is easier to download all files).

  • Mpfit (http://purl.com/net/mpfit)

    • mpfit.pro

    • cmset_op.pro

    • cmpfit-1.3a.tar.gz

  • Voronoi bin tool (http://www-astro.physics.ox.ac.uk/~mxc/software/)

    • voronoi_2d_binning.pro

When the intention is to only use p3d from the IDL command line (see below), it is necessary to download the required auxiliary programs separately using the tool p3d_acquire_xtools.py; the same tool is called by the p3d Python tool and the p3d bash scripts when the files are non-existent:

p3d_acquire_xtools.py

The files are placed in contrib/astro-lib/ (for the files of Astro-lib) and in contrib/ (mpfit.pro and cmset_op.pro). The archive file cmpfit-1.3a.tar.gz is extracted in src/, where it creates the sub-directory cmpfit-1.3a. Owing to licensing issues, the Voronoi bin tool voronoi_2d_binning is not downloaded automatically, but must be downloaded separately, putting the required file (voronoi_2d_binning.pro) in the p3d contrib directory (all tools will still work without this routine, but the related parts are deactivated).

Note: It is also possible to download all parts and releases of p3d to the same directory by leaving out the "/trunk" part in the command above. Few users will want to do this. (The directory is in this case about 1400MB at around revision 3800.)

Note: Due to updated software on the sourceforge.net side, before 2012-12-07, the repository address was different. The old and the new repositories are incompatible, which is why it is necessary to replace older repositories with one using the address above, to allow any software updates.

Attention: When p3d starts, it places the directories it uses at the front of the IDL path, in the correct order. If other IDL tools depend on p3d, it is necessary to set the environment variable IDL_PATH or the IDL !path variable in the IDL startup file, adding the p3d directory at the required position in the used configuration. Doing this, p3d adds the additional directories at the location of that string. (There will be confusion if p3d versions are mixed up in IDL_PATH or !path.) The way p3d works, it uses the parent directory of the called p3d tool (using the IDL Virtual Machine or from the IDL command line) as the path to p3d. Likewise, p3d sets up the preferences (environment) variable IDL_DLM_PATH to include the src directory in the p3d path.

Attention: If the intention is to use p3d on a Windows-based platform, these environment variables are defined using Windows GUI tools. The exact procedure varies depending on which version of Windows is used.

Windows XP: Control Panel –-> System –-> Advanced system settings –-> Environment Variables

Windows Vista: Start –-> Computer –-> Right Click –-> Properties –-> Advanced System Settings –> Advanced –> Environment Variables

Windows 7: Start –-> right click Computer –-> Properties –-> Advanced System Settings –-> Advanced tab –-> Environment Variables

Windows 8: Start –-> All Apps –-> Control panel –-> System –-> Advanced System Settings –-> Advanced –-> Environment variables

Windows 10: Window Key + X –-> System –-> Advanced System Settings –-> Environment Variables...

Attention: Set the environment variable ${p3d_font} to the name of an existing font (see the above example) if you want to use some other font than the IDL default font; this might not always be a good idea, but the option is available nevertheless. It is important to select a font that exists on the system, check which ones are installed using, e.g., the 'xlsfonts' command. It is, additionally important to check that the used font is not too large; because, if it is, some widgets might not fit inside the respective tool (if this point seems vague, try a very large font, launch p3d or p3d_sv and look at the resulting windows). Due to restrictions in how IDL parses command-line arguments, the font string must not contain any white space!

a) Using the Virtual Machine

The intention is that p3d is used with the IDL Virtual Machine. p3d configures itself when its main tools are launched; the use is more comfortable when the p3d vm directory is added to the system PATH. Additionally, it might be helpful to set the environment variable p3d_data_path to the name of the (base) directory where the raw data are kept. Depending on the system shell, the following lines could be added to a suitable place in the ~/.bashrc file (if using bash; or another "Bourne Shell"-type environment):

export PATH=$PATH:/path/to/p3d/vm        # Useful
export p3d_data_path=/path/to/IFU/data/  # Optional, but useful
export p3d_font=lucidasans-14            # Optional, experts only

or ~/.cshrc (csh):

setenv PATH ${PATH}:/path/to/p3d/vm      # Useful
setenv p3d_data_path /path/to/IFU/data/  # Optional, but useful
setenv p3d_font lucidasans-14            # Optional, experts only

b) Using p3d from the command line

It is convenient to add the p3d directory to the IDL path to avoid having to enter the full path to the tool each time it is used. There are two ways to do this. Either set the environment variable IDL_PATH or the environment variable IDL_STARTUP. Avoid using both approaches, because there will be confusion and possible problems if the used p3d path changes and the change is not made in both places.

The first approach requires IDL_PATH to be set to include the p3d path. Put the p3d path at the front, unless an IDL program is used that itself uses p3d. Add the following line to ~/.bashrc (ensure that the string ":<IDL_DEFAULT>" is added at the end) using a "Bourne Shell"-type environment:

export IDL_PATH="/path/to/p3d:<IDL_DEFAULT> # Modify for your system

Instead add the following line to ~/.cshrc in a "C Shell"-type environment:

setenv IDL_PATH "/path/to/p3d:<IDL_DEFAULT> # Modify for your system

The second approach is to set the environment variable IDL_STARTUP to the name of an IDL program that defines the path. The following lines can be used to do this (the filename used here "IDL_STARTUP.PRO" can of course be chosen freely). For a "Bourne Shell"-type environment, add the following line to ~/.bashrc:

export IDL_STARTUP=/path/to/IDL_STARTUP.PRO # Modify for your system

For a "C Shell"-type environment, add this line to the file ~/.cshrc

setenv IDL_STARTUP /path/to/IDL_STARTUP.PRO # Modify for your system

As in the first approach, the p3d path is added at the beginning of the string unless an IDL program is used that itself uses p3d; in the latter case, the p3d path should come after the other IDL program. Add the following line to the file IDL_STARTUP.PRO:

!path = '/path/to/p3d:' + !path

Once the p3d tools are used, the IDL path is modified to fit p3d. If it is necessary to call one of the p3d modules in the p3d routines directory, it is recommended to first launch p3d with the SETUPONLY keyword, which only configures the IDL !path and then exits:

p3d, /setuponly

1.3) Use p3d with the IDL Virtual Machine, without a license

A Python tool as well as multiple bash scripts are provided to run p3d from the shell, outside of IDL, using the IDL Virtual Machine on all *NIX-type platforms as well as in Windows. To start the p3d main GUI using the p3d queue system, enter the following in a terminal (if p3d/vm was added to the system path):

p3d_dispatch.py [-s] <tool> <options>

or, if p3d/vm was not added to the system path (this is just an example, the full path could be used instead of setting p3d_path):

${p3d_path}/vm/p3d_dispatch.py [-s] <tool> <options>

Add the optional flag –-noqueue (or its equivalent -s) before the tool argument if you want to avoid using the p3d queue system, which might require some effort to use efficiently as well as understand. The purpose of the queue system is to click the IDL 'click-me' popup window fewer times when no IDL license is available; this is facilitated by the use of p3d server sessions that run in the background where each session picks one job at a time from a job queue managed by a p3d Python queue handler (see below).

The tool can be selected as one of the following 15 strings: p3d, p3d_sv, p3d_cmbias, p3d_ctrace, p3d_cflatf, p3d_cobjex, p3d_cvimos_combine, p3d_cr, p3d_darc, p3d_rss2cube, p3d_cexposure, p3d_fluxsens, p3d_fluxcal, p3d_ifsfit, p3d_ifsfit_mosaic, p3d_csim (more will be added). It is also possible to use any of the older bash-based scripts (which all call p3d_dispatch.py).

p3d_vm.sh [-s] [-rt] <options>
p3d_cmbias_vm.sh [-s] [-rt] <options>
p3d_ctrace_vm.sh [-s] [-rt] <options>
p3d_cdmask_vm.sh [-s] [-rt] <options>
p3d_cflatf_vm.sh [-s] [-rt] <options>
p3d_cobjex_vm.sh [-s] [-rt] <options>
p3d_cvimos_combine_vm.sh [-s] [-rt] <options>
p3d_rss_vm.sh [-s] [-rt] <options>
p3d_cr_vm.sh [-s] [-rt] <options>
p3d_darc_vm.sh [-s] [-rt] <options>
p3d_rss2cube_vm.sh [-s] [-rt] <options>
p3d_cexposure_vm.sh [-s] [-rt] <options>
p3d_fluxsens_vm.sh [-s] [-rt] <options>
p3d_fluxcal_vm.sh [-s] [-rt] <options>
p3d_ifsfit_vm.sh [-s] [-rt] <options>
p3d_ifsfit_mosaic_vm.sh [-s] [-rt] <options>
p3d_csim_vm.sh [-s] [-rt] <options>

See the respective tool for detailed information regarding available options:

p3d

(and p3d_tool_gui) for the p3d GUI.

p3d_cmbias

to create a master-bias image.

p3d_ctrace

to create a trace-mask image.

p3d_cdmask

to create a dispersion-mask image.

p3d_cflatf

to create a flat-field image.

p3d_cobjex

to extract science-object spectra.

p3d_fluxsens

to create a sensitivity function from a summed standard-star spectrum. The sensitivity function is then used to flux calibrate the data. A sensitivity function can also be created using the p3d spectrum viewer (p3d_sv).

p3d_fluxcal

to flux calibrate extracted data using the sensitivity function created with p3d_fluxsens or p3d_sv (or even IRAF).

p3d_cvimos_combine

to combine extracted VIMOS spectra of the four separate quadrants into one spectrum image.

p3d_sv

to view and analyze extracted spectra as well as create a sensitivity function for flux calibration. Accepts both RSS images created using p3d and data cubes of any origin.

p3d_cr

to clean single images of cosmic-ray hits.

p3d_darc

to correct data for effects of differential atmospheric refraction; only works with instruments that use square-shaped elements, currently.

p3d_rss2cube

to convert the resulting RSS-formatted images into three-dimensional cubes; only works with instruments that use square-shaped elements, currently.

p3d_cexposure

to combine multiple extracted images; can select files that used different exposure times to replace data in extracted spectrum elements that involved saturated regions in the raw data.

p3d_ifsfit

to fit any number of emission and absorption lines to the spectra, in countless combinations.

p3d_ifsfit_mosaic

to create publication-ready plots of the outcome of p3d_ifsfit.

p3d_csim

to create simulated raw data, which can be used to test instrument properties.

Note! The options are specified in the usual IDL way, almost, but with a space between the options (instead of a comma, as is the case when p3d is used from the IDL command line). Additionally, array values are specified without brackets and without white space between the elements. For example, some arguments and keywords accept multiple files. Such files are separated with a comma, but without any white space; for example, use "file1.fits.gz,file2.fits.gz range=200,2200" in a system shell and "['file1.fits.gz', 'file2.fits.gz'], range=[200, 2200]" on the IDL command line.

Note: The p3d Python tool as well as all shell scripts can be called with the -rt option. An IDL license is required in this case. This behavior could be useful if one wants to avoid having to click in the splash window that always appears each time the IDL VM is launched. Although, with the p3d queue server system, the need to click on those splash screens is delimited to when the p3d queue system is first launched.

Tutorials on how to use p3d in the data reduction and visualization are available in the doc directory. Here are two examples on how to use the tools from the system shell to get started:

p3d_dispatch.py p3d /cwd /ppak4k logfile=dred.log
opath=output_directory
p3d_dispatch.py [--noqueue] p3d_cobjex \
   file1.tar.gz,file2.tar.gz \
   opath=$opath masterbias=${opath}/masterbias.fits.gz \
   tracemask=${opath}/tracemask.fits.gz \
   flatfield=${opath}/flatfield.fits.gz \
   userparfile=${opath}/user_p3d.par logfile=${opath}/dred.log \
   loglevel=2 /verbose /quiet

The p3d IDL server allows any number of jobs to be submitted to the p3d queue. The p3d queue handler feeds the jobs to the p3d servers, which are launched automatically and continue to run until the queue is shut down. Each p3d server uses one thread (core) on the computer; the server is unaware of how many other threads there are and how many threads it can use by itself. To make efficient use of system resources, it is therefore necessary to specify how many threads should be used when a job is submitted. When all jobs are done, the queue is shut down using the following command:

p3d_queue_stop.py

If the queue should stop working for some strange reason, it is possible to kill all IDL sessions launched by the queue handler with the following command:

p3d_queue_kill.py

Note: The p3d_dispatch.py program checks if the required IDL binary meta files (e.g. p3d_*.sav) are present in the vm directory. IDL launches using these files.

Note: The Python tool p3d_dispatch.py as well as all bash scripts p3d*_vm.sh require that Python is installed on your computer, any version >= 2.7 will do. Also note that, if the command /usr/bin/env does not exist, it is necessary to launch this (as well as the other) python program(s) preceeded by the name of the Python command (with the full path):

python p3d_dispatch.py
python p3d_acquire_xtools.py
python p3d_tool_makevm.py

If there is a need to use the bash scripts instead of the Python program, it is necessary to have bash installed as /bin/bash on the computer (this does not apply to Windows computers). Please, edit the first line in the scripts accordingly if this is not the case (i.e. the first line stating "#!/bin/bash"), or execute them as in the following examples:

bash p3d_vm.sh
bash p3d_cmbias_vm.sh

Note: With an IDL license, it is possible to recompile the p3d binary files used with the IDL Virtual Machine. For this purpose, it is strongly advised to use the provided Python tool p3d_tool_makevm.py (see section 5.0).

Note: The debug option (/DEBUG) is unavailable when using p3d from the system shell (also when the -rt option is used). However, it is still possible to select not-yet-fully supported instruments from the command line, which can otherwise only be selected with the instrument-selection GUI when /DEBUG is set.

1.4) Use p3d from the IDL command line, with an IDL license

After performing the installation as decribed in Section 1.2, just enter the name of the tool to use on the command line, with the necessary arguments and keywords. For example, to launch the p3d data-reduction GUI, enter:

p3d

Available options are found in the source file, which can be browsed using any available text editor or, e.g., the commands more or less), but they can also be shown using the HELP keyword:

p3d, /help

The help option is available with all tools and routines of p3d.

Note: The exact order of entries in the IDL path is important.

Note: Do not include the p3d vm directory in the IDL !path variable! If this is done, there will be a confusion in the path if any of the program routines is changed. So, even though a p3d routine was changed, IDL might choose to use the pre-compiled version of the same routine in the binary meta files p3d*.sav. Consequently, if the vm directory is kept in the IDL !path, it is necessary to recompile all routiness after any change has been made. Avoid this hassle by not having the vm directory in !path.

1.5) Additional details about the auxiliary software

Some routines or parts of routines are, in addition to their IDL implementation, also provided in C. The reason is that compiled C code in computationally intensive cases (involving nested loops) run faster than the corresponding IDL code; often drastically faster. Additionally, in C routines it is possible to make use of more of the computing power on a multi-processor computer; through OpenMP (http://www.openmp.org).

Here are some of the tasks available in both C and IDL code:

  • The calculation of cross-dispersion line profiles.

  • The calculation of a median-filtered science image, in regions where there are no spectra, to subtract a scattered-light, or a dark-current, component.

  • The spectrum extraction, using optimal extraction with the multi-profile decvonvolution method and the band-diagonal option.

  • The conversion from pixel space to spatial-element space when selecting a region of interest in the spectrum viewer.

  • The calculation of pixel-based cross-dispersion profile images used in the optimal extraction.

  • The combination of multiple images using the min/max filter, and the "slowimcombine" mode.

  • Cosmic-ray cleaning of single images.

  • The resampling during wavelength calibration from pixel-based coordinates to wavelength.

These particular routines were implemented in C because their IDL implementations are more-to-extremely time-consuming.

The C routines are compiled automatically by p3d whenever they are required. With the exception of the spectrum viewer p3d_sv, all available threads on a multi-processor computer are used by default, use the NTHREADS and CRNTHREADS keywords to change this number. The spectrum viewer uses one thread by default. To use the compiled C routines, ensure a working C compiler is available. If the compilation fails, for any reason, p3d falls back to use the corresponding IDL version. Use the keyword CDEBUG to make the compilation process more verbose. It is also possible to completely deactivate the C modules by setting the keyword NOC or the user parameter 'dontusecroutines' to 'yes'. The C-based cosmic-ray cleaning routine can be deactivated separately by setting the NOCRC keyword (NOC with p3d_cr) or the 'nocrc' user parameter.

Note: If one or several of the C routines in are updated after the shared library is built, it is necessary to either set the rebuild library parameter ('rebuildlib yes') or to remove the library file in src/p3d_clib[.so,.dll]. A new library file is then compiled from the source files.

Note: The C routines are parallelized using OpenMP. Please try a more recent compiler in case messages are shown indicating that OpenMP-related routines are not found.

Note: The native compiler on MacOSX platforms clang does not include support for OpenMP (it seems). Therefore, p3d uses the gcc compiler, which needs to be installed on the system.

Note: The C routines are currently only compiled on Linux and MacOSX platforms. Please contact us if you need support on one of the other platforms; i.e. Windows and Solaris. In principle, it is only a question of adding an appropriate case-statement block in the routine "p3d_misc_get_callext_exlib", but it can get complicated if a non-standard compiler is used. Any input on this issue is well received.

Warning: If p3d is used on a platform while switching between the 32 bit and the 64 bit IDL environments, please remember to delete the shared-library files between the switches, otherwise the shared library will be of an incompatible architecture, which means big problems. p3d can use its scripts with the 32 bit environment on 64 bit machines if the IDL_OPTS environment variable is set to "-32". However, it is not recommended to do this switching between 32 and 64 bits, as the outcome is system-dependent and therefore generally unpredictable.

2) Contact

All collaborators, unless noted otherwise, have the following affiliation:

Leibniz Institute for Astrophysics Potsdam (AIP) Instrumentation An der Sternwarte 16 D-14482 Potsdam GERMANY Tel: +49-(0)331-74990 http://www.aip.de

The p3d-project web site is:

http://p3d.sourceforge.net

where the software can be downloaded, the documentation viewed, and much more. Please send an e-mail to the following address, should you want to use the p3d-specific IDL distribution:

olestreicher or p2w@users.sourceforge.net

Documentation is available in the individual routines – the same text is also available on the Documentation page at the project web site; this documentation is updated whenever there is a release. Detailed descriptions of the algorithms are provided in the (freely available) Astronomy & Astrophysics paper about p3d:

Sandin et al. 2010, A&A, 515, 35 http://www.aanda.org/articles/aa/pdf/2010/07/aa14022-10.pdf

And also see the Messenger and SPIE papers (download links are available at the p3d web site):

Sandin et al. 2011, The Messenger, 144, 13-16 "p3d ~ A Data Reduction Tool for the Integral-field Modes of VIMOS and FLAMES"

Sandin et al. 2012, in: "Software and Cyberinfrastructure for Astronomy II", Radziwill N. M., Chiozzi G., eds, Proc. SPIE, 8451, 84510F-1~16 "Automated and generalized integral-field spectroscopy data reduction using p3d"

2.1) Funding acknowledgments

This project has been made possible through funds from the following sources: BMBF: 03Z2A51, 05A08BA1

2.2) Collaborators

All programming contributors of p3d

Christer Sandin

Project leader and programmer, until 2016-07-22.

If you have comments of bugs or feature requests, which are directly related to p3d, and that you cannot add to the tracker at the project web site (see below for instructions), please send them to the e-mail address given in the Contact section.

Fachreddin Tabataba-Vakili

Has ported the multi-profile deconvolution spectrum extraction routine from IDL (p3d_extract_optimal_mpd.pro) to automatically compiled C (p3d_cmpd.c), which is also parallelized with OpenMP. He has also made the main work on the inclusion of interactive flux calibration (using a graphical user interface) in p3d. He has written a tool that generates IDL runtime distributions, and the framework tools around the p3d IDL server. He is finally working on the conversion of various slow routines to compiled and parallelized C.

Peter Weilbacher

Has ported the slowest routine of p3d, the cross-dispersion line profile fitting routine, to C, which is also parallelized using OpenMP. Has also included an option to calibrate the dispersion mask against sky emission lines. He also reviews the routine documentation.

Ole Streicher

Has provided p3d with a first version of the tutorials at the project web site. Has also written various tools that automatically translate the routine documentation into xhtml documents, which can be viewed at the project documentation web page: http://p3d.sourceforge.net/index.php?page=doc

Additional acknowledgments

Please see the p3d project web site for further acknowledgments.

2.3) Reporting Bugs

To report any kind of bugs we urge you to use the bug tracker available at the p3d-project web site (Develop/Tracker/Bugs); this way the bugs will be seen by all users and programmers, and won't be lost in an e-mail program.

At the bug-tracker page you can search previously reported bugs, to see if your bug has already been reported. Or you can open your own bug description (by clicking on "Add new"). Doing this, please ensure that you use the latest version on the subversion trunk of p3d (or specify the revision or version of the release you use). Thereafter, specify a category for the bug, and provide a short summary of the problem, and most importantly, describe the problem with enough detail that somebody who does not use your setup and your data files is able to follow your arguments. Bug reports without a useful description will be assigned a low priority, or even be deleted. If you think that it would help us, attach all relevant data files to your report (please compress fits-files using tar/gzip/bzip2/xz, zip, or use the compress-option of p3d to save space). Additionally, please specify a reasonable priority to your bug. While our intention is to correct serious bugs as soon as possible ~ these are the ones that prevent p3d from producing spectra ~ GUI-related issues are assigned a lower priority.

Note: When reporting a bug, turn on the debugging mode (/DEBUG) of p3d and paste the resulting output from the IDL terminal/window (once the bug has been reached) into your bug description. The reason we ask you to do this is that the information returned by the error handler, when the DEBUG keyword is unset, only provides the name of the file where the problem occurred, and an IDL-specific number. Such sparse information is nearly always insufficient to interpret.

2.4) Feature requests

If p3d is missing some functionality that you would very much like to have included, you can make a feature request using the tracker on the p3d project web site (Develop/Tracker/Feature request). Following the description on how to add a new bug description we ask you, also in this case, to be as detailed as possible. Please understand that we are a team with limited resources. If your request concerns a huge modification, which requires many days, or even months, to code, it might not happen at all. On the other hand, we are highly interested in taking on project members who are interested in contributing to p3d with their own time and coding. Please drop us a note if you think this could apply to you.

Alternatively, send an e-mail with your feature request to the e-mail address provided in Section 2.

3) Software Platform

p3d is nearly completely based on the language of the (proprietary) Interactive Data Language (IDL; cf. http://www.exelisvis.com), and cannot be used if IDL is unavailable ~ the easiest way to get IDL is to ask us for the p3d-specific IDL distribution, see Section 1.2. Although, it is necessary to have an IDL license to modify the code, p3d can be run without an IDL-license using the supplied binary meta-files (vm/p3d*.sav). In this case, using the, so-called, IDL Virtual Machine; p3d has been implemented in such a way that it can be used with full functionality without a license.

Since p3d is based on IDL it can be used on any platform supported by IDL. However, the installation procedure differs slightly for an installation on a Windows-based platform compared to on the other supported platforms; the required environment variables need to be setup differently (see above) and the compilation of the C routines was not yet configured [please inform us if you are willing to help us finish this task].

3.1) Coding Philosophy

p3d is a large and growing piece of software. To keep it manageable, we have applied rules of coding practice; use of this practice has so far made debugging easy as bugs are located quickly.

These are the rules, which we would ask any project programming member to follow:

  • All files must have filenames which conform to: p3d_task_name.pro

  • Code layout: + All files must have a fully explained header. + All routines use the options: compile_opt hidden, IDL2 + The maximum number of character columns per line is 79. + All blocks are indented by 2 characters. + Comment lines begin with ';;' + Integers are with few exceptions specified as long with L, e.g. '0L'. Exceptions: VERBOSE and ERROR variables use integers. + All commands and statements are written with small characters.

  • Common blocks are only allowed for variables which obviously will never ever change, i.e. pretty much never.

  • An error handler is setup for every routine, unless the user keyword DEBUG is set.

  • Logging comment statements must always use 'p3d_misc_logger', adopting the different levels of verbosity.

  • Input parameters should be checked for consistency in terms of type, size, value and range, as far as possible.

  • All command keywords are for clarity specified in full, i.e. no abbreviations (for example, /xstyle not /xst).

  • Text widgets must include error checking to ensure that entered values are of the right type and range.

  • All widgets in GUI tools with a status line must include the UNAME field that specifies what the widget in questions does.

Issues related to the subversion repository of p3d:

  • No file on the trunk should, as far as possible, be committed before the modifications are confirmed to work as intended.

  • All commits to the repository should include a description of the modification (the longer the better, usually). An example is:

    svn ci --message 'Correct/modify this and that.' <file>

  • Larger modifications should use a branch, and not the trunk.

  • The idea is to copy the trunk to branch/ in preparations for a release. Then we only make bug fixes on that branch before we copy the final release version to tags/.

4) Supported Instruments

We have developed p3d to be a general tool. General in the sense that it can be used with any integral field unit (IFU) of fiber-fed integral-field spectrographs. Additionally, the p3d spectrum viewer can be used with a data cube of any origin, such as MUSE.

Before p3d can be used with any instrument, it must be properly setup. p3d has been setup, tested, and found to work well with the following instruments:

PMAS

PPAK and LARR, for both the 2kx4k, and the newer 4kx4k CCD.

VIRUS-P

For fiber bundles 1 and 2.

SPIRAL

For both the red and the blue arm.

VIMOS

The MR, HR-Blue, HR-Orange, and HR-Red modes are supported, for both the older configuration as well as the refurbished instrument (2010.08- and 2012.03.15-). There are no plans to include any support for the low-resolution modes (LR).

FLAMES

The ARGUS IFU and the two mini-IFU modes.

GMOS-S/N

The red and the blue slit modes.

MPFS

For the newer 3kx2k CCD.

VIRUS-W

Both the low-resolution and the high-resolution modes.

INTEGRAL

Both the older single CCD and the newer double CCD setups are implemented.

ERA2

A life-sciences instrument at AIP.

ERASMUS-F

A test bench at AIP.

MRS1

A life-sciences instrument at AIP.

(MRS2

A life-sciences instrument at AIP – to come.)

SAMI

Both the red and the blue arms are basically supported.

(IMACS

For all (known) modes – under development)

See the p3d-project web page for more detailed information. Specifically, have a look at the documentation WIKI for instrument-specific recommendations for your data-reduction work: http://sourceforge.net/apps/mediawiki/p3d/index.php

Note: Cosmic-ray cleaning of single images and both approaches of optimal extraction are supported on all instruments.

5) Recompiling the binary meta-files for use with the IDL VM

The simplest way to (re-)compile the binary meta-files, which are used with the shell scripts (and the IDL Virtual Machine), is to use the provided script in p3d_path/vm/. An IDL license is a must to recompile p3d! The following steps could be used from a terminal:

1.

Set the p3d root directory and the IDL_DIR environment variables. In a "Bourne Shell"-type environment (e.g. bash), type:

export p3d_path=/path/to/p3d/   # Optional
export IDL_DIR=/path/to/IDL/    # IDL version >=6.2 is required!

In a "C Shell"-type environment (e.g. csh), type:

setenv p3d_path /path/to/p3d/   # Optional
setenv IDL_DIR /path/to/IDL/    # IDL version >=6.2 is required!

2.

Download the required auxiliary files (unless this was already done); this could, most conveniently, be done using the provided Python tool:

p3d_acquire_xtools.py

Note: The binary meta-file generation script requires the additional files to be present in p3d_path/contrib/astro-lib (with all .pro-files in p3d_path/contrib/astro-lib/pro/, and subdirectories) and in p3d_path/contrib/. It may sometimes happen that a routine has been moved to another directory. In this case, it will be necessary to modify the path of p3d – if the development version of p3d is used, this might already have been solved in a more recent revision.

3.

Generate the binary meta-file in vm/p3d_*.sav; this is most conveniently done using the provided Python tool:

p3d_tool_makevm.py

Note: If there is any problem in the binary meta-file generation process, please check the output of the scripts to find out what the problem is.

= 6) IDL Terms of use

The following terms of use apply when using IDL; the text is pasted from the web page of exelisvis:

IMPORTANT: READ CAREFULLY

Licensee acknowledges that it is not a citizen, national, or resident of, and is not under the control of the government of Cuba, Iran, North Korea, Sudan or Syria, and that Licensee will not download or otherwise export or re-export the Software Product and any related technical data directly or indirectly to the above mentioned countries nor to citizens, nationals, or residents of those countries. Licensee will comply with all applicable export, re-export and foreign policy controls and restrictions imposed by the United States and will take the necessary actions and precautions to ensure that it complies with all such laws or regulations.

I am not located in and that I am not a national of Cuba, Iran, North Korea, Syria, or Sudan, and I agree not to transfer the Software Product to any such locations or individuals;

I am not listed or affiliated with anyone on any of the following lists:

Denied Persons List – U.S. Department of Commerce Unverified List – U.S. Department of Commerce Entity List – Supplement No. 4 to part 744 of the Export Administration Regulations Specially Designated Nationals – U.S. Treasury

I also agree to comply with all U.S. and international laws (including the obtaining of any required import license or authorization), and I understand and agree that Exelis Visual Information Solutions is not responsible for my failure to comply with any such laws.

You hereby certify that you are authorized to act on behalf of your company. I Agree Cancel

View our Export Compliance document for information regarding the export of the Software Product. We recommend you seek legal counsel if you have any questions regarding the legality of your use of the Software Product.

page last updated on: Wednesday June 28, 2017 11:06:27; retrieved on: Thursday June 29, 2017 03:41:40