User Tools

Site Tools


wg:pipeline

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

wg:pipeline [2018/11/20 06:36]
yjyun created
wg:pipeline [2019/04/11 10:23]
Line 1: Line 1:
-====== KVN Pipeline ====== 
- 
-//Current version: 1.0.// 
- 
----- 
- 
-Jeff Hodgson - (KASI, jhodgson@kasi.re.kr) 
- 
-<WRAP center round download 60%> 
-[[http://​radiowiki.kasi.re.kr/​radiowiki/​lib/​exe/​fetch.php?​media=kvnpipe.zip|KVNPipe.zip]] 
-</​WRAP>​ 
- 
- 
-===== Description ===== 
- 
-The KVN Pipeline is a ParselTongue based pipeline for the automatic reduction of VLBI data obtained using the [[http://​kvn-web.kasi.re.kr/​en/​en_normal_info.php|KVN]]. Currently the KVN Pipeline operates in single polarisation only, but automatically handles simultaneous frequency data. For a full description of the Pipeline, see PAPER. If you use the pipeline, please add a citation to PAPER. 
- 
-===== Installation and Requirements ===== 
- 
-The KVN Pipeline requires both ParselTongue and AIPS to be installed. 
- 
-  * Installation instructions for ParselTongue can be found [[http://​www.jive.nl/​jivewiki/​doku.php?​id=parseltongue:​parseltongue|here]]. 
-  * Installation instructions for AIPS can be found [[http://​www.aips.nrao.edu/​dec16.shtml|here]]. 
- 
-The KVN Pipeline consists of two files: 
- 
-  * ** KVNpipeline_v1_0.py **  
-  * ** KVN_Pipeline_Control_File.py ** 
- 
-The KVN Pipeline can then be run from the command line as: 
-<​code>​ 
-ParselTongue KVN_Pipeline_Control_File.py 
-</​code>​ 
- 
-===== Operation ===== 
- 
-The KVN Pipeline is contained within the ** KVNpipeline_v1_0.py ** file, which also contains many function based from the [[http://​www.evlbi.org/​pipeline/​user_expts.html|EVN Pipeline]]. This file is not directly run by the user but is imported into a separate script (** KVN_Pipeline_Control_File.py **) which //is// run. The Control File is only an example and you can easily create your own. The control file requires that you import the KVN Pipeline and some ParselTongue functions: 
-<​code>​ 
-import KVNpipeline_v1_0 as KVNpipeline 
-from AIPSTask import AIPSTask, AIPSList 
-</​code>​ 
-Due to the way this operates, if you upgrade to a newer version of the Pipeline, simply change the version imported here, as new versions should be backwards compatible. ​ 
- 
-The minimum required inputs are: 
- 
-  * AIPS user number 
-  * Input FITS file 
-  * Experiment name 
-  * ANTAB files. The script should automatically handle changing station names. Files should be included within a single array (see example). 
-  * Directory to output files. 
- 
-There are many optional parameters which can be specified. These can be useful for speeding up the pipeline when there are problems in the data etc. and the data must not need be re-loaded from scratch again. 
- 
-^ Parameter ​  ^ Description ^ 
-| LOAD_DATA="​y" ​     | skip loading data | 
-| AMP_CAL="​y" ​       | skip amplitude calibrations | 
-| PLOT="​y" ​          | do not create plots | 
-| ACCOR="​y" ​         | do not perform autocorrelation corrections | 
-| BPASS="​y" ​         | do not perform bandpass corrections | 
-| UVSPLIT="​y" ​       | do not split the '​big'​ file into single frequencies | 
-| IFALIGN="​y" ​       | do not perform the manual phase calibrations | 
-| BIG_FRING="​y" ​     | do not perform the big fringe fit | 
-| PFT="​y" ​           | do not perform the frequency phase transfer | 
-| SPLIT="​y" ​         | do not split the data into individual sources and export them | 
-| EXPORT="​y" ​        | do not export the big unsplitted datasets | 
-| CLEAN_UP="​y" ​      | do not remove AIPS catalogs created by the script | 
-| SECOND_IFALIGN="​y"​ | do no perform the second manual phase cal step to remove additional station based delays | 
-| FITLD_SOURCES=AIPSList(["​ ","​ "])) | Put sources in here if only want to load a subset of the observed sources | 
-| INITIAL_FRING="​n" ​ | Run an initial global FRING, this can sometimes help with the IF-Aligner, default is off | 
-| NO_PFT_MODE="​n" ​   | Run the scipt without performing Phase transfer. Default is off. | 
-| FRING_SOLINT=5 ​    | Set the solution interval for global FRING runs. Default is 6 mins (scan length for KVN). | 
-| FRING_SNR=6 ​       | Set the SNR limit for the global FRING. Default 5. | 
-| MAN_PCAL="​n" ​      | Do standard manual phase cal. If "​y",​ must also set the timerrange in MAN_PCAL_TIMERANGE. If MAN_PCAL is set, IF-Aligner will not be performed | 
-| MAN_PCAL_TIMERANGE=AIPSList([0,​0,​0,​0,​0,​0,​0,​0]) | Define the time-range used for Manual phase cal. Entered in standard AIPS format. | 
- 
-===== Operating via SSH ===== 
- 
-It is easy to run the script via SSH, but you must have ParselTongue and AIPS installed on the remote computer. If they are installed, make sure they are sourced with the following commands: 
- 
-<​code>​ 
-source /​home/​aips/​LOGIN.SH 
-</​code>​ 
- 
-For your AIPS installation,​ and: 
- 
-<​code>​ 
-export PATH=$PATH:/​opt/​parseltongue-2.1/​bin 
-</​code>​ 
- 
-For ParselTongue. The above examples will work for the //earth// shared cluster at KASI. The exact paths may differ on other machines. 
- 
-===== Example ===== 
- 
-An example of how to code the Control File: 
- 
-It is good practice to set up the inputs using arrays: 
-<​code>​ 
-FITS = ["/​home/​test/​AIPS/​earth/​AIPS/​FITS/​IMOGABA32.FITS"​] 
-NAMES = ["​IMO32Y"​] 
-ANTAB = [["/​home/​test/​AIPS/​earth/​AIPS/​FITS/​p16sl01bKTN.ANTAB.1",​ 
-"/​home/​test/​AIPS/​earth/​AIPS/​FITS/​p16sl01bKUS.ANTAB.1",​ 
-"/​home/​test/​AIPS/​earth/​AIPS/​FITS/​p16sl01bKYS.ANTAB.1"​]] 
-EXPORT = ["/​home/​test/​AIPS/​earth/​AIPS/​FITS/​TEST1/​IMO32_TESTING1"​] 
-</​code>​ 
- 
-With the KVN Pipeline then being run as: 
-<​code>​ 
-KVNpipeline.KVN_pipeline(502,​FITS[0],​NAMES[0],​ANTAB[0],​EXPORT[0]) 
-</​code>​ 
-This will run the KVN Pipeline with default settings (performing the frequency phase transfer, IF-aligning and exporting data). Various parameters can be turned on and off at the users desire: 
-<​code>​ 
-KVNpipeline.KVN_pipeline(502,​FITS[0],​NAMES[0],​ANTAB[0],​EXPORT[0],​REFANT=2,​LOAD_DATA="​y",​AMP_CAL="​y",​ACCOR="​y",​BPASS="​y",​UVSPLIT="​y",​PLOT="​y",​CLEAN_UP="​y",​INITIAL_FRING="​n",​NO_PFT_MODE="​n",​FRING_SOLINT=5.5,​FITLD_SOURCES=AIPSList(["​3C454.3","​BLLAC"​]),​SPLIT="​n",​EXPORT="​n",​MAN_PCAL="​y",​MAN_PCAL_TIMERANGE=AIPSList([0,​6,​5,​0,​0,​6,​11,​0])) 
-</​code>​ 
-In this case, the KVN Pipeline will load data, perform amplitude calibrations,​ ACCOR and BPASS, split the data into smaller parts, but not perform an initial fringe-fit, only load two sources, and perform a manual phase calibration rather than using the IF-Aligner. It will also not export the data. 
- 
-The advantage of putting the inputs into an array is that multiple datasets can then be easily run in a for-loop: 
- 
-<​code>​ 
- 
-INPATH = "/​home/​test/​AIPS/​earth/​AIPS/​FITS"​ 
-OUTPATH = "/​home/​test/​AIPS/​earth/​AIPS/​FITS/​TEST1"​ 
- 
-FITS = [INPATH+"/​IMOGABA24.FITS",​INPATH+"/​IMOGABA26.FITS",​INPATH+"/​IMOGABA27.FITS",​INPATH+"/​IMOGABA28.FITS",​ 
-        #​INPATH+"/​IMOGABA29.FITS",​INPATH+"/​IMOGABA30.FITS",​INPATH+"/​IMOGABA31.FITS",​INPATH+"/​IMOGABA32.FITS"​] 
-NAMES = ["​IMO24A","​IMO26","​IMO27","​IMO28","​IMO29","​IMO30","​IMO31","​IMO32"​] 
-ANTAB = [[INPATH+"/​p15sl01c"​+"​KTN.ANTAB.1",​INPATH+"/​p15sl01c"​+"​KUS.ANTAB.1",​INPATH+"/​p15sl01c"​+"​KYS.ANTAB.1"​],​ 
-         ​[INPATH+"/​p15sl01g"​+"​KTN.ANTAB.1",​INPATH+"/​p15sl01g"​+"​KUS.ANTAB.1",​INPATH+"/​p15sl01g"​+"​KYS.ANTAB.1"​],​ 
-         ​[INPATH+"/​p15sl01h"​+"​KTN.ANTAB.1",​INPATH+"/​p15sl01h"​+"​KUS.ANTAB.1",​INPATH+"/​p15sl01h"​+"​KYS.ANTAB.1"​],​ 
-         ​[INPATH+"/​p15sl01i"​+"​KTN.ANTAB.1",​INPATH+"/​p15sl01i"​+"​KUS.ANTAB.1",​INPATH+"/​p15sl01i"​+"​KYS.ANTAB.1"​],​ 
-         ​[INPATH+"/​p15sl01j"​+"​KTN.ANTAB.1",​INPATH+"/​p15sl01j"​+"​KUS.ANTAB.1",​INPATH+"/​p15sl01j"​+"​KYS.ANTAB.1"​],​ 
-         ​[INPATH+"/​p15sl01k"​+"​KTN.ANTAB.1",​INPATH+"/​p15sl01k"​+"​KUS.ANTAB.1",​INPATH+"/​p15sl01k"​+"​KYS.ANTAB.1"​],​ 
-         ​[INPATH+"/​p16sl01a"​+"​KTN.ANTAB.1",​INPATH+"/​p16sl01a"​+"​KUS.ANTAB.1",​INPATH+"/​p16sl01a"​+"​KYS.ANTAB.1"​],​ 
-         ​[INPATH+"/​p16sl01b"​+"​KTN.ANTAB.1",​INPATH+"/​p16sl01b"​+"​KUS.ANTAB.1",​INPATH+"/​p16sl01b"​+"​KYS.ANTAB.1"​]] 
-EXPORT = [OUTPATH+"/​IMO24A",​OUTPATH+"/​IMO26",​OUTPATH+"/​IMO27",​OUTPATH+"/​IMO28",​OUTPATH+"/​IMO29",​OUTPATH+"/​IMO30",​OUTPATH+"/​IMO31",​OUTPATH+"/​IMO32"​] 
- 
-for i in range(len(FITS)):​ 
-   try: 
-      KVNpipeline.KVN_pipeline(502,​FITS[i],​NAMES[i],​ANTAB[i],​EXPORT[i],​REFANT=2,​LOAD_DATA="​y",​AMP_CAL="​y",​ACCOR="​y",​BPASS="​y",​UVSPLIT="​y",​PLOT="​y",​CLEAN_UP="​y",​ 
-     IFALIGN="​y",​INITIAL_FRING="​n",​NO_PFT_MODE="​n",​FRING_SOLINT=5.5,​FITLD_SOURCES=AIPSList(["​3C454.3","​BLLAC"​]),​SPLIT="​n",​EXPORT="​n",​ 
-     MAN_PCAL="​y",​MAN_PCAL_TIMERANGE=AIPSList([0,​6,​5,​0,​0,​6,​11,​0])) 
-   ​except Exception: 
-      print "​Something went wrong with:",​NAMES[i] 
-      pass 
- 
-</​code>​ 
- 
-Then running: 
-<​code>​ 
-ParselTongue KVN_Pipeline_Control_File.py 
-</​code>​ 
-will execute the pipeline. 
- 
-===== Outputs ===== 
- 
-By default, the pipeline outputs the following files (where <EXP> is the experiment name): 
- 
-^ Output ^ Desciption ^ 
-| <​EXP>​15MM.FITTP | The pre "​splitted"​ 15 mm dataset | 
-| <​EXP>​7MM.FITTP | The pre "​splitted"​ 7 mm dataset | 
-| <​EXP>​3MM.FITTP | The pre "​splitted"​ 3 mm dataset | 
-| <​EXP>​2MM.FITTP | The pre "​splitted"​ 2 mm dataset | 
-| <​EXP>​ScanSummary | LISTR output | 
-| <​ANTABFILE>​.orig | The original pre-editing ANTAB files | 
- 
-In addition, it creates two directories:​ 
- 
-^ Directory ^ Description ^ 
-| plots | Output of plots at every stage of the pipeline | 
-| SPLIT | Location of output source files after SPLIT | 
- 
-In the SPLIT directory, it is further subdivided into separate directories for each frequency. ​ 
- 
-===== Feedback and Contact ===== 
- 
-Feel free to use the pipeline and report any issues back to me. If you have any suggestions for new features, also send me an email. 
- 
-jhodgson@kasi.re.kr 
- 
-2016 - Jeff Hodgson 
- 
- 
  
wg/pipeline.txt ยท Last modified: 2019/04/11 10:23 (external edit)