-- JoshSchroder - 23 Sep 2014

Joshua Schroeder's Page

Tasks (2015-2016)

Log Entries (2015-2016)

October 13, 2015
  • Recently I have been working with the FDR scripts. (I have aerie on my computer, but this could also be done on the UMD cluster)
    • I started by logging in to the umd cluster and downloading the smoothed map files from /data/archive/hawcroot/maps/maps-20150710-aligned/SmoothedMaps, as well as /data/archive/hawcroot/maps/maps-20150710-aligned/cuts.txt. (scp -r your_username@pa-pub.umd.edu:/data/archive/hawcroot/maps/maps-20150710-aligned/SmoothedMaps /path/to/folder/on/local/machine/, and a similar command for the cuts file)
    • I acquired three scripts from Tolga: getHistogramData.py, drawHistogram.py, and processHistogramData.py
    • I combined the 10 bins in SmoothedMaps into one significance map using the command: aerie-apps-make-significance-map --input combined_bin0.fits.gz 0 combined_bin1.fits.gz 1 combined_bin2.fits.gz 2 combined_bin3.fits.gz 3 combined_bin4.fits.gz 4 combined_bin5.fits.gz 5 combined_bin6.fits.gz 6 combined_bin7.fits.gz 7 combined_bin8.fits.gz 8 combined_bin9.fits.gz 9 -o significance_combined_map.fits.gz --sensi cuts.txt -M 90 -m -89
    • When I plot the significance map using </path/to/aerie/src/map-maker/scripts/plotMollweide.py significance_combined_map.fits.gz> I get Sigmap.png
    • You can see the crab on the right and the galactic plane on the left as a strip of green. However, other sources are somewhat hard to see.
    • I now run Tolga's scripts over the significance map. So first I run <python getHistogramData.py --input significance_combined_map.fits.gz --output output.fits.gz>. this should output four files. I was concerned only with output.fits.gzoffplane.dat and output.fits.gzonplane.dat. I ran processHistogramData.py on both of these files (<python processHistogramData.py --input output.fits.gzoffplane.dat>), generating two more files, output.fits.gzoffplane_conservative_sigmas.dat and output.fits.gzonplane_conservative_sigmas.dat.
    • Taking a look at <less output.fits.gzonplane_conservative_sigmas.dat>, we see a list of values. The value corresponding to 0.000100 will give you the sigma cutoff after which the chance of seeing a random source at this sigma is .0001 (this is called the contamination). My value was 4.225825. This is the cut we want to place on minimum sigma for spotting sources. So to plot the significance map with this cut, I ran </path/to/aerie/src/map-maker/scripts/plotMollweide.py significance_combined_map.fits.gz -m 4.225825 -M 20> This sets the minimum sigma to the value we obtained from Tolga's scripts and the maximum to 20. This plot looks like this:sigcombinedmap2.png

      Edit: The file I should have been using to look at offplane sources is in file_name.offplane_conservative_sigmas.dat, and should have had a cutoff of about 5.49 sigma if I wanted a contamination of .0001

November 13, 2015
  • I have learned a lot more about FDR after reading a paper about it and playing around with the output. The getHistogramData.py file actually generates two outputs that are histograms of data. Each pixel on the map is determined to be on the galactic plane or off the galactic plane, and its sigma value is put into one of the bins of the histogram. This output is in the files filenameonplane.dat and filenameoffplane.dat. processHistogramData.py will give you analyze one of these histograms and give you a sigma cutoff for that region for various levels of contamination, which are organized in a table in the output file, filenameoffplane_conservative_sigmas.dat, etc. The onplane sigma values should only be used when looking at the galactic plane, and so to look for offplane sources, one should use the output of the offplane sigma file.
  • Once you have the sigma cutoff, you can use the file plotHotSpots.py to output all of the sources on the map with a sigma higher than the cutoff using the script /path/to/hawc_software/aerie/src/map-maker/scripts/plotHotSpots.py. When I ran this code with (the incorrect) 4.225 sigma cutoff, I found bizarrely bright pixels on the very edges of the map, looking like this:
    • significance_combined_map-hs1.pngsignificance_combined_map-hs3.pngsignificance_combined_map-hs7.png
    • As you can see, they are around -43 to -44 in Declination. In fact, all such suspicious sources occured within a 2 degree strip along the top or bottom of the equitorial map. Therefore, I advise removing everything above 80 deg and everything below -41 deg when making the significance map. The bright single pixel sources above do not seem to be present in the maps of the data and the background that i looked at when we zoomed in on these sources, so I suspect there is some error in the smoothing or significance map making code that is generating these.
    • All in all, I saw 15 hotspots with a sigma > 4.225 that have not been seen with other projects so far, however the cutoff I should have used is 5.49 sigma, so these candidate sources cannot be verified to 99.99% certainty with the methods I have used thus far. However, If anyone wants to examine these further, I will attach their RA and Dec coordinates here.
November 18, 2015
  • I have been installing SExtractor for some time now. It's been a journey, and I have decided to make a separate page describing how I installed SExtractor.
  • <update> the problem with SExtractor is that our fits files are missing the necessary headers to run SExtractor with, so until we change the format of our fits files, we will not be able to run sextractor. Udara was able to get sextractor to work by adding additional headers, but did not remember which headers he added.
December 14, 2015
  • I have updated the HAWC website, including changing the tenses of certain phrases ("hawc will be completed" to "hawc was completed", etc) and updating to people list. We are planning to update the photo to one of HAWC 300 and move the people list to the right side of the page and the collaborator list to the bottom, but these changes will need to be made after winter break, as I am out of time this semester
  • I did a little more FDR stuff: firstly, I reran the FDR scripts on the offplane skymap, this time masking out the Crab and the Markarians. The modification basically amounted to copying the section of Tolga's getHistogramData.py and changing the coordinates excluded from those of the galactic plane to those of a small bin around the crab. Then, you repeat for the markarians. The final script looks like this. The output sigmas are on average .25 sigma higher than before masking out the crab and the markarians, and looks like this.
  • I ran getHotSpots.py with 50% contamination or less to see what we got. The resulting sourcelist may be interesting to compare with when we get more data. The candidate sources with contamination of 10%-50% are here.Those with a contamination of <10% are listed here. The full sourcelist, including known sources is here. I didn't have time to make a separate file with only the known sources.
  • A curiousity I leave for others to figure out I discovered while looking at some HAWC 250 data. For the May 4, 2015 reconstruction, (maps-2015-aligned in the maps folder on the UMD cluster) there are two significance maps, one made with only the HAWC 250 data, and one with the HAWC 111 and the HAWC 250 data. When looking at my best candidate source in the old data, you see a small 2-3 significance bump at the location where the source forms in the HAWC 300 data for the HAWC 250 data, but when you take both the HAWC 111 data and the HAWC 250 data, the source disappears. Shown below, in order of left to right, are: HAWC 300 data, HAWC 250 data, HAWC 250 and HAWC 111 data.
    • significance_combined_map2-hs15.pngHAWC250.pngHAWC111+250.png
    • It may be that there is some asymmetry in the way we reconstructed the 111 vs the 250 data, or maybe the 111 data was anomalous enough to drown out the significance bump, but regardless, this may be interesting, or it may be nothing. Possibly something to look into later?

March 24, 2016
  • Firstly, early this semester I looked at some significance maps with the newly reconstructed data. Unfortunately, it appears that most to all of the prospective sources I saw with the prior FDR analysis shrank or disappeared entirely in the new reconstruction. Bummer.
  • I wrote and submitted an abstract for the UURAF event this April 8. For future reference, the abstract can be found here.
  • I started Dark Matter Analysis, with a comparison of Pythia 6 and Pythia 8. I followed the outline of some scripts Tolga sent me to generate plots of the lifetime vs mass curves for the sources Draco and Segue1, for the channels ttbar, bbbar, tautau, mumu, and WW for both Pythia 6 and Pythia 8. First, there is a script that converts livetime to flux; it takes in an assumed lifetime, a source, a channel, and a mass measurement and calculates what the flux should be. An iterative script I willl list below runs through the energies 1, 2, 4, 6, 10, 20, 40, 60, 100, 200, 400, 600, and 1000TeV for each source listed and each channel. Then, I us liff to perform a fit on the data to determine by what factor the lifetime would need to be scaled to give the appropriate flux for this source (I'm not solid on the nuances of how this works), which the script also does iteratively. Next, I call on a python script that runs over all the output files this makes and sorts the scale factors into an array for each channel for each source (mass in one column, scale factor in the other column), and outputs this to a file. Finally, I run a script that iteratively plots these arrays in pyplot.
  • All of the work was done on the cluster. convert_lifetime_to_flux.py.txt is the script that converted lifetime to flux using pythia 8, convert_spectra_to_flux_decay_pat.py.txt does the same thing, but using pythia 6, make_arrays.py.txt takes the outpout from these files and turns it into an array, and draw_plots.py.txt will plot those arrays (in all of these files, remove the .txt). The script bigkahoona.sh will run all of the above scripts, for all of the sources and channels listed in the array at the top as long as the above scripts are in the same directory as bigkahoona.sh (the arrays in make_arrays.py and draw_plots.py also must be modified).
  • The output of the above procedures were the following plots:
  • Draco_bbbar_Pythia_lifetime_comparison.pngDraco_mumu_Pythia_lifetime_comparison.pngDraco_tautau_Pythia_lifetime_comparison.png
  • Draco_ttbar_Pythia_lifetime_comparison.pngSegue1_WW_Pythia_lifetime_comparison.pngSegue1_bbbar_Pythia_lifetime_comparison.png
  • Segue1_mumu_Pythia_lifetime_comparison.pngSegue1_tautau_Pythia_lifetime_comparison.pngSegue1_ttbar_Pythia_lifetime_comparison.png
  • The above procedure failed to generate the scale factors for Draco WW for pythia 6, and so it isn't shown above. Also, the plot for Segue1 WW looks strange, as it has a larger disparity than any of the other channels for pythia 6 vs pythia 8.

April 28, 2016
  • I made my presentation for UURAF on FDR and gave it. I have attached the pdf of the slideshow here.
  • I tried to run the FDR again on the newest maps, but while the onplane output of getHistogramData.py was fine, the Offplane data ouput a histogram with only sigma values that were absurdly low (as in negative 3 trillion sigma). I am not sure what went wrong.
  • This last week I started a small FDR project to look at the TeVCat list and see what the sigma and FDR cutoffs for this list are. After some phenagling, Tolga and I got the Tevcat list loaded with its RA and DEC coordinates, and managed to extract their sigma. (see here for the tevcat list and here for the output of this script that got the sigma at each tevcat source (make sure to set a small radius, such as .001)) For sources out of the HAWC field of view, it has a sigma of -3275********* (very large number). I have not had time to make any plots with this yet.


New shift wizard list 2/3/15

  • water levels link
  • graphs (good)
    • TDC error rate
    • TDC error ratio
    • max PE
    • log NPE
    • Reconstructed Core
    • azimuth angle
    • zenith angle
    • Dec vs Ra
    • edge hits
  • TDC error rate
  • TDC trigger rate
  • Status of HVAC
  • input/output voltages and temp of UPS
  • Status of GTC
  • Weather/temperature
  • temp in counting house, main computer rack, and FEB1 crate: Analog Front + glance at others
  • TDC and scalar disk space

Event Display

  • Slideshow (done)
  • New poster
  • Boot up automatically (Fixed)
  • Sam's List

Things to fix
  • Translate data into more readable form (fixed)
  • Update HV editing page (fixed)
  • Set warning and alarm thresholds (fixed)

Fix up HAWC website

Things to fix
  • header image? (fixed)
  • Milagro links

Set up Shift Wizard

Things to fix
  • Can't find TDC DAQ on monitor (fixed)
  • HV Link doesn't seem to work (fixed)
  • LV link doesn't show status (fixed)
  • Disk space link doesn't seem to show status (fixed)
  • Weather/temperature link doesn't seem to work
  • Need data for status of GTC (doesn't exist yet)
  • Need data for input/output voltages and temperature of UPS (Doesn't exist yet)
  • Need data for status and temperature of HVAC (Doesn't exist yet)
  • logNPE plot link doesn't seem to work (fixed)
  • Main Dashboard link fix (fixed)
  • Reference plots

List of Log Entries (2014-2015)

September 23, 2014
  • Tried to figure out the wiki
  • Played with the Shift Wizard
September 24, 2014 September 29, 2014
  • Finished reading the HAWC Construction Proposal
  • Read how to to edit the shift wizard here: I need an Admin account before I can edit it.
  • Added a few items to the Shift Wizard problem list
September 30, 2014
  • Practiced coding on Codecademy
  • Read part of A Thin Cosmic Rain

October 6, 2014
  • Practiced coding
  • Read more of A Thin Cosmic Rain
  • Read how to edit the HAWC website here: did not have username/password that the server would accept when I tried to log in.
October 7, 2014
  • Resolved some issues with Shiftwizard, as noted under the shiftwizard heading
October 13, 2014
  • Did some background reading
October 15, 2014 October 21, 2014
  • Logged into HAWC public website server and tried to figure out how to edit it
October 22, 2014
  • Started entering HV tank names and set values
  • Downloaded web editor
October 27, 2014
  • Continued entering HV tank names and set values
October 28, 2014
  • Finished entering HV tank names and set values
  • Updated the page describing how to edit the HV Measurements
November 5, 2014
  • Fixed up the MSU HAWC public website
November 12, 2014
  • Made further changes to the MSU HAWC public website
January 14, 2015
  • got the event display to begin displaying events automatically on reboot. I used the crontab to change this by entering the command: sudo crontab -e. From there, you enter the time you want the command to execute followed by the command. I went ahead and entered the command to run on boot for both the superuser and the "hawc" user, just in case.
  • I have been working on a slideshow to go with the event display.
January 27, 2015
  • finished the powerpoint presentation.
Feb 3, 2015
  • Tolga showed me how to enter questions into the new shift wizard, and I have entered some questions
  • Creating new to do list for shiftwizard.
Apr 6, 2015
  • Edited the poster for the Event Display. The new poster should be in the HAWC dropbox
Topic attachments
I Attachment Action Size Date Who Comment
Draco_bbbar_Pythia_lifetime_comparison.pngpng Draco_bbbar_Pythia_lifetime_comparison.png manage 39 K 24 Mar 2016 - 16:07 JoshSchroder  
Draco_mumu_Pythia_lifetime_comparison.pngpng Draco_mumu_Pythia_lifetime_comparison.png manage 35 K 24 Mar 2016 - 16:07 JoshSchroder  
Draco_tautau_Pythia_lifetime_comparison.pngpng Draco_tautau_Pythia_lifetime_comparison.png manage 36 K 24 Mar 2016 - 16:07 JoshSchroder  
Draco_ttbar_Pythia_lifetime_comparison.pngpng Draco_ttbar_Pythia_lifetime_comparison.png manage 37 K 24 Mar 2016 - 16:07 JoshSchroder  
HAWC111+250.pngpng HAWC111+250.png manage 115 K 14 Dec 2015 - 19:54 JoshSchroder  
HAWC250.pngpng HAWC250.png manage 131 K 14 Dec 2015 - 19:54 JoshSchroder  
HAWC_abstract.docxdocx HAWC_abstract.docx manage 91 K 24 Mar 2016 - 16:09 JoshSchroder  
HAWC_sticker.pngpng HAWC_sticker.png manage 47 K 14 Apr 2016 - 16:26 JoshSchroder This is test
Segue1_WW_Pythia_lifetime_comparison.pngpng Segue1_WW_Pythia_lifetime_comparison.png manage 39 K 24 Mar 2016 - 16:08 JoshSchroder  
Segue1_bbbar_Pythia_lifetime_comparison.pngpng Segue1_bbbar_Pythia_lifetime_comparison.png manage 40 K 24 Mar 2016 - 16:08 JoshSchroder  
Segue1_mumu_Pythia_lifetime_comparison.pngpng Segue1_mumu_Pythia_lifetime_comparison.png manage 39 K 24 Mar 2016 - 16:08 JoshSchroder  
Segue1_tautau_Pythia_lifetime_comparison.pngpng Segue1_tautau_Pythia_lifetime_comparison.png manage 36 K 24 Mar 2016 - 16:08 JoshSchroder  
Segue1_ttbar_Pythia_lifetime_comparison.pngpng Segue1_ttbar_Pythia_lifetime_comparison.png manage 37 K 24 Mar 2016 - 16:26 JoshSchroder  
Sigmap.pngpng Sigmap.png manage 437 K 13 Oct 2015 - 15:13 JoshSchroder Significance map for 10 bins before cuts
UURAF_Presentationv2.2.pdfpdf UURAF_Presentationv2.2.pdf manage 2 MB 28 Apr 2016 - 15:51 JoshSchroder  
bigkahoona.shsh bigkahoona.sh manage 2 K 25 Mar 2016 - 18:01 JoshSchroder  
candidate_above_.1contamination.datdat candidate_above_.1contamination.dat manage 10 K 14 Dec 2015 - 19:47 JoshSchroder  
candidate_below_.1contamination.datdat candidate_below_.1contamination.dat manage 1 K 14 Dec 2015 - 19:47 JoshSchroder  
convert_lifetime_to_flux.py.txttxt convert_lifetime_to_flux.py.txt manage 4 K 24 Mar 2016 - 16:06 JoshSchroder  
convert_spectra_to_flux_decay_pat.py.txttxt convert_spectra_to_flux_decay_pat.py.txt manage 3 K 24 Mar 2016 - 16:06 JoshSchroder  
cutoutputoffplane_conservative_sigmas.datdat cutoutputoffplane_conservative_sigmas.dat manage 126 bytes 14 Dec 2015 - 19:45 JoshSchroder  
draw_plots.py.txttxt draw_plots.py.txt manage 1 K 24 Mar 2016 - 16:06 JoshSchroder  
getPixelData2.py.txttxt getPixelData2.py.txt manage 3 K 28 Apr 2016 - 16:01 JoshSchroder  
getcutHistogramData.py.txttxt getcutHistogramData.py.txt manage 6 K 14 Dec 2015 - 19:40 JoshSchroder  
goodsources.datdat goodsources.dat manage 916 bytes 16 Nov 2015 - 18:58 JoshSchroder  
make_arrays.py.txttxt make_arrays.py.txt manage 3 K 24 Mar 2016 - 16:07 JoshSchroder  
out3EXT out3 manage 6 K 28 Apr 2016 - 15:58 JoshSchroder  
sigcombinedmap2.pngpng sigcombinedmap2.png manage 55 K 13 Oct 2015 - 15:29 JoshSchroder Significance map for 10 bins after cuts
significance_combined_map-hs1.pngpng significance_combined_map-hs1.png manage 173 K 16 Nov 2015 - 18:46 JoshSchroder  
significance_combined_map-hs3.pngpng significance_combined_map-hs3.png manage 188 K 16 Nov 2015 - 18:46 JoshSchroder  
significance_combined_map-hs7.pngpng significance_combined_map-hs7.png manage 222 K 16 Nov 2015 - 18:46 JoshSchroder  
significance_combined_map2-hotspots-sig.gt.2.8814.datdat significance_combined_map2-hotspots-sig.gt.2.8814.dat manage 13 K 14 Dec 2015 - 19:48 JoshSchroder  
significance_combined_map2-hs15.pngpng significance_combined_map2-hs15.png manage 423 K 14 Dec 2015 - 19:56 JoshSchroder  
tevcat_list.datdat tevcat_list.dat manage 5 K 28 Apr 2016 - 15:57 JoshSchroder  
Topic revision: r32 - 28 Apr 2016, JoshSchroder
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback