Difference: LogBookTCR (1 vs. 10)

Revision 10
05 Feb 2018 - Main.TomasCap
Line: 1 to 1
 
META TOPICPARENT name="TomasCap"
-- TomasCap - 25 Aug 2017 This is my logbook

August

Week 1

  • Check in at MSU office.
Changed:
<
<
  • Get the Spartan card. It helps me to access to building and office.
  • AERIE installation (in my personal computer) was solved.
>
>
  • Get the Spartan card. It helps me to access building and office.
  • AERIE installation (on my personal computer) was solved.
 

Week 2

  • A network with 15 inputs was trained for distinguishing from gamma to hadrons.
Line: 14 to 14
 
  • Create a module that computes the network's output.

Goals from August, 14th to August 25th (Week 3 & 4)

  1. Create a Foswiki
Changed:
<
<
  1. Made an analysis with the most important variable (ranking).
>
>
  1. Made an analysis of the most important variable (ranking).
 
  1. Choose the most significant variables and use them as inputs to the network.
Changed:
<
<
  1. Make a Crab maps with the networks trained
>
>
  1. Make a Crab map with the networks trained
 
  1. Train a network using real data as Background and MC as Signal. Compare the results

Week 3

  • Three networks were trained with 4,5 & 6 inputs.
Changed:
<
<
  • Optimals cuts were found and made a Crab maps.
>
>
  • Optimal cuts were found and made Crab maps.
 

Week 4

  • A network was trained with all variables of HAWC data stream in order to obtain the ranking of these variables (MC were used).
Line: 44 to 44
 
  • Train other BDT.

Week 1

Changed:
<
<
  • Make an analysis between Training data vs. Verification data.
>
>
  • Make an analysis of Training data vs. Verification data.
 
  • Create a code that can make Crab map using different months of the year.
  • Using the NN that was trained with MC data, Significance maps were done with data of 2015 and 2017.

Goal from September 11th to 15th

Changed:
<
<
  • Compare the variable between Training data set vs. Testing data set in oder to look for which variable use as input of the machine learning.
>
>
  • Compare the variable between Training dataset vs. Testing dataset in order to look for which variable use as the input of the machine learning.
 
  • Train a NN and BDT, using MC as signal and real data as Bkg.

Week2

Changed:
<
<
  • In the comparation of Training and verification data. On one hand we compara Bkg vs. Bkg, we need to exclude the data around the crab nebula, and the other hand, the data around the crab nebula are used to compare with signal (MC)
>
>
  • In the comparison of Training and verification data. On one hand we compare Bkg vs. Bkg, we need to exclude the data around the crab nebula, and the other hand, the data around the crab nebula are used to compare with signal (MC)
 

Goal from September 18 to 22

  • Doing an analysis in order to choose which variables will be used to G/H sep
  • Train a NN and BDT.
Line: 69 to 69
 

Goal from September 25 to 29

  • Use mc.delAngle in the training, verification and testing stage.
  • Compare my results with John's cut and Bredran's cut.
Changed:
<
<
  • Choose a potencial parameter.
>
>
  • Choose a potential parameter.
 

Week 4

  • Slide of the compare NN, RC and SC 09/26/2017
  • mc.delAngle is used for training a NN and looking for an optimal Rectangle cut.
Changed:
<
<
  • A comparation of using delAngle and do not use it is made.
  • Comparation of John's cut adn Breand's cut vs. my NN is started but it does not complete.
>
>
  • A comparison of using delAngle and do not use it is made.
  • Comparison of John's cut and Breand's cut vs. my NN is started but it does not complete.
 

October

Goal from October 2 to 6

  • Compare my results with John's cut and Bredran's cut.
Changed:
<
<
  • Choose a potencial parameter.
  • Use these parameter as input of NN.
>
>
  • Choose a potential parameter.
  • Use these parameters as the inputs of NN.
 

Week 1

  • Slide of the compare using mc.delAngle and does not use it 10/03/2017
Added:
>
>

Summary of the month from November 2017 and January 2018

This is the list of the presentation in the MSU-HAWC Meeting:

The main results were:

  • Search how John computes his cuts. (See 01/12/2018)
  • Use RC method and compute one cut in each variable per fbin and ebin.
  • Compare the cuts that we get versus John's cuts (See 01/26/2018).
    • In this task, we learned that we depend on the data that we used in the training stage.
  • We should verify each input variable in order to be confident that the MC data has the same distribution on the real data.
  • In the last results using Neural Networks, we see in some bins we have a bit increased when we used MC data but when the Significance Crab maps are made, in some bins this increase don't see reflected.

February 2018

Monthly goal

  • Make a comparison of MC Vs. Real data using the most recent simulation (take 4) and real data of 2017.
  • Select potencial variable to use as input parameter.
  • Start to work a new variable or do a version 2 of disMax
  • Explore other option to train like LLP (Learning from label proportions), CWoLa ( Clasification without labels), etc.
 

META FILEATTACHMENT attachment="BringingToTheTable.pdf" attr="" comment="What things I did about gamma/hadron separation using neural network" date="1503696663" name="BringingToTheTable.pdf" path="BringingToTheTable.pdf" size="1126732" user="TomasCap" version="1"
META FILEATTACHMENT attachment="20170818_ChooseFeatures.pdf" attr="" comment="" date="1503698764" name="20170818_ChooseFeatures.pdf" path="20170818_ChooseFeatures.pdf" size="2632027" user="TomasCap" version="1"
Revision 9
05 Oct 2017 - Main.TomasCap
Line: 1 to 1
 
META TOPICPARENT name="TomasCap"
-- TomasCap - 25 Aug 2017 This is my logbook
Line: 72 to 72
 
  • Choose a potencial parameter.

Week 4

Added:
>
>
  • mc.delAngle is used for training a NN and looking for an optimal Rectangle cut.
  • A comparation of using delAngle and do not use it is made.
  • Comparation of John's cut adn Breand's cut vs. my NN is started but it does not complete.

October

Goal from October 2 to 6

  • Compare my results with John's cut and Bredran's cut.
  • Choose a potencial parameter.
  • Use these parameter as input of NN.

Week 1

  • Slide of the compare using mc.delAngle and does not use it 10/03/2017
 

META FILEATTACHMENT attachment="BringingToTheTable.pdf" attr="" comment="What things I did about gamma/hadron separation using neural network" date="1503696663" name="BringingToTheTable.pdf" path="BringingToTheTable.pdf" size="1126732" user="TomasCap" version="1"
META FILEATTACHMENT attachment="20170818_ChooseFeatures.pdf" attr="" comment="" date="1503698764" name="20170818_ChooseFeatures.pdf" path="20170818_ChooseFeatures.pdf" size="2632027" user="TomasCap" version="1"
Revision 8
26 Sep 2017 - Main.TomasCap
Line: 1 to 1
 
META TOPICPARENT name="TomasCap"
-- TomasCap - 25 Aug 2017 This is my logbook
Line: 59 to 59
 
  • Doing an analysis in order to choose which variables will be used to G/H sep
  • Train a NN and BDT.

Week 3

Changed:
<
<
  • Slide of the compare Training and verification data. Result of a NN and BDT trained: 09/19/2017
>
>
  • Slide of the compare Training and verification data. Result of an NN and BDT trained: 09/19/2017
  • The rectangle cut (RC) was applied using Compactness and PINCness, in order to check if the phases to train is correct.
  • A comparison is done using RC, NN, and SC.
  • There are three sets: Training, Verification, and Testing.
    • Training set is used to fit the model.
    • Validation set is used to estimate prediction error for model selection.
    • Test set is used for assessment of the generalization error of the final chosen model.

Goal from September 25 to 29

  • Use mc.delAngle in the training, verification and testing stage.
  • Compare my results with John's cut and Bredran's cut.
  • Choose a potencial parameter.

Week 4

 

META FILEATTACHMENT attachment="BringingToTheTable.pdf" attr="" comment="What things I did about gamma/hadron separation using neural network" date="1503696663" name="BringingToTheTable.pdf" path="BringingToTheTable.pdf" size="1126732" user="TomasCap" version="1"
META FILEATTACHMENT attachment="20170818_ChooseFeatures.pdf" attr="" comment="" date="1503698764" name="20170818_ChooseFeatures.pdf" path="20170818_ChooseFeatures.pdf" size="2632027" user="TomasCap" version="1"
Revision 7
19 Sep 2017 - Main.TomasCap
Line: 1 to 1
 
META TOPICPARENT name="TomasCap"
-- TomasCap - 25 Aug 2017 This is my logbook
Line: 65 to 65
 
META FILEATTACHMENT attachment="20170818_ChooseFeatures.pdf" attr="" comment="" date="1503698764" name="20170818_ChooseFeatures.pdf" path="20170818_ChooseFeatures.pdf" size="2632027" user="TomasCap" version="1"
META FILEATTACHMENT attachment="20170905BDT.pdf" attr="" comment="The first results using Boost Decision Tree" date="1504626635" name="20170905BDT.pdf" path="20170905BDT.pdf" size="3159920" user="TomasCap" version="1"
META FILEATTACHMENT attachment="20170912_MapsBin0And1.pdf" attr="" comment="Maps of the Crab nebula in Bin 0 and 1. Compare the maps of Standartd cut vs NN that was trained using MC data" date="1505768932" name="20170912_MapsBin0And1.pdf" path="20170912_MapsBin0And1.pdf" size="1855141" user="TomasCap" version="1"
Added:
>
>
META FILEATTACHMENT attachment="20170919_compareTAndV.pdf" attr="" comment="Presentation of Sep 19th, 2017" date="1505833546" name="20170919_compareTAndV.pdf" path="20170919_compareTAndV.pdf" size="5161489" user="TomasCap" version="1"
Revision 6
18 Sep 2017 - Main.TomasCap
Line: 1 to 1
 
META TOPICPARENT name="TomasCap"
-- TomasCap - 25 Aug 2017 This is my logbook
Line: 44 to 44
 
  • Train other BDT.

Week 1

Changed:
<
<
  • Make a faster analysis between Training data vs. Verification data.
  • With the NN trained with MC data, Significance maps were done with data of 2015 and 2017. *
>
>
  • Make an analysis between Training data vs. Verification data.
  • Create a code that can make Crab map using different months of the year.
  • Using the NN that was trained with MC data, Significance maps were done with data of 2015 and 2017.

Goal from September 11th to 15th

  • Compare the variable between Training data set vs. Testing data set in oder to look for which variable use as input of the machine learning.
  • Train a NN and BDT, using MC as signal and real data as Bkg.

Week2

  • Crab maps in the Bin 0 using NN (with 15 inputs) that was trained with MC data.
  • A NN and simple BDT was trained with 10 inputs: Compactness, rec.PINCness, rec.planeChi2, rec.SFCFChi2, rec.logNPE, rec.CxPE40SPTime, rec.LDFAge, rec.LDFAmp, rec.LDFChi2 and rec.disMax
  • Make some Crab maps to check if the NN or BDT has a good performance.
  • In the comparation of Training and verification data. On one hand we compara Bkg vs. Bkg, we need to exclude the data around the crab nebula, and the other hand, the data around the crab nebula are used to compare with signal (MC)

Goal from September 18 to 22

  • Doing an analysis in order to choose which variables will be used to G/H sep
  • Train a NN and BDT.

Week 3

  • Slide of the compare Training and verification data. Result of a NN and BDT trained: 09/19/2017
 

META FILEATTACHMENT attachment="BringingToTheTable.pdf" attr="" comment="What things I did about gamma/hadron separation using neural network" date="1503696663" name="BringingToTheTable.pdf" path="BringingToTheTable.pdf" size="1126732" user="TomasCap" version="1"
META FILEATTACHMENT attachment="20170818_ChooseFeatures.pdf" attr="" comment="" date="1503698764" name="20170818_ChooseFeatures.pdf" path="20170818_ChooseFeatures.pdf" size="2632027" user="TomasCap" version="1"
META FILEATTACHMENT attachment="20170905BDT.pdf" attr="" comment="The first results using Boost Decision Tree" date="1504626635" name="20170905BDT.pdf" path="20170905BDT.pdf" size="3159920" user="TomasCap" version="1"
Added:
>
>
META FILEATTACHMENT attachment="20170912_MapsBin0And1.pdf" attr="" comment="Maps of the Crab nebula in Bin 0 and 1. Compare the maps of Standartd cut vs NN that was trained using MC data" date="1505768932" name="20170912_MapsBin0And1.pdf" path="20170912_MapsBin0And1.pdf" size="1855141" user="TomasCap" version="1"
Revision 5
18 Sep 2017 - Main.TomasCap
Line: 1 to 1
 
META TOPICPARENT name="TomasCap"
-- TomasCap - 25 Aug 2017 This is my logbook
Line: 44 to 44
 
  • Train other BDT.

Week 1

Added:
>
>
  • Make a faster analysis between Training data vs. Verification data.
  • With the NN trained with MC data, Significance maps were done with data of 2015 and 2017. *
 

META FILEATTACHMENT attachment="BringingToTheTable.pdf" attr="" comment="What things I did about gamma/hadron separation using neural network" date="1503696663" name="BringingToTheTable.pdf" path="BringingToTheTable.pdf" size="1126732" user="TomasCap" version="1"
META FILEATTACHMENT attachment="20170818_ChooseFeatures.pdf" attr="" comment="" date="1503698764" name="20170818_ChooseFeatures.pdf" path="20170818_ChooseFeatures.pdf" size="2632027" user="TomasCap" version="1"
Revision 4
05 Sep 2017 - Main.TomasCap
Line: 1 to 1
 
META TOPICPARENT name="TomasCap"
-- TomasCap - 25 Aug 2017 This is my logbook
Line: 43 to 43
 
  • Repeat the analysis of variable importance but now with the BDT.
  • Train other BDT.

Week 1

Added:
>
>
 

META FILEATTACHMENT attachment="BringingToTheTable.pdf" attr="" comment="What things I did about gamma/hadron separation using neural network" date="1503696663" name="BringingToTheTable.pdf" path="BringingToTheTable.pdf" size="1126732" user="TomasCap" version="1"
META FILEATTACHMENT attachment="20170818_ChooseFeatures.pdf" attr="" comment="" date="1503698764" name="20170818_ChooseFeatures.pdf" path="20170818_ChooseFeatures.pdf" size="2632027" user="TomasCap" version="1"
Revision 3
05 Sep 2017 - Main.TomasCap
Line: 1 to 1
 
META TOPICPARENT name="TomasCap"
-- TomasCap - 25 Aug 2017 This is my logbook
Line: 9 to 9
 

Week 2

Changed:
<
<
  • A network with 15 inputs was trained for distinguising from gamma to hadrons.
>
>
  • A network with 15 inputs was trained for distinguishing from gamma to hadrons.
 
  • Choose the best variables and use them as input (variable importance analysis).
Changed:
<
<
  • Create a module that compute the network's output.
>
>
  • Create a module that computes the network's output.
 

Goals from August, 14th to August 25th (Week 3 & 4)

  1. Create a Foswiki
Changed:
<
<
  1. Made an analysis with the most importance variable (ranking).
  2. Choose the most significance variables and use them as input of the network.
>
>
  1. Made an analysis with the most important variable (ranking).
  2. Choose the most significant variables and use them as inputs to the network.
 
  1. Make a Crab maps with the networks trained
Changed:
<
<
  1. Train a network using real data as Background and MC as signal. Compare the results
>
>
  1. Train a network using real data as Background and MC as Signal. Compare the results
 

Week 3

Week 4

Changed:
<
<
  • A network was trained with all variables of HAWC data stream in order to obtain the raking of these variables (MC were used).
  • The best variables were chose and fed as input of the networks.
>
>
  • A network was trained with all variables of HAWC data stream in order to obtain the ranking of these variables (MC were used).
  • The best variables were chosen and fed as inputs to the networks.
 
  • Create a Crab Map with the new networks trained.
Changed:
<
<
  • Obtain the raiking (variable importnace) using real data.
>
>
  • Obtain the ranking (variable importance) using real data.
 

Goal from August 28th to September 1st

Changed:
<
<
  1. Work on new version of disMax.
>
>
  1. Work on a new version of disMax.
 
  1. Read information about Boosted Decision Tree (BDT).
  2. Train a BDT with 15 features.
Changed:
<
<
  1. Check why don't have a good results when real data are used.
>
>
  1. Check why don't have good results when real data are used.

Week 5

  • I've read the Chapter 11 "Decision Tree" in order to understand how to work it.
  • A BDT was trained with Real data as BKG and MC as Signal. But It doesn't answer that I've expected.
  • A BDT was trained with only MC as BKG and Signal. It has a good performance in the bins 4,5,6.
  • Presentation of the MSU meeting (Tuesday, September 5th,2017): The first result using BDT.

September

Goal from September 5th to 8th

  • Make the plot of Energy Vs Q factor of BDT and NN.
  • Repeat the analysis of variable importance but now with the BDT.
  • Train other BDT.

Week 1

 

META FILEATTACHMENT attachment="BringingToTheTable.pdf" attr="" comment="What things I did about gamma/hadron separation using neural network" date="1503696663" name="BringingToTheTable.pdf" path="BringingToTheTable.pdf" size="1126732" user="TomasCap" version="1"
META FILEATTACHMENT attachment="20170818_ChooseFeatures.pdf" attr="" comment="" date="1503698764" name="20170818_ChooseFeatures.pdf" path="20170818_ChooseFeatures.pdf" size="2632027" user="TomasCap" version="1"
Added:
>
>
META FILEATTACHMENT attachment="20170905BDT.pdf" attr="" comment="The first results using Boost Decision Tree" date="1504626635" name="20170905BDT.pdf" path="20170905BDT.pdf" size="3159920" user="TomasCap" version="1"
Revision 2
28 Aug 2017 - Main.TomasCap
Line: 1 to 1
 
META TOPICPARENT name="TomasCap"
-- TomasCap - 25 Aug 2017 This is my logbook
Line: 9 to 9
 

Week 2

Changed:
<
<
  • A network with 15 inputs was trained for separate from gamma to hadrons.
>
>
  • A network with 15 inputs was trained for distinguising from gamma to hadrons.
 
  • Choose the best variables and use them as input (variable importance analysis).
  • Create a module that compute the network's output.

Goals from August, 14th to August 25th (Week 3 & 4)

Line: 28 to 28
 
  • Create a Crab Map with the new networks trained.
  • Obtain the raiking (variable importnace) using real data.

Goal from August 28th to September 1st

Changed:
<
<
  1. Work on new variable.
  2. Read information about Boosted Decision Tree (BDT) and Random Forest (RF).
  3. Train a BDT and RF with 15 features.
>
>
  1. Work on new version of disMax.
  2. Read information about Boosted Decision Tree (BDT).
  3. Train a BDT with 15 features.
  4. Check why don't have a good results when real data are used.
 

META FILEATTACHMENT attachment="BringingToTheTable.pdf" attr="" comment="What things I did about gamma/hadron separation using neural network" date="1503696663" name="BringingToTheTable.pdf" path="BringingToTheTable.pdf" size="1126732" user="TomasCap" version="1"
META FILEATTACHMENT attachment="20170818_ChooseFeatures.pdf" attr="" comment="" date="1503698764" name="20170818_ChooseFeatures.pdf" path="20170818_ChooseFeatures.pdf" size="2632027" user="TomasCap" version="1"
 
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback