Difference: AnalysisVersion1405006ComparisonTMVASPR (1 vs. 4)

Revision 4
07 May 2009 - PatRyan
Line: 1 to 1
 
META TOPICPARENT name="AnalysisVersion1405006"

Comparisons between TMVA and SPR

Changed:
<
<

Compare the Two Methods of Calculating Significance in TMVA

>
>

Two Methods of Calculating Significance in TMVA

 

Using TMVAnalysis and TMVApplication

The first method available in TMVA to calculate significance is to first run TMVAnalysis and the TMVApplication. TMVAnalysis was run as
Line: 96 to 96
  = Signal_MVA_BDT Maximum Significance (Left): 7.41332483292
Changed:
<
<

Compare Significance from TMVA and SPR

>
>

Significance from SPR

 

Run the training
Line: 144 to 144
 

= Signal_Bagger Maximum Significance (From Right): 4.29537010193 = Signal_Bagger Maximum Significance (From Left): 4.26745414734
Deleted:
<
<
 
Added:
>
>

Conclusions

The significance from the three different methods are:

  • TMVAnalysis and TMVApplication: 3.991
  • TMVAnalysis and direct calculation: 5.371
  • SPR and direct calculation: 3.991

TMVAnalysis with TMVApplication and SPR give the same total significance. TMVAnalysis with a direction calculation gives a different significance, which may be due to the fact that a different sample was used for training and validation. However, this significance is not at all close to the other significances, which means that the difference is most likely due to TestAllMethods() doing something different than the validation in TMVApplication and SPR.
  -- PatRyan - 04 May 2009 \ No newline at end of file
Revision 3
07 May 2009 - PatRyan
Line: 1 to 1
 
META TOPICPARENT name="AnalysisVersion1405006"

Comparisons between TMVA and SPR

Line: 46 to 46
 

The output is
Changed:
<
<
Signal Events: 84.6193445325 Background Events: 449.649060309
>
>
== Weighted Signal Events: 84.6193445325 Weighted Background Events: 449.649060309 == Total Significance: 3.99055050669
 
Changed:
<
<
= BDT Maximum Significance (Right): 5.54816436768 = BDT Maximum Significance (Left): 3.99055051804
>
>
= BDT Maximum Significance (From Right): 5.54816436768 = BDT Maximum Significance (From Left): 3.99055051804
 

TMVAnalysis and then calculate significance directly

Line: 87 to 88
 

The output included
Changed:
<
<
Signal Events: 347.999909878 Background Events: 4197.4060111
>
>

== Weighted Signal Events: 347.999909878 Weighted Background Events: 4197.4060111 == Total Significance: 5.37141418101
 

= Signal_MVA_BDT Maximum Significance (Right): 5.37141418457 = Signal_MVA_BDT Maximum Significance (Left): 7.41332483292
Line: 132 to 135
 

Output included
Added:
>
>
= Bagger
  = Method: Bagger Signal Events: 119 Background Events: 1268
Deleted:
<
<
Info in <TCanvas::Print>: eps file eps/Overlay/SPR_Signal_Bagger.eps has been created Info in <TCanvas::Print>: eps file eps/Overlay/Norm.SPR_Signal_Bagger.eps has been created Info in <TCanvas::Print>: eps file eps/SigVsBkg/SigVsBkg_SPR_Signal_Bagger.eps has been created Signal Events (weighted): 84.6193494797 Background Events (weighted): 449.649265766
 
Changed:
<
<
= Signal_Bagger Maximum Significance (Right): 4.29537010193 = Signal_Bagger Maximum Significance (Left): 4.26745414734
>
>
== Weighted Signal Events: 84.6193494797 Weighted Background Events: 449.649265766 == Total Significance: 3.99054982829

= Signal_Bagger Maximum Significance (From Right): 4.29537010193 = Signal_Bagger Maximum Significance (From Left): 4.26745414734
 

-- PatRyan - 04 May 2009
Revision 2
06 May 2009 - PatRyan
Line: 1 to 1
 
META TOPICPARENT name="AnalysisVersion1405006"

Comparisons between TMVA and SPR

Line: 80 to 80
 
Factory : -----------------------------------------------------------------------------
Changed:
<
<
Run SignificanceAlt.py
>
>
Run SignificanceAlt.py
 
 python SignificanceAlt.py -m BDT -i TMVAout.CompareTMVA.Method2.1405006.root -o SignificanceOutput.Compare.Method2.1405006.root -p "TMVA" -w2
Line: 95 to 95
 

Compare Significance from TMVA and SPR

Added:
>
>
Run the training
/usr/local/bin/bin/SprBaggerDecisionTreeApp  \
  -n 1100 -l 6 -s 33 -g 1 -y '0:1' -d 5\
  -f baggerCompare.spr -o baggerOutputTraining.Compare.root \
  data/baggerTraining.pat \

The output includes the following
Total number of points read: 1238
Training data filtered by class.
Points in class 0(1):   1123
Points in class 1(1):   115
 
Added:
>
>
Validation was run by
/usr/local/bin/bin/SprOutputWriterApp baggerCompare.spr -y '0:1'  \
  data/baggerValidation.pat baggerOutputValidation.Compare.root

Output included
Total number of points read: 1387
Training data filtered by class.
Points in class 0(1):   1268
Points in class 1(1):   119

Significance was calculated by
python SignificanceAlt.py -m BDT -i /work/jever/pryan/SPR-3.3.1/baggerOutputValidation.Compare.root -o SignificanceOutput.Compare.SPR.1405006.root -p "SPR" -w3

Output included
=== Method:  Bagger
    Signal Events:  119     Background Events:  1268
Info in <TCanvas::Print>: eps file eps/Overlay/SPR_Signal_Bagger.eps has been created
Info in <TCanvas::Print>: eps file eps/Overlay/Norm.SPR_Signal_Bagger.eps has been created
Info in <TCanvas::Print>: eps file eps/SigVsBkg/SigVsBkg_SPR_Signal_Bagger.eps has been created
Signal Events (weighted):  84.6193494797 Background Events (weighted):  449.649265766

===  Signal_Bagger Maximum Significance (Right):  4.29537010193
===  Signal_Bagger Maximum Significance (Left):   4.26745414734
 

-- PatRyan - 04 May 2009 \ No newline at end of file
 
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback