Comparisons between TMVA and SPR

Compare the Two Methods of Calculating Significance in TMVA

Using TMVAnalysis and TMVApplication

The first method available in TMVA to calculate significance is to first run TMVAnalysis and the TMVApplication. TMVAnalysis was run as
python TMVAnalysis.py \
   -i /home/root_files/single_top/TopPhysDPDMaker/14.05.006/EarlyData/CombineSigBkg/Topology.SingleTop.1405006.FDR2.Electron.Training.NoNeg.root \
   -t "TopTreeSig TopTreeBkg" -m BDT\
   -o "TMVAout.CompareTMVA.1405006.root" \

The following line was used in TMVAnalysis so that all the events except for 2 were used for training
factory.PrepareTrainingAndTestTree( mycutSig, mycutBkg, "NSigTrain=113:NBkgTrain=1121::NSigTest=2:NBkgTest=2:SplitMode=Alternate:NormMode=NumEvents:!V" )

The output of TMVAnalysis includes the following
--- DataSet        : - Training signal entries     : 113
--- DataSet        : - Training background entries : 1121
--- DataSet        : - Testing  signal entries     : 2
--- DataSet        : - Testing  background entries : 2

TMVApplication was run as
python TMVApplication.py \
   -i /home/root_files/single_top/TopPhysDPDMaker/14.05.006/EarlyData/merged/Topology.SingleTop.1405006.FDR2.Electron.Signal.Validation.root -m BDT\
   -o Signal.Compare.1405006.root 

python TMVApplication.py \
   -i /home/root_files/single_top/TopPhysDPDMaker/14.05.006/EarlyData/merged/Topology.SingleTop.1405006.FDR2.Electron.Background.Validation.root -m BDT \
   -o Background.Compare.1405006.root 

Significance was calculated as
python Significance.py -S Signal.Compare.1405006.root -B Background.Compare.1405006.root -w 3 -o SignificanceOutput.Compare.1405006.root

The output is
Signal Events:  84.6193445325 Background Events:  449.649060309

===  BDT Maximum Significance (Right):  5.54816436768
===  BDT Maximum Significance (Left):   3.99055051804

TMVAnalysis and then calculate significance directly

python TMVAnalysis.py \
   -i /home/root_files/single_top/TopPhysDPDMaker/14.05.006/EarlyData/CombineSigBkg/Topology.SingleTop.1405006.FDR2.Electron.NoNeg.root \
   -t "TopTreeSig TopTreeBkg" -m BDT\
   -o "TMVAout.CompareTMVA.Method2.1405006.root" \

The following line was used in TMVAnalysis so that half of the events were used for training and half for testing
factory.PrepareTrainingAndTestTree( mycutSig, mycutBkg, "NSigTrain=0:NBkgTrain=0::NSigTest=0:NBkgTest=0:SplitMode=Alternate:NormMode=NumEvents:!V" )

The output included
--- DataSet        : - Training signal entries     : 174
--- DataSet        : - Training background entries : 1677
--- DataSet        : - Testing  signal entries     : 174
--- DataSet        : - Testing  background entries : 1677

--- Factory        : -----------------------------------------------------------------------------
--- Factory        : MVA              Signal efficiency at bkg eff. (error):  |  Sepa-    Signifi-
--- Factory        : Methods:         @B=0.01    @B=0.10    @B=0.30    Area   |  ration:  cance:  
--- Factory        : -----------------------------------------------------------------------------
--- Factory        : BDT            : 0.045(15)  0.047(16)  0.052(16)  0.268  |  0.511    0.276
--- Factory        : -----------------------------------------------------------------------------

Run SignificanceAlt.py
 python SignificanceAlt.py -m BDT -i TMVAout.CompareTMVA.Method2.1405006.root -o SignificanceOutput.Compare.Method2.1405006.root -p "TMVA" -w2

The output included
Signal Events:  347.999909878 Background Events:  4197.4060111

===  Signal_MVA_BDT Maximum Significance (Right):  5.37141418457
===  Signal_MVA_BDT Maximum Significance (Left):   7.41332483292

Compare Significance from TMVA and SPR

-- PatRyan - 04 May 2009
Edit | Attach | Print version | History: r4 < r3 < r2 < r1 | Backlinks | View wiki text | Edit WikiText | More topic actions...
Topic revision: r1 - 05 May 2009, PatRyan
 

This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback