You are here: Foswiki>AtlasSingleTop Web>AnalysisVersion1405006TMVA (revision 6)EditAttach

Analysis Comparisons for Code Version 14.05.006

Run Over D3PDs using the MSU analysis code

The DPDPs are stored on the CERN cluster in the directory /rooms/living/ntuples/single_top/TopPhysDPDMaker/14.05.006/ The following samples were included in the analysis:

Signal: 5500, 5502 (note that there is no 5501 sample for v14.05.006)

Background: 5200, 5204, 6280, 6281, 6282, 6283, 7680, 7681, 7682, 7683, 7684, 7685, 7690, 7691, 7692, 7693, 7694, 7695, 7700, 7701, 7702, 7703, 7704, 7705

Unlike v13.0.30, all input files ran without crashing and it was not necessary to comment out any of them in the list files.

The output files from the MSU analysis code are stored in the CERN cluster in the directory /home/root_files/single_top/TopPhysDPDMaker/14.05.006/EarlyData/ The samples in the directory unweighted were processed without an event weight. The samples in the directory weighted were weighted so that the number of events corresponds to a luminosity of 100 pb-1. The weighted samples were used for this analysis.

Merge Signal and Background Files

This comparison was done for the electrons chain. Note that "Electron" is replaced by "Muon" in the file names for the muon chain.

The merging was done using the !MergeTrees.C routine located in the macros/TreeManipulation directory in the MSU analysis package and the merged files are in the directory /home/root_files/single_top/TopPhysDPDMaker/14.05.006/EarlyData/merged/.

The 2 signal files were combined into a single signal file called Topology.!SingleTop.1405006.FDR2.Electron.Signal.root. The 24 background files were combined into a signal background file called Topology.!SingleTop.1405006.FDR2.Electron.Background.root. The merged files were then divided into training, validation, and yield samples, which was achieved by setting the Split flag to 1 in the config file. The events were split according to their order in the merged file. For example, the first event was categorized as training, the second as validation, the third as yield, the fourth as training, and so on. Functionality to randomly assign variables to the different categories will be added in the future.

Events with negative weights were excluded from the Training sample to retain compatibility with SPR, which cannot handle negative weights. Validation samples both with and without events with negative weights were made. The files without events having negative weights are signified by the label NoNeg.

Put Signal and Background Trees into One File

In order to be compatible with the TMVA input requirements, the signal tree from the signal file and the background tree from the background file were placed in one file. This was accomplished by the CombineSigBkg.py routine located in the macros/TreeManipulation directory. This routine is run, for example, as
python CombineSigBkg.py \
   -S /home/root_files/single_top/TopPhysDPDMaker/14.05.006/EarlyData/merged/Topology.SingleTop.1405006.FDR2.Electron.Signal.root \
   -B /home/root_files/single_top/TopPhysDPDMaker/14.05.006/EarlyData/merged/Topology.SingleTop.1405006.FDR2.Electron.Background.root \
   -t TopTree \
   -o /home/root_files/single_top/TopPhysDPDMaker/14.05.006/EarlyData/CombineSigBkg/Topology.SingleTop.1405006.FDR2.Electron.root \

Variables Used

For comparison purposes, only the following variables were considered for training: HT, Jet1Pt =, =DeltaRJet1Jet2, WTransverseMass.

Ranking the Classifiers

The effectiveness of a classifier is determined by both its Signal Efficiency and Background Rejection. From a plot of Background Rejection vs. Signal Efficiency, there are two methods by which to rank the classifiers. The first is by the length of a straight line connecting the upper-right point in the plot to the upper-right axis intersection of (1,1) and the second is by the area under the classifier curve. A shorter line or larger area infers a better classifier.

To produce the file used for classifier ranking, TMVAnalysis.py was run over the merged root files. Note that these are the merged files containing all events, and not those split into training, validation, and yield samples.

In TMVAnalysis.py, half the events were assigned to training and the other half to validation by the line
factory.PrepareTrainingAndTestTree( mycutSig, mycutBkg, "NSigTrain=0:NBkgTrain=0::NSigTest=0:NBkgTest=0:SplitMode=Alternate:NormMode=NumEvents:!V" )

The kNN classifier was not used because it generated the following fatal error message
 <FATAL> KNN            : kNN result list is empty or has wrong size

The following command was used to execute TMVAnalysis.py
 
python TMVAnalysis.py \
   -i /home/root_files/single_top/TopPhysDPDMaker/14.05.006/EarlyData/CombineSigBkg/Topology.SingleTop.1405006.FDR2.Electron.root \
   -t "TopTreeSig TopTreeBkg" \
   -o "TMVAout.ClassifierRanking.1405006.root" \

The program MaxEfficiency.py produces a plot showing Background Rejection vs. Signal Efficiency and ranks the classifiers. The plot and the rankings are shown below. Both rankings have the methods PDERS, CutsGA, and Fisher as the top 3.

MaxEfficiency.png

Area Under the Curve
Rank Method Area
1 PDERS 0.668661
2 CutsGA 0.659969
3 Fisher 0.643790
4 LikelihoodPCA 0.535585
5 SVM_Gauss 0.528220
6 BDT 0.493489
7 Likelihood 0.491559
8 BDTD 0.490003
9 MLP 0.434129
10 HMatrix 0.413259
11 RuleFit 0.393389
12 FDA_MT 0.355496

Length of Line to Upper Right Corner
Rank Method Length
1 PDERS 0.475490
2 CutsGA 0.538328
3 Fisher 0.538659
4 BDT 0.609361
5 LikelihoodPCA 0.633015
6 Likelihood 0.633601
7 SVM_Gauss 0.675473
8 MLP 0.708466
9 BDTD 0.740976
10 HMatrix 0.744161
11 FDA_MT 0.783945
12 RuleFit 0.793519

Run TMVAnalysis over Training files

TMVAnalysis.py was run over the Training sample of the merged root files. The following command was used to execute the python script:
 
python TMVAnalysis.py \
   -S /home/root_files/single_top/TopPhysDPDMaker/14.05.006/EarlyData/merged/Topology.SingleTop.1405006.FDR2.Electron.Signal.Training.NoNeg.root \
   -B /home/root_files/single_top/TopPhysDPDMaker/14.05.006/EarlyData/merged/Topology.SingleTop.1405006.FDR2.Electron.Background.Training.NoNeg.root \
   -t "TopTree TopTree" \
   -o TMVAout.1405006.root \
2 signal events were used for signal validation and 2 background event were used for background validation. This was achieved by the following line in TMVAnalysis.py:
factory.PrepareTrainingAndTestTree( mycutSig, mycutBkg, "NSigTrain=10000000000:NBkgTrain=100000000000::NSigTest=2:NBkgTest=2:SplitMode=Alternate:NormMode=NumEvents:!V" )
Using 0 or 1 validation events for NSigTest and NBkgTest, instead of 2 validation events, caused the program to crash. I'm not sure why this happens.

TMVAnalysis outputs the root file TMVAout.1405006.root and information in the weights directory.

The text output from running the program can be found here: AnalysisTxt1405006

Run TMVApplication over Validation files

TMVApplication.py was run over the Validation samples (with events having negative weights). Signal and background samples were run separately using the following commands.

Signal:
 python TMVApplication.py \
   -i /home/root_files/single_top/TopPhysDPDMaker/14.05.006/EarlyData/merged/Topology.SingleTop.1405006.FDR2.Electron.Signal.Validation.root \
   -o Signal.1405006.root 

Background:
 python TMVApplication.py \
   -i /home/root_files/single_top/TopPhysDPDMaker/14.05.006/EarlyData/merged/Topology.SingleTop.1405006.FDR2.Electron.Background.Validation.root \
   -o Background.1405006.root 

Note that the number of bins in the classifier outputs is 50. This is set by nbin = 50.

Calculate Significance

Significance.py was run in order to calculate the significance. For now, the significance was taken as Signal/sqrt(Background). A more accurate, and complicated, calculation of the significance will be performed in the future. Events were weighted by a factor of 3 to account for the splitting into training, validation, and yield samples.

Significance.py was run as
python Significance.py -S Signal.1405006.root -B Background.1405006.root -w 3 -o SignificanceOutput.1405006.root

The classifier output distributions are shown below for signal and background.
BDT.png

The significance was calculated for different cuts on the classifier output. This was done in two ways, by cutting as you move right across the values on x-axis and by cutting as you move left across the values on the x-axis. Both are shown below.
significanceBDT.Hist.Right.png significanceBDT.Hist.Left.png

Comparisons with Jenny


compare.png

-- PatRyan - 03 Mar 2009
Topic attachments
I Attachment Action SizeSorted ascending Date Who Comment
significanceBDT.Hist.Left.epseps significanceBDT.Hist.Left.eps manage 6.5 K 05 Mar 2009 - 13:59 UnknownUser  
significanceBDT.Hist.Left.pngpng significanceBDT.Hist.Left.png manage 14.1 K 05 Mar 2009 - 14:00 UnknownUser  
BDT.pngpng BDT.png manage 14.8 K 05 Mar 2009 - 13:58 UnknownUser  
significanceBDT.Hist.Right.pngpng significanceBDT.Hist.Right.png manage 15.0 K 05 Mar 2009 - 14:00 UnknownUser  
MaxEfficiency.pngpng MaxEfficiency.png manage 28.6 K 04 Mar 2009 - 16:21 UnknownUser Background Rejection vs. Signal Efficiency for various classifiers
Edit | Attach | Print version | History: r13 | r7 < r6 < r5 < r4 | Backlinks | View wiki text | Edit WikiText | More topic actions...
Topic revision: r6 - 09 Mar 2009, PatRyan
 

This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback