Results

Summary

Unstructured toolSemistructured toolImproved tool
Reported Conflicts19,238 (206%)14,544 (156%)9,343 (100%)
Merge Scenarios with Conflicts2,995 (8.8% / 155%)2,420 (7.1% / 125%)1,935 (5.7% / 100%)
Arrows indicate whether the number is underestimated (↑, meaning the numbers should be bigger in practice) or overestimated (↓).
Unstructured toolImproved tool
Added False Positives7,423 ↑0
Added False Negatives2,714 ↑2,489 ↓
Unstructured toolSemistructured tool
Added False Positives7,958 ↑5,201 ↓
Added False Negatives2,714 ↑3,260 ↓

Results for False Positive and False Negative Analysis

1. False Positives

Numbers by Conflicts

Unstructured Merge Added False Positives by Conflicts
#ProjectUnstructured Merge ConflictsOrdering Conflicts(%)
1Activiti1668048.19
2AndEngine3937.69
3andlytics823947.56
4AntennaPod1424632.39
5antlr41275341.73
6atmosphere1183126.27
7BroadleafCommerce85137644.18
8Bukkit12758.33
9cassandra6796263238.73
10cgeo30412942.43
11clojure4--
12closure-compiler---
13cloudify1632112.88
14commafeed2150.00
15commons11100.00
16Conversations21838.10
17cxf773140.26
18deeplearning4j25315159.68
19dropwizard191157.89
20Equivalent-Exchange-3684566.18
21Essentials23313.04
22gradle28815052.08
23graylog2-server1458357.24
24groovy-core21512658.60
25infinispan902325.56
26jedis2938027.30
27jenkins42922752.91
28jitsi291034.48
29jsoup7571.43
30junit1347253.73
31k-926614454.14
32kotlin1718247.95
33lucene-solr167752731.43
34mct44715.91
35mockito574273.68
36netty16710160.48
37OG-Platform3066150048.92
38okhttp6350.00
39OpenRefine59--
40OpenTripPlanner64927041.60
41orientdb50917233.79
42Osmand31112038.59
43realm-java32211936.96
44retrofit352160.00
45roboguice1004242.00
46rstudio1457148.97
47rundeck18950.00
48RxJava764255.26
49Spout22412154.02
50voldemort43812127.63
MEAN  43.47
STANDARD DEVIATION  19.01
Semistructured Merge Added False Positives by Conflicts
#ProjectSemistructured Merge ConflictsRenaming Conflicts(%)
1Activiti1235645.53
2AndEngine37410.81
3andlytics491632.65
4AntennaPod1528455.26
5antlr41155850.43
6atmosphere891011.24
7BroadleafCommerce51615830.62
8Bukkit12866.67
9cassandra419178818.80
10cgeo2358837.45
11clojure5120.00
12closure-compiler1--
13cloudify1543120.13
14commafeed1--
15commons---
16Conversations2514.00
17cxf45920.00
18deeplearning4j1002525.00
19dropwizard9333.33
20Equivalent-Exchange-3786785.90
21Essentials20--
22gradle1596742.14
23graylog2-server783747.44
24groovy-core19312464.25
25infinispan711825.35
26jedis2805921.07
27jenkins1783519.66
28jitsi21733.33
29jsoup1--
30junit581424.14
31k-91889852.13
32kotlin1122825.00
33lucene-solr117127823.74
34mct4025.00
35mockito6116.67
36netty1117264.86
37OG-Platform3572222462.26
38okhttp9333.33
39OpenRefine138--
40OpenTripPlanner51322644.05
41orientdb44513430.11
42Osmand2163013.89
43realm-java30313845.54
44retrofit13538.46
45roboguice1055653.33
46rstudio72912.50
47rundeck161062.50
48RxJava311032.26
49Spout1394230.22
50voldemort3486719.25
 MEAN  30.82
 STANDARD DEVIATION  20.64

Numbers by Merge Scenarios

Unstructured Merge Added False Positives by Merge Scenarios
#ProjectMerge ScenariosMerge Scenarios with
Ordering Conflicts
(%)
1Activiti786273.44
2AndEngine11554.35
3andlytics560111.96
4AntennaPod519214.05
5antlr4656253.81
6atmosphere24493.69
7BroadleafCommerce89814816.48
8Bukkit19421.05
9cassandra336054816.31
10cgeo1890934.92
11clojure37--
12closure-compiler233--
13cloudify21383.76
14commafeed24110.41
15commons20810.48
16Conversations48191.87
17cxf7134.23
18deeplearning4j731567.66
19dropwizard25662.34
20Equivalent-Exchange-338710.26
21Essentials57230.52
22gradle5546111.01
23graylog2-server2122511.79
24groovy-core678385.60
25infinispan23417.39
26jedis1922814.58
27jenkins2008803.98
28jitsi7833.85
29jsoup42511.90
30junit350277.71
31k-9426337.75
32kotlin499316.21
33lucene-solr2095827.75
34mct19963.02
35mockito38410.53
36netty17584.57
37OG-Platform45223106.86
38okhttp103830.29
39OpenRefine7022.86
40OpenTripPlanner6839914.49
41orientdb817536.49
42Osmand3864611.58
43realm-java782597.54
44retrofit28031.07
45roboguice731115.07
46rstudio1463503.42
47rundeck54961.09
48RxJava418286.70
49Spout854354.10
50voldemort457378.10
MEAN  6.58
STANDARD DEVIATION  6.07
Semistructured Merge Added False Positives by Merge Scenarios
#ProjectMerge ScenariosMerge Scenarios with
Renaming Conflicts
(%)
1Activiti786101.27
2AndEngine11543.48
3andlytics56050.89
4AntennaPod519112.12
5antlr4656213.20
6atmosphere24462.46
7BroadleafCommerce898576.35
8Bukkit19210.53
9cassandra33602998.90
10cgeo1890462.43
11clojure3712.70
12closure-compiler233--
13cloudify21352.35
14commafeed241--
15commons208--
16Conversations48110.21
17cxf7122.82
18deeplearning4j731131.78
19dropwizard25631.17
20Equivalent-Exchange-338710.26
21Essentials572--
22gradle554335.96
23graylog2-server212115.19
24groovy-core678223.24
25infinispan2314.35
26jedis192136.77
27jenkins2008190.95
28jitsi7822.56
29jsoup42--
30junit35072.00
31k-9426184.23
32kotlin499163.21
33lucene-solr2094019.14
34mct19921.01
35mockito3812.63
36netty17552.86
37OG-Platform45221763.89
38okhttp103820.19
39OpenRefine70--
40OpenTripPlanner683456.59
41orientdb817232.82
42Osmand3864140.36
43realm-java782253.20
44retrofit28020.71
45roboguice73912.33
46rstudio146360.41
47rundeck54920.36
48RxJava41861.44
49Spout854141.64
50voldemort457235.03
MEAN  3.12
STANDARD DEVIATION  3.59

2. False Negatives

Numbers by Conflicts

Unstructured Merge Added False Negatives by Conflicts
#ProjectUnstructured Merge ConflictsDuplicated Declaration Errors(%)
1Activiti16684.60
2AndEngine391020.41
3andlytics8244.65
4AntennaPod14221.39
5antlr4127128.63
6atmosphere1182919.73
7BroadleafCommerce851596.48
8Bukkit1217.69
9cassandra6796190721.91
10cgeo304113.49
11clojure4--
12closure-compiler-4100.00
13cloudify1635424.88
14commafeed2--
15commons1--
16Conversations21--
17cxf771213.48
18deeplearning4j25362.32
19dropwizard19--
20Equivalent-Exchange-368--
21Essentials232350.00
22gradle288216.80
23graylog2-server145117.05
24groovy-core215135.70
25infinispan90--
26jedis2936618.38
27jenkins429224.88
28jitsi29412.12
29jsoup7--
30junit13453.60
31k-9266176.01
32kotlin171147.57
33lucene-solr1677603.45
34mct44--
35mockito57--
36netty16784.57
37OG-Platform30661133.55
38okhttp6225.00
39OpenRefine592731.40
40OpenTripPlanner649466.62
41orientdb5097112.24
42Osmand31161.89
43realm-java3223910.80
44retrofit3525.41
45roboguice100--
46rstudio14553.33
47rundeck1815.26
48RxJava76--
49Spout22462.61
50voldemort438132.88
MEAN  9.62
STANDARD DEVIATION  16.29
Semistructured Merge Added False Negatives by Conflicts
#ProjectSemistructured Merge ConflictsType Ambiguity ErrorsInitialization BlocksAcidental Conflicts(%)
1Activiti123-62320.40
2AndEngine37--511.90
3andlytics49-11018.95
4AntennaPod152-218966.64
5antlr4115-31816.46
6atmosphere89--1514.42
7BroadleafCommerce516-314223.33
8Bukkit12--214.29
9cassandra4191145797121.73
10cgeo235-103517.04
11clojure5----
12closure-compiler1----
13cloudify154--2312.99
14commafeed1----
15commons-----
16Conversations25-2318.12
17cxf45--1119.64
18deeplearning4j100--3827.54
19dropwizard9--325.00
20Equivalent-Exchange-378--1314.29
21Essentials20--14.76
22gradle159--4622.68
23graylog2-server78--2525.27
24groovy-core193-23918.00
25infinispan71--1416.47
26jedis280--268.56
27jenkins178-116332.45
28jitsi21--519.23
29jsoup1--150.00
30junit58--2126.58
31k-9188-133823.28
32kotlin112-112225.36
33lucene-solr1171-627519.81
34mct40--715.87
35mockito6--1062.50
36netty111-12921.92
37OG-Platform357212048313.03
38okhttp9-1120.00
39OpenRefine138-196.84
40OpenTripPlanner513-310417.73
41orientdb445--7314.52
42Osmand216-293525.90
43realm-java303-74515.19
44retrofit13--631.58
45roboguice105-31414.54
46rstudio72-12023.11
47rundeck16--315.79
48RxJava31--1431.11
49Spout139-33622.93
50voldemort348-255821.56
 MEAN    20.60
 STANDARD DEVIATION    21.30

Numbers by Merge Scenarios

Unstructured Merge Added False Negatives by Merge Scenarios
#ProjectMerge ScenariosMerge Scenarios with
Duplicated Declaration Errors
(%)
1Activiti78630.38
2AndEngine11532.61
3andlytics56020.36
4AntennaPod51910.19
5antlr465640.61
6atmosphere24441.64
7BroadleafCommerce898141.56
8Bukkit1915.26
9cassandra3360581.73
10cgeo189060.32
11clojure37--
12closure-compiler23310.43
13cloudify21352.35
14commafeed241--
15commons208--
16Conversations481--
17cxf7122.82
18deeplearning4j73130.41
19dropwizard256--
20Equivalent-Exchange-3387--
21Essentials57230.52
22gradle55481.44
23graylog2-server21220.94
24groovy-core67840.59
25infinispan23--
26jedis19242.08
27jenkins2008160.80
28jitsi7833.85
29jsoup42--
30junit35041.14
31k-942630.70
32kotlin49920.40
33lucene-solr20931.44
34mct199--
35mockito38--
36netty17510.57
37OG-Platform4522310.69
38okhttp103810.10
39OpenRefine7011.43
40OpenTripPlanner68391.32
41orientdb817151.84
42Osmand386440.10
43realm-java782141.79
44retrofit28010.36
45roboguice73--
46rstudio146320.14
47rundeck54910.18
48RxJava418--
49Spout85430.35
50voldemort45730.66
MEAN  0.88
STANDARD DEVIATION  1.08
Semistructured Merge Added False Negatives by Merge Scenarios
#projectMerge ScenariosMerge Scenarios with
Type Ambiguity Errors
Merge Scenarios with
Initialization Blocks
Merge Scenarios with
Acidental Conflicts
(%)
1Activiti786-6203.30
2AndEngine115--32.61
3andlytics560-171.43
4AntennaPod519-2224.63
5antlr4656-2111.97
6atmosphere244--62.46
7BroadleafCommerce898-3263.23
8Bukkit19--15.26
9cassandra336022333410.68
10cgeo1890-10302.12
11clojure37----
12closure-compiler233----
13cloudify213--62.82
14commafeed241----
15commons208----
16Conversations481-110.42
17cxf71--22.82
18deeplearning4j731--344.65
19dropwizard256--10.39
20Equivalent-Exchange-3387--10.26
21Essentials572--10.17
22gradle554--315.60
23graylog2-server212--136.13
24groovy-core678-2162.65
25infinispan23--313.04
26jedis192--136.77
27jenkins2008-9613.49
28jitsi78--33.85
29jsoup42--12.38
30junit350--123.43
31k-9426-3154.22
32kotlin499-4113.00
33lucene-solr209-15325.84
34mct199--42.01
35mockito38--410.53
36netty175-12615.43
37OG-Platform45221132094.93
38okhttp1038-110.20
39OpenRefine70-124.29
40OpenTripPlanner683-38212.45
41orientdb817--404.89
42Osmand3864-8130.55
43realm-java782-5334.86
44retrofit280--20.71
45roboguice73-3815.07
46rstudio1463-1362.53
47rundeck549--20.36
48RxJava418--71.67
49Spout854-2293.63
50voldemort457-9247.22
 MEAN    4.42
 STANDARD DEVIATION    5.53

Plots

Results for Improved Semistructured tool Performance Analysis