Results

	Unstructured tool	Semistructured tool	Improved tool
Reported Conflicts	19,238 (206%)	14,544 (156%)	9,343 (100%)
Merge Scenarios with Conflicts	2,995 (8.8% / 155%)	2,420 (7.1% / 125%)	1,935 (5.7% / 100%)

Arrows indicate whether the number is underestimated (↑, meaning the numbers should be bigger in practice) or overestimated (↓).
	Unstructured tool	Improved tool
Added False Positives	7,423 ↑	0
Added False Negatives	2,714 ↑	2,489 ↓

	Unstructured tool	Semistructured tool
Added False Positives	7,958 ↑	5,201 ↓
Added False Negatives	2,714 ↑	3,260 ↓

Unstructured Merge Added False Positives by Conflicts
#	Project	Unstructured Merge Conflicts	Ordering Conflicts	(%)
1	Activiti	166	80	48.19
2	AndEngine	39	3	7.69
3	andlytics	82	39	47.56
4	AntennaPod	142	46	32.39
5	antlr4	127	53	41.73
6	atmosphere	118	31	26.27
7	BroadleafCommerce	851	376	44.18
8	Bukkit	12	7	58.33
9	cassandra	6796	2632	38.73
10	cgeo	304	129	42.43
11	clojure	4	-	-
12	closure-compiler	-	-	-
13	cloudify	163	21	12.88
14	commafeed	2	1	50.00
15	commons	1	1	100.00
16	Conversations	21	8	38.10
17	cxf	77	31	40.26
18	deeplearning4j	253	151	59.68
19	dropwizard	19	11	57.89
20	Equivalent-Exchange-3	68	45	66.18
21	Essentials	23	3	13.04
22	gradle	288	150	52.08
23	graylog2-server	145	83	57.24
24	groovy-core	215	126	58.60
25	infinispan	90	23	25.56
26	jedis	293	80	27.30
27	jenkins	429	227	52.91
28	jitsi	29	10	34.48
29	jsoup	7	5	71.43
30	junit	134	72	53.73
31	k-9	266	144	54.14
32	kotlin	171	82	47.95
33	lucene-solr	1677	527	31.43
34	mct	44	7	15.91
35	mockito	57	42	73.68
36	netty	167	101	60.48
37	OG-Platform	3066	1500	48.92
38	okhttp	6	3	50.00
39	OpenRefine	59	-	-
40	OpenTripPlanner	649	270	41.60
41	orientdb	509	172	33.79
42	Osmand	311	120	38.59
43	realm-java	322	119	36.96
44	retrofit	35	21	60.00
45	roboguice	100	42	42.00
46	rstudio	145	71	48.97
47	rundeck	18	9	50.00
48	RxJava	76	42	55.26
49	Spout	224	121	54.02
50	voldemort	438	121	27.63
	MEAN			43.47
	STANDARD DEVIATION			19.01

Semistructured Merge Added False Positives by Conflicts
#	Project	Semistructured Merge Conflicts	Renaming Conflicts	(%)
1	Activiti	123	56	45.53
2	AndEngine	37	4	10.81
3	andlytics	49	16	32.65
4	AntennaPod	152	84	55.26
5	antlr4	115	58	50.43
6	atmosphere	89	10	11.24
7	BroadleafCommerce	516	158	30.62
8	Bukkit	12	8	66.67
9	cassandra	4191	788	18.80
10	cgeo	235	88	37.45
11	clojure	5	1	20.00
12	closure-compiler	1	-	-
13	cloudify	154	31	20.13
14	commafeed	1	-	-
15	commons	-	-	-
16	Conversations	25	1	4.00
17	cxf	45	9	20.00
18	deeplearning4j	100	25	25.00
19	dropwizard	9	3	33.33
20	Equivalent-Exchange-3	78	67	85.90
21	Essentials	20	-	-
22	gradle	159	67	42.14
23	graylog2-server	78	37	47.44
24	groovy-core	193	124	64.25
25	infinispan	71	18	25.35
26	jedis	280	59	21.07
27	jenkins	178	35	19.66
28	jitsi	21	7	33.33
29	jsoup	1	-	-
30	junit	58	14	24.14
31	k-9	188	98	52.13
32	kotlin	112	28	25.00
33	lucene-solr	1171	278	23.74
34	mct	40	2	5.00
35	mockito	6	1	16.67
36	netty	111	72	64.86
37	OG-Platform	3572	2224	62.26
38	okhttp	9	3	33.33
39	OpenRefine	138	-	-
40	OpenTripPlanner	513	226	44.05
41	orientdb	445	134	30.11
42	Osmand	216	30	13.89
43	realm-java	303	138	45.54
44	retrofit	13	5	38.46
45	roboguice	105	56	53.33
46	rstudio	72	9	12.50
47	rundeck	16	10	62.50
48	RxJava	31	10	32.26
49	Spout	139	42	30.22
50	voldemort	348	67	19.25
	MEAN			30.82
	STANDARD DEVIATION			20.64

Numbers by Merge Scenarios

Unstructured Merge Added False Positives by Merge Scenarios
#	Project	Merge Scenarios	Merge Scenarios with Ordering Conflicts	(%)
1	Activiti	786	27	3.44
2	AndEngine	115	5	4.35
3	andlytics	560	11	1.96
4	AntennaPod	519	21	4.05
5	antlr4	656	25	3.81
6	atmosphere	244	9	3.69
7	BroadleafCommerce	898	148	16.48
8	Bukkit	19	4	21.05
9	cassandra	3360	548	16.31
10	cgeo	1890	93	4.92
11	clojure	37	-	-
12	closure-compiler	233	-	-
13	cloudify	213	8	3.76
14	commafeed	241	1	0.41
15	commons	208	1	0.48
16	Conversations	481	9	1.87
17	cxf	71	3	4.23
18	deeplearning4j	731	56	7.66
19	dropwizard	256	6	2.34
20	Equivalent-Exchange-3	387	1	0.26
21	Essentials	572	3	0.52
22	gradle	554	61	11.01
23	graylog2-server	212	25	11.79
24	groovy-core	678	38	5.60
25	infinispan	23	4	17.39
26	jedis	192	28	14.58
27	jenkins	2008	80	3.98
28	jitsi	78	3	3.85
29	jsoup	42	5	11.90
30	junit	350	27	7.71
31	k-9	426	33	7.75
32	kotlin	499	31	6.21
33	lucene-solr	209	58	27.75
34	mct	199	6	3.02
35	mockito	38	4	10.53
36	netty	175	8	4.57
37	OG-Platform	4522	310	6.86
38	okhttp	1038	3	0.29
39	OpenRefine	70	2	2.86
40	OpenTripPlanner	683	99	14.49
41	orientdb	817	53	6.49
42	Osmand	3864	61	1.58
43	realm-java	782	59	7.54
44	retrofit	280	3	1.07
45	roboguice	73	11	15.07
46	rstudio	1463	50	3.42
47	rundeck	549	6	1.09
48	RxJava	418	28	6.70
49	Spout	854	35	4.10
50	voldemort	457	37	8.10
	MEAN			6.58
	STANDARD DEVIATION			6.07

Semistructured Merge Added False Positives by Merge Scenarios
#	Project	Merge Scenarios	Merge Scenarios with Renaming Conflicts	(%)
1	Activiti	786	10	1.27
2	AndEngine	115	4	3.48
3	andlytics	560	5	0.89
4	AntennaPod	519	11	2.12
5	antlr4	656	21	3.20
6	atmosphere	244	6	2.46
7	BroadleafCommerce	898	57	6.35
8	Bukkit	19	2	10.53
9	cassandra	3360	299	8.90
10	cgeo	1890	46	2.43
11	clojure	37	1	2.70
12	closure-compiler	233	-	-
13	cloudify	213	5	2.35
14	commafeed	241	-	-
15	commons	208	-	-
16	Conversations	481	1	0.21
17	cxf	71	2	2.82
18	deeplearning4j	731	13	1.78
19	dropwizard	256	3	1.17
20	Equivalent-Exchange-3	387	1	0.26
21	Essentials	572	-	-
22	gradle	554	33	5.96
23	graylog2-server	212	11	5.19
24	groovy-core	678	22	3.24
25	infinispan	23	1	4.35
26	jedis	192	13	6.77
27	jenkins	2008	19	0.95
28	jitsi	78	2	2.56
29	jsoup	42	-	-
30	junit	350	7	2.00
31	k-9	426	18	4.23
32	kotlin	499	16	3.21
33	lucene-solr	209	40	19.14
34	mct	199	2	1.01
35	mockito	38	1	2.63
36	netty	175	5	2.86
37	OG-Platform	4522	176	3.89
38	okhttp	1038	2	0.19
39	OpenRefine	70	-	-
40	OpenTripPlanner	683	45	6.59
41	orientdb	817	23	2.82
42	Osmand	3864	14	0.36
43	realm-java	782	25	3.20
44	retrofit	280	2	0.71
45	roboguice	73	9	12.33
46	rstudio	1463	6	0.41
47	rundeck	549	2	0.36
48	RxJava	418	6	1.44
49	Spout	854	14	1.64
50	voldemort	457	23	5.03
	MEAN			3.12
	STANDARD DEVIATION			3.59

2. False Negatives

Numbers by Conflicts

Unstructured Merge Added False Negatives by Conflicts
#	Project	Unstructured Merge Conflicts	Duplicated Declaration Errors	(%)
1	Activiti	166	8	4.60
2	AndEngine	39	10	20.41
3	andlytics	82	4	4.65
4	AntennaPod	142	2	1.39
5	antlr4	127	12	8.63
6	atmosphere	118	29	19.73
7	BroadleafCommerce	851	59	6.48
8	Bukkit	12	1	7.69
9	cassandra	6796	1907	21.91
10	cgeo	304	11	3.49
11	clojure	4	-	-
12	closure-compiler	-	4	100.00
13	cloudify	163	54	24.88
14	commafeed	2	-	-
15	commons	1	-	-
16	Conversations	21	-	-
17	cxf	77	12	13.48
18	deeplearning4j	253	6	2.32
19	dropwizard	19	-	-
20	Equivalent-Exchange-3	68	-	-
21	Essentials	23	23	50.00
22	gradle	288	21	6.80
23	graylog2-server	145	11	7.05
24	groovy-core	215	13	5.70
25	infinispan	90	-	-
26	jedis	293	66	18.38
27	jenkins	429	22	4.88
28	jitsi	29	4	12.12
29	jsoup	7	-	-
30	junit	134	5	3.60
31	k-9	266	17	6.01
32	kotlin	171	14	7.57
33	lucene-solr	1677	60	3.45
34	mct	44	-	-
35	mockito	57	-	-
36	netty	167	8	4.57
37	OG-Platform	3066	113	3.55
38	okhttp	6	2	25.00
39	OpenRefine	59	27	31.40
40	OpenTripPlanner	649	46	6.62
41	orientdb	509	71	12.24
42	Osmand	311	6	1.89
43	realm-java	322	39	10.80
44	retrofit	35	2	5.41
45	roboguice	100	-	-
46	rstudio	145	5	3.33
47	rundeck	18	1	5.26
48	RxJava	76	-	-
49	Spout	224	6	2.61
50	voldemort	438	13	2.88
	MEAN			9.62
	STANDARD DEVIATION			16.29

Semistructured Merge Added False Negatives by Conflicts
#	Project	Semistructured Merge Conflicts	Type Ambiguity Errors	Initialization Blocks	Acidental Conflicts	(%)
1	Activiti	123	-	6	23	20.40
2	AndEngine	37	-	-	5	11.90
3	andlytics	49	-	1	10	18.95
4	AntennaPod	152	-	2	189	66.64
5	antlr4	115	-	3	18	16.46
6	atmosphere	89	-	-	15	14.42
7	BroadleafCommerce	516	-	3	142	23.33
8	Bukkit	12	-	-	2	14.29
9	cassandra	4191	14	57	971	21.73
10	cgeo	235	-	10	35	17.04
11	clojure	5	-	-	-	-
12	closure-compiler	1	-	-	-	-
13	cloudify	154	-	-	23	12.99
14	commafeed	1	-	-	-	-
15	commons	-	-	-	-	-
16	Conversations	25	-	2	3	18.12
17	cxf	45	-	-	11	19.64
18	deeplearning4j	100	-	-	38	27.54
19	dropwizard	9	-	-	3	25.00
20	Equivalent-Exchange-3	78	-	-	13	14.29
21	Essentials	20	-	-	1	4.76
22	gradle	159	-	-	46	22.68
23	graylog2-server	78	-	-	25	25.27
24	groovy-core	193	-	2	39	18.00
25	infinispan	71	-	-	14	16.47
26	jedis	280	-	-	26	8.56
27	jenkins	178	-	11	63	32.45
28	jitsi	21	-	-	5	19.23
29	jsoup	1	-	-	1	50.00
30	junit	58	-	-	21	26.58
31	k-9	188	-	13	38	23.28
32	kotlin	112	-	11	22	25.36
33	lucene-solr	1171	-	6	275	19.81
34	mct	40	-	-	7	15.87
35	mockito	6	-	-	10	62.50
36	netty	111	-	1	29	21.92
37	OG-Platform	3572	1	20	483	13.03
38	okhttp	9	-	1	1	20.00
39	OpenRefine	138	-	1	9	6.84
40	OpenTripPlanner	513	-	3	104	17.73
41	orientdb	445	-	-	73	14.52
42	Osmand	216	-	29	35	25.90
43	realm-java	303	-	7	45	15.19
44	retrofit	13	-	-	6	31.58
45	roboguice	105	-	3	14	14.54
46	rstudio	72	-	1	20	23.11
47	rundeck	16	-	-	3	15.79
48	RxJava	31	-	-	14	31.11
49	Spout	139	-	3	36	22.93
50	voldemort	348	-	25	58	21.56
	MEAN					20.60
	STANDARD DEVIATION					21.30

Numbers by Merge Scenarios

Unstructured Merge Added False Negatives by Merge Scenarios
#	Project	Merge Scenarios	Merge Scenarios with Duplicated Declaration Errors	(%)
1	Activiti	786	3	0.38
2	AndEngine	115	3	2.61
3	andlytics	560	2	0.36
4	AntennaPod	519	1	0.19
5	antlr4	656	4	0.61
6	atmosphere	244	4	1.64
7	BroadleafCommerce	898	14	1.56
8	Bukkit	19	1	5.26
9	cassandra	3360	58	1.73
10	cgeo	1890	6	0.32
11	clojure	37	-	-
12	closure-compiler	233	1	0.43
13	cloudify	213	5	2.35
14	commafeed	241	-	-
15	commons	208	-	-
16	Conversations	481	-	-
17	cxf	71	2	2.82
18	deeplearning4j	731	3	0.41
19	dropwizard	256	-	-
20	Equivalent-Exchange-3	387	-	-
21	Essentials	572	3	0.52
22	gradle	554	8	1.44
23	graylog2-server	212	2	0.94
24	groovy-core	678	4	0.59
25	infinispan	23	-	-
26	jedis	192	4	2.08
27	jenkins	2008	16	0.80
28	jitsi	78	3	3.85
29	jsoup	42	-	-
30	junit	350	4	1.14
31	k-9	426	3	0.70
32	kotlin	499	2	0.40
33	lucene-solr	209	3	1.44
34	mct	199	-	-
35	mockito	38	-	-
36	netty	175	1	0.57
37	OG-Platform	4522	31	0.69
38	okhttp	1038	1	0.10
39	OpenRefine	70	1	1.43
40	OpenTripPlanner	683	9	1.32
41	orientdb	817	15	1.84
42	Osmand	3864	4	0.10
43	realm-java	782	14	1.79
44	retrofit	280	1	0.36
45	roboguice	73	-	-
46	rstudio	1463	2	0.14
47	rundeck	549	1	0.18
48	RxJava	418	-	-
49	Spout	854	3	0.35
50	voldemort	457	3	0.66
	MEAN			0.88
	STANDARD DEVIATION			1.08

Semistructured Merge Added False Negatives by Merge Scenarios
#	project	Merge Scenarios	Merge Scenarios with Type Ambiguity Errors	Merge Scenarios with Initialization Blocks	Merge Scenarios with Acidental Conflicts	(%)
1	Activiti	786	-	6	20	3.30
2	AndEngine	115	-	-	3	2.61
3	andlytics	560	-	1	7	1.43
4	AntennaPod	519	-	2	22	4.63
5	antlr4	656	-	2	11	1.97
6	atmosphere	244	-	-	6	2.46
7	BroadleafCommerce	898	-	3	26	3.23
8	Bukkit	19	-	-	1	5.26
9	cassandra	3360	2	23	334	10.68
10	cgeo	1890	-	10	30	2.12
11	clojure	37	-	-	-	-
12	closure-compiler	233	-	-	-	-
13	cloudify	213	-	-	6	2.82
14	commafeed	241	-	-	-	-
15	commons	208	-	-	-	-
16	Conversations	481	-	1	1	0.42
17	cxf	71	-	-	2	2.82
18	deeplearning4j	731	-	-	34	4.65
19	dropwizard	256	-	-	1	0.39
20	Equivalent-Exchange-3	387	-	-	1	0.26
21	Essentials	572	-	-	1	0.17
22	gradle	554	-	-	31	5.60
23	graylog2-server	212	-	-	13	6.13
24	groovy-core	678	-	2	16	2.65
25	infinispan	23	-	-	3	13.04
26	jedis	192	-	-	13	6.77
27	jenkins	2008	-	9	61	3.49
28	jitsi	78	-	-	3	3.85
29	jsoup	42	-	-	1	2.38
30	junit	350	-	-	12	3.43
31	k-9	426	-	3	15	4.22
32	kotlin	499	-	4	11	3.00
33	lucene-solr	209	-	1	53	25.84
34	mct	199	-	-	4	2.01
35	mockito	38	-	-	4	10.53
36	netty	175	-	1	26	15.43
37	OG-Platform	4522	1	13	209	4.93
38	okhttp	1038	-	1	1	0.20
39	OpenRefine	70	-	1	2	4.29
40	OpenTripPlanner	683	-	3	82	12.45
41	orientdb	817	-	-	40	4.89
42	Osmand	3864	-	8	13	0.55
43	realm-java	782	-	5	33	4.86
44	retrofit	280	-	-	2	0.71
45	roboguice	73	-	3	8	15.07
46	rstudio	1463	-	1	36	2.53
47	rundeck	549	-	-	2	0.36
48	RxJava	418	-	-	7	1.67
49	Spout	854	-	2	29	3.63
50	voldemort	457	-	9	24	7.22
	MEAN					4.42
	STANDARD DEVIATION					5.53

Results

Summary

Results for False Positive and False Negative Analysis

1. False Positives

Numbers by Conflicts

Numbers by Merge Scenarios

2. False Negatives

Numbers by Conflicts

Numbers by Merge Scenarios

Plots

Results for Improved Semistructured tool Performance Analysis