Experimental Evalulation in Computer Science: A Quantitative Study

Get BibTex-formatted data

Author

W.F. Tichy,P. Lukowicz,L. Prechelt,E.A. Heinz

Entry type

techreport

Abstract

A survey of 400 recent research articles suggests that computer scientists publish relatively few papers with experimentally validated results. The survey includes complete volumes of several referred computer science journals, a conference, and 50 titles drawn at random from all articles published by the ACM in 1993. The journals, Optical Engineering and Neural Computation were used for comparison. Of the papers in the random samplethat would require experimental validation, 40 have none at all. In journals related to software engineering, this fraction is over 50. In comparison, the fraction of papers lacking qunatitative evaluation in OE and NC is only 15 an 12 respectively. Conversely, the fraction of papers that devote 1/5 or more of their space to experimental validation is almost 70 for OE and NC, while it is a mere 30 for the CS random sample and 20 for software engineering. The low ratio of validated results appears to be a serious weakness in the area of computer science research. This weakness should be rectified for the long-term health of the field.

Date

1995 – January

Address

University of Karlsruhe, Germany

Institution

University of Karlsruhe

Journal

Journal of Systems and Software

Key alpha

Tichy

Publication Date

0000-00-00

Location

A hard-copy of this is in the Papers Cabinet

BibTex-formatted data

To refer to this entry, you may select and copy the text below and paste it into your BibTex document. Note that the text may not contain all macros that BibTex supports.

@Techreport{ Tichy,
	title = "Experimental Evalulation in Computer Science: A Quantitative Study",
	author = "W.F. Tichy,P. Lukowicz,L. Prechelt,E.A. Heinz",
	year = "1995",
	month = "January",
	address = "University of Karlsruhe, Germany",
	institution = "University of Karlsruhe",
	journal = "Journal of Systems and Software",
	abstract = "A survey of 400 recent research articles suggests that computer scientists
publish relatively few papers with experimentally validated results.
The survey includes complete volumes of several referred computer science
journals, a conference, and 50 titles drawn at random from all articles
published by the ACM in 1993. The journals, Optical Engineering and Neural
Computation were used for comparison.
Of the papers in the random samplethat would require experimental validation,
40 have none at all. In journals related to software engineering, this
fraction is over 50. In comparison, the fraction of papers lacking
qunatitative evaluation in OE and NC is only 15 an 12 respectively.
Conversely, the fraction of papers that devote 1/5 or more of their space to
experimental validation is almost 70 for OE and NC, while it is a mere 30
for the CS random sample and 20 for software engineering.
The low ratio of validated results appears to be a serious weakness in the
area of computer science research. This weakness should be rectified for
the long-term health of the field.",
}