Data Simplification: Hitting the Complexity Barrier

Conclusions have no value until they are independently validated. Anyone who attempts to stay current in the sciences soon learns that much of the published literature is irreproducible (8); and that almost anything published today might be retracted tomorrow. This appalling truth applies to some of the most respected and trusted laboratories in the world (9), (10), (11), (12), (13), (14), (15), (16). Those of us who have been involved in assessing the rate of progress in disease research are painfully aware of the numerous reports indicating a general slowdown in medical progress (17), (18), (19), (20), (21), (22), (23), (24). For the optimists, it is tempting to assume that the problems that we may be experiencing today are par for the course, and temporary. It is the nature of science to stall for a while and lurch forwards in sudden fits. Errors and retractions will always be with us so long as humans are involved in the scientific process. For the pessimists, such as myself, there seems to be something going on that is really new and different; a game changer. This game changer is the "complexity barrier", a term credited to Boris Beizer, who used it to describe the impossibility of managing increasingly complex software products (25). The complexity barrier, known also as the complexity ceiling, reflects the intricacies of Big Science and applies to most of the data analysis efforts undertaken these days (26), (27). Some of the mistakes that lead to erroneou...
Source: Specified Life - Category: Information Technology Tags: complexity computer science data analysis data repurposing data simplification data wrangling information science simplifying data taming data Source Type: blogs