Data Dredging for Dollars, EPA Style

As a person who likes to stay abreast of our ever-expanding government in my areas of specialization (energy and environment), I periodically survey the website of the U.S. Environmental Protection Agency (EPA) to see what they are funding with my taxpayer dollars.

Imagine my surprise when I encountered a novel Request for Proposals at their National Center for Environmental Research seeking to recruit people at non-profit institutions to dredge through EPA’s databases in order to gin up new new things for the agency to worry about and possibly regulate.

Specifically,

The U.S. Environmental Protection Agency (EPA), as part of its Science to Achieve Results (STAR) program, is seeking applications proposing to use existing datasets from health studies to analyze health outcomes for which the link to air pollution is not well established, or to evaluate underlying heterogeneity in health responses among subgroups defined by susceptibility or extent and/or composition of exposure.

And, ever helpful, EPA gives some examples of what such data-dredging exercises might look like:

For example, while air pollution associations with respiratory and cardiovascular disease have been studied most extensively, evidence is beginning to emerge of possible air pollution impacts on additional health conditions including diabetes, neurological disorders, and reproductive and developmental outcomes. Studies also might evaluate factors that confer increased sensitivity to air pollution effects such as compromised health status, genetic variants, social and neighborhood conditions, higher exposure and others. In addition, some research groups have developed innovative methods and models to characterize exposure that might be applied to health effects analyses in other cohorts to understand whether certain sources or atmospheric components contribute to observed geographic heterogeneity in health-exposure associations.

Further, EPA has specific outcomes in mind. This is not random data dredging, which would be bad enough. This program seeks to fund directional data dredging that looks only for relationships suggesting that exposures to various air pollutants causes harm to human health. In EPA’s words:

EPA is interested in research to explain heterogeneity in health responses to air pollutants. Heterogeneity might be explained by: 1) Individual characteristics and other environmental/social conditions that increase the likelihood of an adverse health outcome among a subset of the population. [emphasis mine]

To pay for this innovative regulatory fishing expedition, EPA proposes to give away $1.4 million dollars in portions up to $300,000, for projects that could last up to three years.

Now, there’s nothing wrong with trying to ensure that people’s health is protected from dangerous air pollutants (in fact, I’d argue that it’s a very legitimate function of government), but there is something wrong with organizing taxpayer funded fishing expeditions to probe for new regulatory potential by seeking out obscure relationships in large databases. And those problems are intrinsic to data dredging, an frequently abused form of data mining.

Data dredging, according to Wikipedia, is “the inappropriate (sometimes deliberately so) use of data mining to uncover misleading relationships in data. These relationships may be valid within the test set but have no statistical significance in the wider population.” Wikipedia gives a particularly relevant example: “Suppose that observers note that a particular town appears to be a cancer cluster, but lack a firm hypothesis of why this is so. However, they have access to a large amount of demographic data about the town and surrounding area, containing measurements for the area of hundreds or thousands of different variables, mostly uncorrelated. Even if all these variables are independent of the cancer incidence rate, it is highly likely that at least one variable will be significantly correlated with the cancer rate across the area.”

Or, as the Congressional Research Office explains (in the context of fishing for terrorists in air-travel databases):

Although data mining can help reveal patterns and relationships, it does not tell the user the value or significance of these patterns. These types of determinations must be made by the user. Similarly, the validity of the patterns discovered is dependent on how they compare to “real world” circumstances. For example, to assess the validity of a data mining application designed to identify potential terrorist suspects in a large pool of individuals, the user may test the model using data that includes information about known terrorists. However, while possibly re-affirming a particular profile, it does not necessarily mean that the application will identify a suspect whose behavior significantly deviates from the original model.

Another limitation of data mining is that while it can identify connections between behaviors and/or variables, it does not necessarily identify a causal relationship. For example, an application may identify that a pattern of behavior, such as the propensity to purchase airline tickets just shortly before the flight is scheduled to depart, is related to characteristics such as income, level of education, and Internet use. However, that does not necessarily indicate that the ticket purchasing behavior is caused by one or more of these variables. In fact, the individual’s behavior could be affected by some additional variable(s) such as occupation (the need to make trips on short notice), family status (a sick relative needing care), or a hobby (taking advantage of last minute discounts to visit new destinations).

In other words, with data dredging, it really is a situation of “Seek and ye shall find.”

It is one thing for scientists to identify sick populations, and to investigate what it is that might be making them sick. It is another thing entirely to sift through large data bases in order to come up with correlations that may have no causal relationship, but that might, nonetheless, cause EPA to spend scarce taxpayer money researching the potential linkage, or worse, to endlessly dredge through databases in search of ever lower, ever more obscure health impacts to justify expanded regulation and EPA intrusion into the economy. This is one EPA funding proposal that should be scrapped. If EPA has more money than it knows what to do with, there’s always the crazy idea of giving it back to the taxpayer.