Twenty tips for interpreting scientific claims. The authors suggest 20 concepts that should be part of the education of civil servants, politicians, policy advisers and journalists — and anyone else who may have to interact with science or scientists. (Nature)
Parkinson’s Disease: The Pesticide Connection. The rats in a room at the University of Pittsburgh regularly get hit with doses of pesticide. But the researchers in J. Timothy Greenamyre’s lab don’t expose the rodents because of an infestation problem. They give the neurotoxin to the animals to learn more about Parkinson’s disease. (C&EN)
What are you afraid of? Picture the scene: A new shampoo comes on the market, advertised as being ‘all natural’, and devoid of ‘chemicals’. Cue outrage among chemists, decrying the ignorance of the public. Don’t they know that everything is made of chemicals? Something should be done. But what? Aside from some internal unhappiness and hand-wringing, how are chemists addressing a chemophobic public? And where does the problem stem from? (Chemistry World)
Scientists reach consensus on EDC thresholds. It is possible thresholds do not exist for activity of endocrine disrupting chemicals (EDCs) agree scientists representing both sides of the EDC debate. The conclusion is one of several reached during a recent meeting hosted by the EU chief scientific advisor Anne Glover, who is seeking a more constructive debate in several areas of disagreement surrounding the subject. (Chemical Watch)
Toxic flame retardants in furniture could be on the way out nationwide. For decades, U.S. manufacturers have filled upholstered furniture with pounds of toxic chemicals to comply with a flammability standard set by a single state, California. The obscure rule, known as Technical Bulletin 117, brought flame retardants into homes across the country. American babies came to be born with the highest recorded average concentrations of the chemicals among any infants in the world. But California has now thrown out the 38-year-old rule and approved a new one that furniture manufacturers can meet without using flame retardants. (Chicago Tribune)
Generation Toxic. That Perera was even looking at environmental causes of illness was both unusual and unfashionable. For the past two decades, cancer researchers and other molecular biologists have spent much of their time riveted by the genome, believing it would unlock the secrets of disease. But cellular life isn’t determined simply by the blueprints of DNA: to truly understand human health and disease, scientists need to look at both the genome and—to use a term coined in 2005 by Christopher Wild, a cancer epidemiologist—the “exposome.” (OneEarth)
The Toxins That Affected Your Great-Grandparents Could Be In Your Genes. Michael Skinner’s biggest discovery began, as often happens in science stories like this one, with a brilliant failure. Back in 2005, when he was still a traditional developmental biologist and the accolades and attacks were still in the future, a distraught research fellow went to his office to apologize for taking an experiment one step too far: she had accidentally bred the grandchildren of rats being used to test the toxicity of vinclozolin. (Smithsonian)
You do not have to travel very far to find controversy in chemical safety: if you eat canned food there is probably some in your refrigerator. In this case, we would most likely be talking about bisphenol A, a polymerising agent used for manufacturing durable plastics and resins and a poster-child for controversy in chemicals policy.
The concern with BPA is that, while it is widely used in food contact materials, it might act in the body as an oestrogen. Researchers are looking at whether it increases the risk of breast cancer, reduced fertility, abnormalities of the reproductive tract, obesity and accelerated puberty.
A number of EU Member States have restricted the use of BPA in food contact materials intended for infant use. The French Government intends to ban BPA from food contact materials altogether, and the Swedish Government is even planning to ban its use in receipt papers. Such measures are supported by environmental health NGOs and a number of scientists have spoken out against the use of the substance.
Such moves against BPA are not, however, uncontested. The current opinion of the European Food Safety Authority (EFSA) is that BPA at current exposure levels poses no threat to people’s health – a view echoed by the UK Food Standards Agency. Professor Richard Sharpe of the MRC, no stranger to endocrine disruptors, thinks BPA is the wrong target for regulatory control and that research into the toxicity of the compound should desist (Sharpe 2010).
Whose opinion, then, are you supposed to believe? With over 6000 published studies, it is unlikely that you have the personal time or resources to get sufficiently familiar with the research into BPA to decide for yourself who is most likely to be right. So how do you pick the opinion which is most likely to be right?
Further reading: Systematic Review and the Future of Evidence in Chemicals Policy. A new report from the Policy from Science Project, looking at how systematic review techniques used in evidence-based medicine can advance the credibility and utility of chemical risk assessments.
Sign up here for project updates, including alerts for upcoming webinars for discussing report findings.
Reading what the experts think
All of these experts have at some point justified what they think, publishing in some shape or form reviews of the evidence which underpin their opinions. You might expect that reading these and deciding which one is the best presentation of the evidence should be a decent short-cut having to do all that research yourself – after all, that is the whole point of the academic literature review.
Some of these literature reviews will be better than others. Since you would not want to be misled by a review which was either inadvertently or deliberately biased, you would want a literature review to fulfil at least the following six criteria:
1. Stating a clear objective. The objective of the review should address a clear question, such that there is no room for misunderstanding what the review is trying to find out.
2. Applying a consistent method. The method used in the conduct of the piece of research shouldn’t change as the research is being conducted, otherwise (deliberately or not) the researchers risk finding what they think the evidence should be saying, rather than what the evidence actually says.
3. Including all relevant evidence. The process for finding evidence and selecting studies for analysis should deliver a representative sample of the evidence base, otherwise the review risks finding only what a particular segment of the overall evidence says, not what all of the evidence says (sampling and selection bias in finding evidence).
4. Assessing the quality of included evidence. Because the individual pieces of evidence in a review will be of variable quality and should therefore have varying weight in the overall conclusions, each piece of evidence should subjected to the same fair test of quality, so the findings of weaker studies are appropriately downgraded and stronger studies not unduly rejected.
5. Stating the interests of the reviewers. Because the interests of the reviewers can have a detrimental effect on the objectivity of the review findings, or at least an important role in shaping the course of the review, the interests and contributions of each author and contributor to the review should be stated.
6. Accurately reporting findings. The findings reported in the review document are an accurate representation of what the reviewers actually found.
All this probably sounds like a no-brainer. But there is in fact a problem here: analysis of EFSA’s two most recent Scientific Opinions on BPA by this author reveals serious shortcomings in methodology and documentation of the review process (Whaley 2013, PDF – see recommended reading above), on precisely these points.
Are expert opinions sufficiently scientifically robust?
The two Opinions in question are the 2010 Opinion on BPA and the draft version of the next Opinion on BPA, the exposure part, published in 2013. Neither Opinion performed well when compared to what is expected from a scientifically-robust literature review process, in an evaluation based around the criteria listed above.
For example, the objective of the 2010 Opinion was ambiguous, being unclear if it was concerned with evidence which would permit a recalculation of the safe level of exposure for BPA, or if there was evidence which undermined the current calculation. The Opinion addressed the former – but the question of whether or not any new studies permit the safe intake can be recalculated is a very different one to whether or not new studies mean the safe intake level is still credible.
Neither the 2010 nor 2013 Opinions used pre-published protocols, which have an important role in preventing the sort of ad-hoc decision-making which can bias a review’s findings. Nor did either Opinion include a statement of interests and contributions; instead, users have to approach EFSA for a full statement of interests for the Opinion Working Group and Panel members, before having to work out for themselves which interests might have been relevant to the writing of the Opinion.
There were weaknesses in the search and selection processes for finding and including evidence for analysis in the review which suggested that only a partial selection of the overall evidence was analysed by EFSA; nor was there a clear and consistent method for evaluating the quality of included studies, so it was unclear if studies were being correctly weighted in accordance with their methodological quality.
All this doesn’t mean the EFSA Opinions are wrong. What it does mean, however, is that they provide insufficient assurance that they are correct: in the absence of a transparent, reproducible methodology it is an overwhelmingly arduous task to evaluate the likelihood that EFSA’s Opinions represent the best possible synthesis of all the available evidence on BPA – because if you don’t know what they are doing, how can you judge if they are likely to be correct?
This has all been seen before
A study published in 1987 found that of 50 literature reviews published in the top 4 medical journals, only 1 had clearly specified its methods for identifying, selecting and validating information included in the review (Mulrow 1987).
This helped precipitate a step-change in how evidence was reviewed in medicine, catalysing the development of systematic review techniques designed to integrate the fundamental scientific concept of reproducibility of method into the literature review process. These include the publication of protocols prior to conducting reviews, advanced search techniques for assembling all the available evidence relevant to a question, and reproducible methods for evaluating the quality of evidence.
It seems natural, then, that these techniques could also be applied to toxicological research in the course of assessing the safety of chemicals.
Moves are already being made in this direction, with the US Environmental Protection Agency’s IRIS project, the University of California San Francisco’s Navigation Guide, the US National Toxicology Panel’s draft protocols for systematic review; the Evidence-Based Toxicology Collaboration (arguably the first to articulate the concept of systematic review for toxicological data); the Policy from Science Project (the author’s own group), work at Stockholm University, the Swiss Ecotoxicological Institute, and others.
This work is important not only for finally getting clear and transparent statements of what is known about chemical toxicity (rather than statements of merely what is thought to be known). It has important consequences for everyone’s engagement in chemical regulation – because without access to a systematic presentation of the evidence base, how can any of us expect to engage in an informed manner in the debate about chemical safety?
When scientists attack. Spend extended time reading the science press, and it’s easy to think that science is a one-note story about the amazing discoveries that happen in test tubes and laboratories. In reality, there’s a plethora of under-covered science angles, most notably the politics of research funding and science policy. See also: Scientists who attacked EU chemicals policy had industry ties; Scientist with extensive industry ties quits EU advisory panel; Scientists’ Ties to Food Industry Raise Questions in Europe.
Stronger laws steer innovation on the right path. Correspondence. Sir, Your article “Government in danger of stifling bright ideas” (Innovation and the Economy, Special Report, October 17) unfortunately quotes self-interested assertions by 12 large corporations regarding their position on precaution, without adequate discussion of opposing views from businesses or civil society.
California law may cut use of flame retardants in buildings. Officials will have to consider whether flame retardants are necessary for fire safety in light of studies that show that some of the most commonly used compounds have dangerous or unknown health effects.
Chemical Safety Legislation Backed by ANA, Physicians. ANA has long advocated for more tightly regulating toxic chemicals in the health care workplace and the environment. ANA has supported previous legislative efforts to reform the toxic substance approval, monitoring and restriction process, including reform of the Toxic Substances Control Act.
Getting Real About Chemical Risks. Many people assume that the chemicals in their detergents, floor cleaners, and other household products have undergone rigorous safety testing. But little is known about the potential risks associated with most of the estimated 80,000 chemicals in commerce today. While industry tries to dispel links to illnesses that go beyond what science can prove, the public is skeptical because companies have a financial stake in showing their products are safe. This leads both sides to look to the federal government for help.
Health fears over rise in pesticides on fruit. A new report shows that the amount of pesiticides applied by soft-fruit growers centred in Angus, Scotland, leapt by 38 per cent per hectare between 2010 and 2012. This included a 36 per cent rise in the use of chlorpyrifos, an organophosphate insecticide that has been linked to ill health in agricultural workers.
This Is Your Brain on Toxins. So what are the lessons from the human catastrophe of lead poisoning over so many decades? To Nicholas Krystof, today’s version of the lead industry is the chemical industry — companies like Exxon Mobil, DuPont, BASF and Dow Chemical — over the years churning out endocrine-disruptor chemicals that mimic the body’s hormones. Endocrine disruptors are found in everything from plastics to pesticides, toys to cosmetics, and there are growing concerns about their safety.
Firms act on problem chemicals. Three major U.S. companies took steps recently to replace problematic chemicals with safer ones in consumer products. Environmental and health advocates are cheering, predicting that the changes will ripple outward and help reshape the market, potentially transforming thousands of cleaning, personal care, and baby products. See also: Shoppers move Target to address toxic chemicals.