part 2: suppressed results.

In part one of my writeup on survey results, I talked a lot about the file drawer effect and why we end up not publishing some potentially useful results because we don’t have time. In a high-pressure environment where publication in the best journals is important to advance our careers, we often focus that limited time on the manuscripts with the highest potential impact. In some unfortunate cases, that means that professors do not prioritize giving their students the support necessary to publish results from projects, theses, or dissertations.

There’s no doubt that this can hurt younger scientists’ careers. Helping a student aim high and write higher-quality papers is great…. but it can go too far, too.

“There was no specific pressure NOT to publish, but rather my supervisor could not provide useful and supportive feedback and he was never satisfied with any draft I submitted to him for review,” wrote one respondent. “After numerous iterations of my projects over many years, I became discouraged and decided it wasn’t worth the effort to try and publish my results. Others in my lab have had the same or similar experience.”

Today I will talk about something more insidious: when you are discouraged from publishing something for other reasons, like politics, that your data didn’t support your research group’s hypothesis, or that external partners did not understand the results or the underlying science.

(If you want to know more about the dataset I am working with, its small size, and its various biases, I discussed it in part one: click over here.)

As an ecologist, I didn’t think that this happened a lot in our field, at least not compared to other fields where there’s more often commercial connections and money at stake. Perhaps if you are an environmental consultant or doing impact reports for the government or companies. But in a purely academic community I assumed that it was a fairly rare occurrence for results to be kept out of publication.

One thing quickly became obvious. It does happen, sometimes. There are lots of reasons, some of which are highly case-specific, i.e. the government of the researcher’s native country didn’t allow him/her to import her samples in the end, after all…. but there are some common patterns, too.

With such a small sample size – 40 of the 184 respondents reported this happening – and also the fact that I made it clear online that I really wanted to hear from people who had been discouraged from publishing, it’s impossible to say how prevalent such events actually are. The proportion of responses does not reflect the proportion of total scientists who have had this experience.

I can certainly say that comparatively, many fewer unpublished papers are due to these events than due to the self-created file drawer effect. Two thirds of survey respondents said they had at least one unpublished dataset, if not a handful or more, even though many were just in the first five years of their research careers.

The file drawer effect means that there are tens of thousands of unpublished datasets out there, maybe 100,000. Many probably have no significant results, since some of the most cited reasons for not publishing were inconclusive data, needing to collect more data, and doubting that the results would be accepted by a high impact journal.

Other pressures happen in a smaller number of cases, but primarily for the opposite reason: results did show something interesting, but maybe not what someone – a supervisor or a government employee – wanted to see.

And while I cannot draw any conclusions about prevalence, I can (hopefully) draw some conclusions about why this happens and who it happens to.

A brief table of contents:

First, student-specific challenges.

Second, government and, to a lesser extent, industry challenges.

Third, “internal” and interpersonal political challenges.

Students Bear The Brunt of It

“As a grad student, the concept of this is crazy to me,” wrote one respondent. “In many ways and instances, publications are the currency by which scientists are measured against one another. Thus, not publishing work seems counterintuitive to me. I’d like to hear the reasons behind why it happens.”

Well, dear student, there are many. And being discouraged not to publish in fact seems to happen mostly to students. Here’s who the 40 survey respondents who reported being pressured not to publish their work were:

title at timeMostly students. One explanation is that as we go along in our careers, we get a better concept of what is good and valuable science, and make some of the decisions to jettison a project ourselves rather than being told by a supervisor. We also become more and more crunched for time, meaning that we make more of these types of prioritizations before it gets to the point of having someone else weigh in.

But that’s not the only explanation. Let’s look first at when a direct supervisor was involved. With 32 responses of this type, it was about twice as common in my dataset than when an external person pressured a respondent not to publish. In these supervisor-related cases, it was most frequently a tenured or tenure-track professor discouraging a graduate student from publishing a chapter of their thesis or dissertation.

titles for direct

And as discussed in part one, part of the issue was that these driven supervisors were strapped for time and transferred their own expectations about significance of results and journal quality onto their students, even if students would have been happy settling for a lower-impact publication.

Sometimes this is very appropriate, sometimes less so. Where this line is drawn probably depends on your goals in science.

“A paper was published, but it excluded the results that I found the most interesting because they were not in line with the story that my advisor wished to push,” one respondent wrote of bachelors thesis research. “Instead, results from the same project that I though were not well thought-out were published in a way that made them seem flashy, which seemed to be the main goal for my tenure-track advisor.”

Another respondent had a similar story with a different ending, about work done as part of a masters thesis.

“The situation was not resolved; I just ended up not publishing,” he/she wrote. “I wanted to publish, as I considered the results to be high-quality science and the information very useful to disseminate, but I could not agree to change the research focus entirely to suit my supervisor’s personal interests.”

You can see both sides of the coin in some cases. What is the goal? To advance scientific theory and knowledge, or to share system-specific data that might help someone in the future? Ideally, a manuscript does both, but sometimes that’s not possible and just the second is still a good aim.  In some cases the supervisor is probably guiding the student towards using their data to address some question larger than the one they had initially considered. But, as the bachelors thesis respondent noted, it’s not always appropriate to do so – some people think that overreaching and drawing conclusions based on data not really designed to do so is a big problem in some fields.

“Some datasets and analyses I have collected and analysed don’t tell a clear story that would be readily publishable given the current state of how research articles are assessed for impact thus I tend to move on to things that tell a better story,” wrote another respondent. “This feels disingenuous at times though perhaps it is how science moves forward more quickly.”

A surprising amount of the time, supervisors discouraged students from publishing because the results turned out to not support their hypothesis. This was actually the most common single reason that a supervisor told a student not to publish. I may be naive, but it’s hard for me to think of a situation in which this is not just straight-up bad.

reasons for directI was quite explicit to ask whether the results did not support “our” hypothesis, or whether they did not support a supervisor, department, or company’s hypothesis. Sometimes the two overlapped, but most of the time when this happened the respondent selected the second option: the researcher themself might not have been surprised by the results, but the supervisor, lab group, or company did not like them.

(About 60% of the 32 responses came from ecology and evolution, but many also came from other fields.)

fields for direct

This really surprised me. In our training as scientists it is drilled into us that we might learn as much from a null result or a reversal of our hypothesis as we would if our hypothesis was supported – maybe even more, because it tells us that we have to carefully look at our assumptions and logic, and can lead us down new and more innovative paths.

In the U.S. at least, a substantial proportion of the population just has no respect for science. Whether its climate change deniers or anti-vaxxers, as a science community we tell them: go ahead, prove us wrong! Science is very open to accepting data that disproves something we had previously thought was true. We try to tell the public that we are not close-minded, that we are following evidence, and that if the evidence showed us something else, we’d still accept it.

On some small scale, that might not be true, and it’s very troubling. Without knowing more about the research in question here, it’s impossible to say much more. But it’s not a very inspiring trend. And again: this was happening coming from direct supervisors who were mostly in academia and shouldn’t have had a financial or political conflict of interest or anything like that.

And it also has potentially big implications for the sum of our community’s knowledge. Luckily there are so many researchers out there that probably someone else will ask the same question and publish it eventually, but this sort of attitude can delay learning important and valuable things.

“Unfortunately it’s hard to tell what could become interesting later, or what could be interesting to another researcher, so it’s too bad that these results never see the light of day,” wrote one early-career biologist. “What’s more concerning to me is the tendency of some researchers in my field to ignore or leave out results that they can’t explain, or worse, that contradict their pet hypothesis.”

When pressure came from an external source – someone not supervising the study respondent – the prevalence of this reason for discouraging publication was even higher. The data not supporting someone’s hypothesis rose from roughly two-thirds of respondents citing it, to almost half.

reasons externalAnd relatedly, the person doing the pressuring was afraid that the results would make them, their group, or the government look bad. In other words, these are classic cases of repressing research, the worst case scenario that we think of!

Governments are Not Always Great (for Science)

Sometimes, this external pressure came from within academia, but it was also often from governments.

Screen Shot 2015-10-07 at 1.38.56 PM“Yes, the results were published, yes it created an public uproar, yes all authors were chastised by the agency and external company, and yes all subsequent follow-up research papers on the topic were expressly forbidden,” wrote one federal government employee. “There are considerable research accomplished by state and federal government agencies. Much of those data results never see the light of day because the results may be divergent from what the chain of command’s perspective or directive may be, I.e. support the head official’s alternative energy, logging harvest, endangered species delisting, stream restoration, etc. policy.”

It’s clear that one place where state, local, and federal government officials can be particularly destructive is Canada. Apart from the cuts to research funding which have been hitting many countries, it’s been discussed by people far more knowledgeable than I that the government literally muzzles its scientists by not allowing them to talk to the media, among other policies: see here, here, and here.

Here’s what one anonymous survey respondent had to say: “The Canadian government has been muzzling scientists for years…I was just the latest in their ‘Thou Shalt Not Publish’ scheme. If the research you’re doing will make them look bad in any way, you’re not allowed to publish the results without fear of massive repercussions: job loss, degree removal, job losses of your superiors if they can’t fire you, being blacklisted in the scientific community, being blacklisted for grants, etc.”

Multiple survey respondents cited the Canadian government. So, about those elections coming up….

Consultants and researchers in the corporate/industrial sectors are often muzzled as well, but many of them are aware of this from the time they are hired.

“It is simply understood that if the research results from work we do for clients are inconvenient, they will attempt to redact the reports as trade secrets,” wrote one consultant. “They own the data so they are often able to do this. But not always.”

But even if companies are upfront about data ownership policies, it can still feel tough. One person told me that it was discouraging not to be able to get a patent and get credit for his/her work because a company owned all the intellectual property rights and would use the discovery as proprietary and secret until it was no longer profitable to do so.

In a variety of fields, there’s also some crossover between the industrial and academic sectors of research. Companies often provide funding to students or research groups working in an essential location or on a related topic. The companies shouldn’t be able to use their influence to suppress results, but in some cases they do seem to.

This is actually what happened in the case that inspired me to create my survey: the International Association of Athletics Federations squashed survey results showing that a huge proportion of championships competitors were doping. They were not involved in the research itself, but had provide access to the athletes, and thus felt like it was their prerogative to police the results.

One survey respondent said that he had been let go from his position after publishing research about the effects of pesticides, and had heard a researcher with industry ties imply that the same thing would happen to someone else publishing similar research.

Several people in environmental and earth sciences fields mentioned this happening to people that they knew or had talked to, but it’s hard to pin down other than in news stories.

We Can Be Our Own Worst Enemies

Finally, other politics are more about internal power dynamics, be it within a department or within a research field.

“A person, invited late to the project, was asked to provide simple review in return for coauthor ship,” wrote one respondent. “They hijacked the project and it is still unpublished four years on.”

It’s pretty tragic to see a good experiment, or maybe a whole grant that some agency spent hundreds of thousands of dollars on and researchers spent years of their lives on, get derailed by interpersonal problems and arguments about data ownership or authorship.

In many fields the community of specific experts is fairly small, so you are likely to have to work with people again, or have them review papers, etc etc etc. The problems are hard to resolve once they begin.

It was also clear that sometimes people nixed manuscripts because they didn’t understand the science or the value of this. Sometimes this meant a bureaucrat at a funding agency, but sadly, sometimes it also came from within the scientific community itself.

“Because my scientific community is so small, in some cases only one review has been given by a local expert, and of course the editors don’t have time to fact-check, but my paper will not be accepted because these few experts are, as I perceive it, not wanting recent data contrary to results from their systems to be published, and assume that someone with an M.Sc. cannot be a diligent scientist, in many cases providing lots of evidence in reviews that they have not read the manuscript with care… possible skipping entire sections,” wrote one student.

There’s even outright theft sometimes.

“The results were made partially public at a conference,” wrote one researcher. “Another researcher who has hard feelings towards my former supervisor, and viceversa, started to use the date as if it was a ‘public domain information’ and later my supervisor considered that the publication is not worth going out. The problem has not been resolved yet.”

A Reminder

This has been, in some ways, a worst-ever tour of the scientific research community. We all know someone who has had some terrible experience with their research.

But many of us have had relatively happy tenures in science and research. At least in my field, ecology, I can say that the vast majority of people are good people and fun to work with. It’s part of what I love about my job. If the only people around me were those who stole results, bullied me into not publishing, constantly asked me to change the focus of my research, or demeaned what I did because I was a graduate student, I would quit.

But here I am, and I’m happy! Such people do not make up the majority in our fields. But it’s worth remembering that even one major interaction like this can seriously discourage people from continuing to do research. There are lots of other jobs out there, and if the research environment is malevolent it’s easy to feel that the grass is greener on the other side.

So: with the knowledge that there is some scummy behavior going on, can we try to be nicer and kinder to one another? After all, our goals are to advance scientific knowledge and to create more capable, creative, and conscientious scientists.

Thanks to all who participated in the survey. I hope it has been interesting and helpful to read about.

the contagion of perfectionism & the scientific publication bias.

A few weeks ago I sent out a survey to many of my scientist friends. I wanted to know: why does some research stay unpublished? Those outside academia or research might think that science always proceeds in a linear fashion. A person does a study, they publish it, now it is out there for other scientists to reference. Once research is performed, it is a known quantity. But that’s not necessarily true.

For any number of reasons, a fair chunk of research never makes it to the publication stage. Sometimes it’s because it’s bad research: it is biased or the methods are bad, so during the peer review process the paper is rejected. This might not be through any real fault of the scientists. The problems might have only become apparent after the research was completed. This is pretty inevitable, and can lead a research group to design a second study that is much better and really gets at their question.

But does all the good research even get published? No, definitely not. There’s research out there that remains unpublished even though it probably could have been.

Some possible reasons for this are that the researchers ran out of time to write up the results, or the results just didn’t seem very interesting, or their hypothesis was rejected. For these reasons people might choose to focus on another project they had going at the same time. But that leaves a gap in the record of published science: results with bigger effect sizes are published proportionally more often than null results. Results with no effect might be left in a drawer to be published later, or never.

This phenomena is called the “file drawer effect” and is a major contribution to a bias in publication which is problematic for many reasons, which I’ll discuss later. Here’s a nice paper on the file drawer effect.

With my survey, I wanted to get at why people don’t publish. First I asked about how much research they leave in the file drawer, so to speak, out of their own choice. Then I asked how often other people pressured them to avoid publishing, and why. I’ll get to that second question in part two of this post.

First, as a caveat before getting to what people told me. The responses certainly don’t represent the whole science community, and I can’t draw any conclusions about frequency of the types of things I’m asking about. I had 182 responses, which is not a lot, and the majority were from ecologists and evolutionary biologists relatively early in their careers. The survey was spread by word of mouth so this is just a function of who I know.

Here’s some data on who responded to my call:


Age of researchers

(I’ll also add that most respondents were in academia, but there were also some who worked for government research institutes or companies. I’ll get more into that in part two, but for now I’m going to write primarily from the academia perspective. Just be aware that there are some non-academia responses in here as well.)

Now. One of the first questions I asked was, “How many papers or reports worth of results of your own work remain unpublished, by your own choice?”

It’s obvious now that I could have worded this a little bit better. Some people include all of their unpublished work in this answer, while others said that just because something hadn’t been published yet didn’t mean that it would never be published, and left some work out of the count. They may be right about that: some work does eventually get published years later, when researchers finally have a chunk of free time and nothing “more important”.

(“Pressure was not direct – just lack of support to move the paper forward,” one survey responder wrote of his/her supervisor. “Ultimately he approached me to finally publish the work – after more than 20 years!”)

But that data that you swear you will write up one day can also remain in the file drawer indefinitely.

In any case, here’s what I found:


The other thing that is unclear in the results is how realistic it is to publish all of these datasets – are they each an individual paper, for instance? Some people take a dataset and divide it into as many pieces as possible so that they can get the most publications out of it, when in reality publishing all the data together would have made a more interesting, meaningful, and high-impact single manuscript. So has someone doing research for six years really accumulated ten unpublished datasets? Perhaps they have. Meanwhile, I am impressed by the few people who had been doing research for 25 or even 40 years and had seemingly published every worthwhile dataset they had ever collected. These people must be writing machines. (And I say that as someone who writes quite a lot!)

Adding a regression line is probably inappropriate here, but let’s just say that in the first ten years of their career (depending on how they are counting, this is a bachelors thesis, a masters, a PhD, and maybe a postdoc or two), many people accumulate four or five studies that they could have published, but they didn’t. After that they might accumulate one every five or ten years. It makes sense that more of the unpublished papers come early in the career because people aren’t yet adept or fast at writing papers. They also don’t have as much experience doing research, so data from projects like bachelors theses often go unpublished because of flaws in study design or data collection. These mistakes are what eventually lead us to learn to do better science, but they can keep a piece of research out of a top journal.

As of 2013, there were about 40,000 postdocs in the United States. Add that to the rest of the world and there’s potentially a lot of unpublished research out there – clearly over 100,000 datasets worth! (Is that good or bad? Both, and I’ll get to that later.)

The answers are partially biased, I am sure, by the differences in productivity and funding between different researchers. This might depend on what kind of appointment the researcher has – is their job guaranteed? – and how much funding they have. A bigger lab can generate a lot more results. But someone still needs to write them; labs might go through phases where the writing falls primarily on the PI (primary investigator, a.k.a. lab head) or other phases where there are highly productive postdocs or precocious PhD students who also get a lot of papers out the door.

And one of the biggest constraints is, of course, time. With pressure to publish your best research in order to get that postdoc position, to be competitive for a tenure-track job, and to eventually get tenure, if researchers have to choose between publishing a high-impact paper and low-impact one they will certainly focus their energies on the high-impact results. The results that were confusing or didn’t seem to show much effect of whatever was being investigated might stay in that file drawer.

One thing that was clear is that this problem of time as a limiting resource is contagious. Later, I asked people if they had ever been discouraged from publishing something which they had wanted to publish. Of the 32 cases where a supervisor discouraged publishing, two answers as to why emerged as particularly common.

context_perfectionismLet’s look at the “more data was needed” issue first. What I offered as a potential response in the multiple choice question was, “My supervisor thought that we needed to collect more data before publishing, even though I thought we could have published as-is.”

In some cases, the supervisor might be right. Maybe more data really was needed, maybe the experiment needed to be replicated to ensure the results were really true, maybe the team needed to do a follow-up experiment or correct some design flaws. After all, the supervisor should have more experience and be able to assess whether the research is really good science which will stand up to peer review.

“Simply, what I thought were publishable results were probably not worth the paper it would be printed in,” one responder wrote of research (s)he had done during a bachelors thesis, but which the supervisor had not supported publishing. “The results did serve as the basis for several other successful grant applications.”

But at the same time: as Meghan Duffy recently noted on Dynamic Ecology, perfect is the enemy of good. That can go for writing an email to your lab, and also for doing experiments. In the discussions of her blog post, someone noted that “perfect is the enemy of DONE” and Jeremy Fox wrote that often graduate students can get into the rut of wanting to just add one more experiment to their thesis or dissertation, so that it is complete, but at some point you just have to stop.

“I have not directly been pressed not to publish, but I have 2 paper drafts which have not been published yet,” wrote another respondent. “I wrote them as a PhD student and now I think they will be published, but I have the feeling that for some period, one of my supervisors did not want to publish them because it was just correct but not perfect enough.”

If more data should be collected, but probably never will be, does that mean that the whole study should sit in the file drawer? If it was done correctly, should it still be published so that other people can see the results, and maybe they can do the follow-up work? Different researchers might have different answers to this question depending on how possessive they are of data or an idea, or what level of publication they expect from themselves. But if a student, for example, is the primary person who did the research, their opinions should be taken into account too.

Why is this publication gap a problem?

That gets into the second idea: that as-is, the research won’t be accepted into a top journal.

Ideally, this shouldn’t matter, if the research itself is sound. There are plenty of journals, some of them highly ranked, which accept articles based more on whether the science is good and the methods correct, rather than whether the results are groundbreaking.

(Unfortunately, these journals often have big publication fees, whereas highly-ranked journals have a large time investment but publication is free. Plos One, one of the most well-known journals which focuses on study quality rather than necessarily the outcome, is raising its publication fee from $1350 to $1495, which must be paid by the author. For some labs this doesn’t matter, but for other less-flush research groups the cost of open-access publishing can definitely deter publication.)

It is important to get well-done studies with null results out there in the world. Scientific knowledge is gathered in a stepwise fashion. Other scientists should know about null results, arguably just as much or more than they should know about significant results. We can’t move knowledge forward without also knowing when things don’t work or don’t have an effect.

Here’s two quick examples. First, at least in ecology and evolution, we often rely on meta-analyses to tell us something about whether ideas and theories are correct, or, for example, how natural systems respond to climate change or pollution. The idea is to gather all of the studies that have been done on a particular topic, try to standardize the way the responses were measured, and then do some statistics to see whether, overall, there is a significant trend in the responses in one direction or another. (Or, to get a little bit more sophisticated, to see why different systems might respond in different ways to similar experiments.) This both provides a somewhat definitive answer to a question, and makes it so that we can track down all of the work on a topic in one place rather than each scientist having to scour the literature and try to find every study which might be relevant.

If researchers only 20% of the scientists studying a question find a significant effect, but these are the only results which get published, then literature searches and meta-analyses will show that there is, indeed, a significant effect – even if actually, across all the studies which have been done (including the unpublished ones), it’s a wash. Scientific knowledge is hindered and confounded when this happens.

A second example. When you are designing a study, you search the literature to find out what has been done before. You want to know if someone else has already asked the same question, and if so, what results they found. You also might want to know what methods other people use, so that you can either use the same ones, or improve them. If research is never published, then you might bumble along and make the same mistakes which someone already has made. The same flawed study might be performed several times with each person realizing only later that they should have used a different design, but never bothering to disseminate that information. (And sure, you can ask around to find unpublished results, but if there’s no record of someone ever studying a topic, you’re unlikely to know to ask them!)

Almost everyone in the scientific community acknowledges that the publication bias towards positive or significant results is problematic. But that doesn’t really solve the problem. It’s just a fact that null results are often much harder to publish, and much harder to get into a good journal. And considering the pressure that researchers are under to always shoot for the highest journals, so that they can secure funding and jobs and advance their careers, they are likely to continue neglecting the null results.

“I think a lot of pressure comes from the community rather than individuals to avoid publishing negative results,” one early-career ecologist wrote in a comment. “I think negative results are useful to publish but there needs to be more incentives to do so!”

This pressure can be so great that, I was told in a recent discussion, having publications in low-impact journals can actually detract from your CV, even if you have high-impact publications as well. Two candidates with the same number of Ecology Letters or American Naturalist or even Nature papers (those are good) might be evaluated differently if one of them has a lot of papers in minor regional or topic-specific journals mixed in. Thus, some researchers opt for “quality not quantity” and publish only infrequently, and only their best results. Others continue to publish datasets that they feel are valuable even if they know a search or tenure committee might not see that value, but consider leaving some things off their CV.

One thing I’d like to mention here is that with the “contagion”, students are sometimes affected by their supervisors’ standards of journal quality. While a tenure-track supervisor may only consider a publication worthwhile if it’s in a top journal, a masters student may be greatly benefited by having any publication (well not any, but you see my point) on their CV when applying for PhD positions. I also know from my own experience that there is incredible value, as a student, in going through the publication process as the corresponding author: learning to write cover letters, respond to reviewer comments, prepare publication-quality figures, etc. Doing so at an early stage with a less-important manuscript might be highly beneficial when, a few years later, you have something with a lot of impact that you need to shepherd through the publication process.

There are many good supervisors who balance these two competing needs: to get top publications for themselves, but to also do what is needed to help their students and employees who might be at very different career stages. In many cases, of course, supervisors are indeed the ones pushing a reluctant graduate to publish their results!

Unfortunately, this is not always the case. Again, because of the low number and biased identity of survey participants I can’t say anything about how frequently supervisors hinder their students in publishing. But I think almost everyone has some friend who has experienced this, even if they haven’t themselves.

“I have been in the situation where a supervisor assumed that I would not publish and showed no interest in helping me publish,” wrote one responder. “As a student, being left hanging out to dry like that is rough – might as well have told me not to publish.”

“Depending on the lab I’ve been in, the supervisory filter is strong in that only things deemed interesting and important by them get the go ahead to go towards publication,” wrote another. “Thus, the independence to determine what to publish of the multiple threads of my research is lacking in certain labs and management structures.”

That obviously feeds in to the publication bias. So how do we get past it, in the name of science? There aren’t a lot of answers.

Why is the publication gap maybe not so bad?

At the same time, it’s clear that if all this research (100,000 or more papers!) was submitted for publication there would be some additional problems. Scientific output is roughly doubling every nine years. There are more and more papers being published; there are more postdocs (although less tenure-track professor positions) in today’s scientific world, and I’m pretty sure the number of graduate students increased after the “Great Recession”, about the time when I was finishing my bachelors degree and all of a sudden many of my classmates’ seemingly guaranteed jobs disappeared.

This puts a lot of stress on the peer review system. Scientists are not paid to review research for journals, and reviewing may or may not be included as a performance metric in their evaluations (if it is, it’s certainly not as important as publishing or teaching). With more and more papers being submitted more and more reviews are needed. That cuts time out of, you guessed it, doing their own research. It’s a problem lots of people talk about.

Screen Shot 2015-10-04 at 10.21.16 AM

Others lament that with so many papers out there, it’s getting harder and harder to find the one you need. Science is swamped with papers.

Even without publishing in a journal, there are other ways to find data. For instance, masters theses and PhD dissertations are often published online by their institutions, even if the individual chapters never make it into a peer-reviewed journal (perhaps because the student leaves science and has no motivation to go through the grueling publication process). But this type of literature can be harder to find, and is not indexed in Web of Knowledge, for example. So if it’s the data or methods you need, you might not find it.


I’m not particularly convinced by the argument that there’s too much science out there. Research is still filtered by journal quality. Personally, I read journal tables of contents for the best and most relevant journals in my field. I also have google scholar alerts set for a few topics relevant to my research, so that when someone publishes something in a place that would be harder to find I know about it. This has been useful. I’m glad they published it, even if it’s in an obscure place.

With that in mind, I wonder if there is a way to publish datasets with a methods description and some metadata but without having to write a full paper.

There are, of course, many online data repositories. But I don’t believe people use them for this purpose as much as they could. It is now becoming common for journals to require that data be archived when a paper is published, so much of the data in these repositories is simply data that actually already has been published. In other cases people only bother with publishing a dataset as-is if it is large or has taken a lot of time to curate, and might be of particular interest and use to the community. Smaller datasets of pilot projects or null results are not often given the same treatment.

And while published datasets are searchable within the individual repositories archives, they don’t show up in the most common literature search tools, because they aren’t literature: they are just data.

Is there a way that we could integrate the two? If you have five papers-worth of data that you don’t think you’ll ever publish, why can’t we have a data repository system which includes a robust methods and metadata section, but skips the other parts of a traditional manuscript? If this were searchable like other kinds of literature, it could contribute to more accurate meta-analyses and a faster advancement of science, because people would be able to see what had been done before, whether it “worked” and was published with high impact or not. The peer review process could also be minimal and, as with code or already existing data archives, these data papers could have DOI’s and be citable.

But I’m not sure if this is realistic (and honestly, I haven’t thought through the specifics!). Science seems slow to change in a lot of ways. Methods change fast. Open access and online-only publishing have swept through to success. But creative ideas like post-publication review, preprints, and other innovations have been slower to catch on. These types of ideas tend to generate a group of fierce supporters, but to have a difficult time really permeating the scientific “mainstream”.

The scientific community is big – how can we change the culture to prevent our large and growing file drawers full of unpublished results from biasing the literature?

Stay tuned for part two of this series, about other reasons that people are pressured not to publish results – for instance, internal or external politics, competing hypotheses, stolen data. Part two will be published later this week. If you want to take the survey before it goes up, click here.

recent sciencing.

After a conference in Uppsala, Sweden, I had the chance to catch up with a lot of friends from my masters program, many of whom recognized how totally awesome Sweden is and decided to stay in Uppsala to do their PhD's! left-right: Willian from Brazil, me, Lore from Mexico, Sergio from Colombia. It's fun to be part of such an international bunch.

After a conference in Uppsala, Sweden, I had the chance to catch up with a lot of friends from my masters program, many of whom recognized how totally awesome Sweden is and decided to stay in Uppsala to do their PhD’s! left-right: Willian from Brazil, me, Lore from Mexico, Sergio from Colombia. It’s fun to be part of such an international bunch and I was so, so happy to see them all again.

I always tell people when I’m interviewing for a job, ski racing prepared me very well for the constant criticism and failure you experience in science. Every time you try to publish something, you receive harsh critiques during the peer review process, even if the paper is eventually accepted. The only way to improve is to continually solicit these beatdowns, then lick your wounds and try harder.

Skier or worker (or both), we're all just busy little worker bees in the end.... trying industriously to make ourselves better at what we do.

Skier or worker (or both), we’re all just busy little worker bees in the end…. trying industriously to make ourselves better at what we do. This bumblebee was having a fantastic time in the Uppsala botanical garden.

When that happens, I think back to training with the Craftsbury Green Racing Project, and surviving things I didn’t think I could survive. For instance: the one time Pepa asked us to do max-level 2-minute intervals on the SkiErg before eating anything in the morning, then gave us a quick breakfast, and then we rollerskied for like, four hours. Maybe some people went for five? When we got our instructions, I thought it was impossible that I’d finish. But I did.

So when I science hard and everything feels (temporarily) like a failure, I remind myself: you can get to the finish line. You are tough. Workouts like that one taught me that while I might never be the best (at ski racing, that’s for sure), I can do a lot of suffering and finish tasks that might seem impossible. Positive self-talk helps!

Workshops and conferences are generally a lot more fun than that workout was. If immersing yourself in scientific research – whether that’s by long and grueling trips to the field, toiling for hours over the lab bench, or frying your eyeballs coding on a computer – is like training, then emerging out of that world into a conference is like the first ski race of the year.

All of a sudden you get to check out what other people have been up to, and test your ideas and your data against them. Of course there’s no single winner at a workshop or conference, but that nervousness and excitement about seeing the community again, and revealing your activities to the experts in the hopes that they will be impressed, really does give a bit of the same feeling as West Yellowstone, the first Eastern Cup, or the first college carnival of the year. Along with the job at hand is also fun socializing, hearing about the new job someone has taken, seeing how much their kid has grown in a year, and rehashing stories from the past.

A bonus of the course in Fribourg was that we ate raclette for dinner in the botanical garden.

A bonus of the course in Fribourg was that we ate raclette for dinner in the botanical garden.

As a PhD student, I still feel like I have to prove myself every time I meet a big name. And at the workshop, in Fribourg, Switzerland, there were times when I felt like, gosh, I’m never going to make it in this world.

One of the lecturers does research about some of the same questions that I study, and is even using some of the same study organisms. He has done a lot of cool research in the past, and I’ve read many of his papers and cite them. It was a great opportunity to see him give a two-hour lecture on things that apply to me so directly.

But at the end of his presentation, he talked about a major new grant he had received to replicate his research in four other countries, to use experimental ponds all throughout Europe, and do a few other things. He was answering my questions, but on a vast, globally-replicated scale!

(Briefly, we study freshwater science, so organisms like insect larvae, crustaceans, and fish that live in streams. Also ponds and lakes and rivers, but mostly streams.)

And here I am in Switzerland. He will have dozens of streams all over the world; I am working so hard to check on ten streams near the German border. How can I possibly compete?

“Maybe I should just go get a job at McDonalds,” I joked to another workshop participant.

And yet, the workshop was really helpful and at other times made me feel like all of my ideas were coming together into something that could be really meaningful. I began to see how my data from those ten streams could link together with experiments I am doing in the lab, and go into one big framework to assess ecosystem functioning and explore various future land use schemes.

There were also some comical moments. Students were given the opportunity to bring a poster to the workshop, but in the end nobody else did besides me. That’s like getting to a race and realizing you’re the only entrant. Would you still do it?

Probably you would – we’re all masochists, looking for a good workout, and chances are you would have driven a long way to get to the race – but it would definitely feel silly.

Instead of having 20 people clustered around one poster, the organizers asked me to put the powerpoint of my poster up on the projector and give a ten-minute presentation in front of the whole group in the lecture hall. I hadn’t prepared anything, but I tried to give a concise and not-too-rambling summary of what I’m up to. The responses were positive, even from the guy with millions of dollars in research funding and study sites all over the world.

So I left the workshop feeling that I hadn’t “won”, but that I now had a much better idea of what else I needed to do when I got home and what I needed to focus on in “training”. I had a vision for how to succeed.


This talk won me $50!

Fast forward a few days and I was in Sweden for the International Tundra Experiment (ITEX) 2015 meeting. Climate change researchers from around the world gather every year or two to report results and work on synthesis papers where they pool a lot of data from alpine, low arctic, and high arctic research to make solid conclusions about the effects of climate change on these plant communities.

The conference was a bit of an emotional rollercoaster. I was giving a talk, and was quite nervous: after all, it’s a quite specialized conference, so you know that everyone in the audience knows as much or more than you do about the tundra. If you do something wrong, or don’t know what you’re talking about, you can’t really hide.

At a lot of bigger conferences, people might be more expert than you in general theories and ideas, but probably only a few know your study system inside out. But at a conference like ITEX, every single one of them does. It ups the stakes a little bit. The nerves!

On top of that, the first speaker in my session talked for twice his allotted time period! The moderator was trying to cut him off, but he just kept going. By the time I gave my talk people had been sitting in the room for an hour and a half and were supposed to be on a coffee break already. There’s no way they will stay focused, I thought. I’m screwed.

But the talk, titled “Changes in process, not pattern, after a decade of warming in Adventdalen tundra vegetation” and based on research I did for my masters degree on the polar island of Svalbard, went really well. I actually won the second prize for student talks.

(The winner talked about release of biogenic volatile organic compounds – the chemicals that make fruit smell good, attract pollinators to flowers, or deter herbivores – and for successfully explaining biochemistry to plant ecologists, she definitely deserved the prize! I’m going for the win next time though.)

The next day, the paper I had written based on that same data came back from a prominent ecology journal…. with a rejection. I mean, I hadn’t thought the paper was the greatest thing in the world, but I had been happy with the draft I had submitted and proud that I had made the whole thing myself. I thought I had good ideas. The message was that… I didn’t.

But here’s the thing about science: even while eviscerating my narrative, the reviewers gave some incredibly helpful suggestions and made it clear that they thought that the work was valuable and should be published. There was the stick, but also the carrot. It’s like a coach saying, “I see potential!”

And that gets back to the positive self-talk. In ski racing and in life, isn’t so much ultimately about convincing ourselves that we have potential?

sweet new hampshire home.


A bit ago I made a very quick trip home to New Hampshire. I was lucky to have beautiful sunny weather the whole visit!

My parents recently got a new dog. She has a lot of energy, so she needs a lot of exercise. But she also has heartworms (the cure is in progress!) so she can’t exert herself too too much. This has resulted in a lot of really long walks – not runs, but walks. So I got to visit some of my favorite places.

New Hampshire is not a well-known state within the U.S., and even less so outside the country. I sometimes say I’m from Vermont because there’s a better chance people will know where it is (and I did live there for two years, more recently than I ever actually lived in New Hampshire, so it’s only a partial lie!).

So what is New Hampshire? This is New Hampshire.

With my mom and Missy, we walked up Pinnacle, an iconic hill in Lyme (also in the top photo).

pinnacle 2

On a Sunday morning the whole family headed out to Trout Pond, which is a bit of a walk from my house, but a lovely conservation area established in 1990. It is so quiet out there. This is why we love it so much.

trout pond 2

trout pond 1

The hay was also getting cut in our hayfield, but I escaped before it was time to put it in the barn. Sorry mom and dad.

hayfield 1

hayfield 2

I also ran up Smarts Mountain, at 3,240 feet the tallest peak in the surrounding towns and a favorite spot. It has a fire tower on top which offers awesome views… but it is currently closed by the Forest Service for repairs. It has been closed for over a year now, but no repairs have been made. This makes me angry. You suck, Forest Service.

Since the peak is forested, the views from below the tower aren’t as good. But you can find some openings in the trees along granite outcroppings, and they are great. I love Smarts.


It’s a mountain, but it sure isn’t Switzerland.

I love New Hampshire.

training camp in the Jura.

Skiwalking in the Swiss Jura. (Photo: Roli Eggspühler)

Coming from North America, I often think that the other side of whatever country I’m in is very, very far away.

Happily, here in Switzerland things are a little closer together. I live in Zürich and while the nearest big mountains are at least an hour away, nothing is very far. Going south or southwest through the Alps takes a few hours, but driving across the Swiss Plateau to the French border is easier.

A few weeks ago I was able to take part in a training camp in Les Cernets, which is on the border with France. Literally, after dropping our bags off at the inn where we were staying, Fabian and I ran up a hill a few kilometers and peered into the European Union. We followed a well-marked trail and there was a small monument at the top of the height of land. Anyone could take this route into France, although of course you have to get into Switzerland first, which is no easy feat.

(Certainly there was no border station on our running trail; even the one on the main road in Les Verrières, the bigger town, appeared to be minimally manned and just waved cars through without stopping.)

The camp I joined was with the Swiss Academic Ski Team (SAS), a group of college and older athletes. Once you are a member (I’m not), you’re a member for life, so a few masters-aged athletes also join us and sometimes kick our butts.

cowsAt camps we train hard, double sessions a day like the pros, but only for a few days. I can’t speak for the others, but for myself, I then go back to work, train fairly minimally, and engage in magical thinking to assure myself that these few days will somehow make a difference come winter…

Ironically, the team doesn’t have any athletes I’ve met so far from the French part of Switzerland. But in an effort for geographic fairness and also to keep things new and interesting, we went there.

We spent three days in the Jura mountains. It’s at the same time remote and not remote; growing up in the Upper Valley of New Hampshire and Vermont, I felt right at home. The area is a mix of farms and forest, with some small homestead always hidden behind the next roll of the hill. But the city of Neuchâtel isn’t far, and in no time at all you are back on the big lake, feeling like you’re in metropolitan Switzerland.

There are a lot of dairy farms in the Jura. We missed, by just a few days, the annual festival where the cows walk from the high meadows down to town with flowers braided around their horns. On the main road you can find an unmanned, automated cheese vending machine with the local wares.

morningThis is the region that absinthe comes from, and you can imagine perfectly how even when it was outlawed, production continued just the same. There are infinite places to hide things and you can’t travel too fast on the country roads. All you need to do is call your neighbor to warn him someone was coming, and he could take care of his materials no problem.

The mascot of the Val de Travers region of Canton Neuchâtel region is a small green fairy, and it is plastered everywhere.

Come to the grocery store! With the absinthe fairy.

Take the train! With the absinthe fairy.

Stay at our hotel! With the absinthe fairy.

Here’s some highway information! With the absinthe fairy.

On our last night we tried some absinthe, which probably ruined our training effect. We stuck to one glass each and, it turns out, did not see the absinthe fairy. Shoot, I’ll have to try some again some other time.

creux du van 1But about that training effect: the Jura is a great place to train. There are tons of trails through the forest, some of which are ski trails. Les Cernets is connected to hundreds of kilometers of ski trails, including a few long point-to-point trails like the 65 k Franco-Suisse loop, where you can do inn-to-inn touring. I can’t wait to come explore in the winter.

Jogging the farm roads in the morning through the fog felt mystical. And in the forest, clearings, bogs, and other areas are given fairy-tale names painted on old, peeling signs.

I was also thrilled to return to Creux du Van, a huge rock cliff formation which I had hiked with a friend in the spring. The closest thing I can compare it to is Cannon cliffs in New Hampshire – if you made Cannon much more even and bent it in a gently arching bowl around the valley. And plopped a picturesque farm and some happily grazing cows on top.

Creux du Van speaks to almost everyone, I think. My housemate told me that being up there, with hundreds of meters of empty space in front of you and birds playing on the wind, gives you power.

Sometimes that kind of phrase can sound woo-woo, but when you stand on Creux du Van, it’s not inaccurate.

rollerskiing 2But that’s not why we came to the Jura. A short drive into France is a rollerski loop at the Stade Florence Baverel in Arçon. So every day we would drive to France to ski.

(Inaugurated in 2009, the venue is named after the French gold medalist from the 2006 Olympics. You can also rollerski around the biathlon stadium in Le Seigne, a bit south in the Département Doubs, but we didn’t check it out. Prémanon, the training site for the French national team, is also only an hour away.)

The center has a nice biathlon range, a few kilometers of paved trails to train on. I would describe it as if John Morton had been given the assignment to design some kilometers of trail, but only given half the space that he’s usually given in North America. (After all, there’s less space for basically everything in Europe.) And, in this scenario he was also denied vital information about the length of classic rollerski shafts.

So it was with some trepidation that I first set out around the course. I’m not a particularly timid downhill skier, but the turns are, umm, very tight – and there’s a pretty decent height differential given the tiny postage stamp of land the center is crammed onto, so you come into them with momentum.

There were posters all over the main building for the French biathlon festivals hosted at the venue. I was trying to imagine mass start or even pursuit racing on such narrow trails with such sharp corners. I pictured carnage. I’m interested to try to find video of how it actually works.

That said, once I’d made a few trips around the loop, I wasn’t nervous and instead the twists and turns just made for super fun skiing. One corner was still a little dicey on classic skis, but on skate skis you can tear around with little fear of serious repercussions, at least if you don’t get tangled up with someone else.

It’s an excellent, and tough, loop for intervals. There’s not much recovery because the downhills are short and technical, so you’re always on your toes. And with limited places to easily pass, it’s good practice for rubbing elbows and making tactical choices in where to use your speed… for instance, before the beginning of the next downhill!

I was a bit sad to go back to Zürich and work, and away from the Jura and Doubs regions which seem to be a perfect playground for training in summer and winter.

berglauf racing.


On the advice of a friend, I signed up for an event outside of Zürich called the 5-Tage Berglauf Cup.

That means, in English, the 5-day mountain running cup. It was organized by a local ski club and featured five days of racing from small villages up to the top of hills in the Zürich Oberland, a rolling region of farms and rivers to the east of Zürich itself.

By the Monday that the series was about to start, I realized that things might not be as easy as I thought. Fabian, who had told me about the series in the first place and had done it many times, wasn’t able to come to any of the races after all.

And things are measured in meters here, not in feet as they are in the United States. So when a five-kilometer race has 500 meters of elevation gain, that doesn’t seem like so much… until I thought about what 500 meters really is. 1600 feet. A lot of topo lines all piled on top of each other on a map.

On top of that, I had just gotten back from a three-day training camp in the Jura mountains on the Swiss-French border. I don’t train full-time – I am doing a PhD and hold a part-time job! So I only do these “training camps” with a group of other skiers three times a year or so. When that happens, all of a sudden I’m training twice a day, usually with one session of intervals and one a long recovery workout. It really takes it out of me.

I stood on the start line on Monday, not knowing a single other person at the race, and I didn’t feel so great. And I definitely, definitely, only felt worse once we started climbing.

I felt like I had made a huge mistake, as a member of the Bluth family might say.

At the end of the day, as I was running down from the finish (no transportation is provided, so you have a handy compulsory cool-down unless you make friends with someone whose family drove a car up) I wondered whether to continue or bag the whole thing.

Those five kilometers had been hard, not fun. It was hot and humid and I could wring the sweat from my tank top after just over half an hour of racing. I tried to eat a granola bar but it wasn’t going down.

Tired from the weekend, I hadn’t been able to push myself, get my heart rate very high, or even have much of a killer instinct at the finish, where the next person in front of me was a 62-year-old man.

(Which, by the way, is awesome. You go, guy!)

My dreams of an age-group podium were definitely gone and I knew that I’d be in for a world of pain. I think that I often sign up for races because I believe that going hard will make me feel fit, but this did not make me feel fit!

There are plenty of Swiss people who never, ever run up a mountain. But for those who are interested in running up mountains, there’s lots of places to practice and, well, they are pretty expert.

As I neared the bottom of the hill and the bus stop where I’d start my hour-and-a-half ride back to my apartment via public transportation, I caught up with a middle-aged guy running backwards and sideways, clearly trying to stretch out his legs after the steep run downhill.

“We made it!” I said, holding up my hand for a high five.

I don’t think high-fiving is a universal behavior in Switzerland, and he let me hang.

But we started chatting, and when I stopped at the bus station, he asked where I was going. Did I need a ride? Zürich wasn’t too far out of his way.

My long bus and train ride home was cut to a pleasant 30-minute drive, and my new friend said that “This was the worst one.” By the time I got home, I had decided that I would race the rest of the week. After all, not showing up for a race you paid for seems like chickening out, and I may be slow, but I don’t want to be a chicken.

So the week rolled along. I felt a bit better on day two, and had even a good day on day three (good enough that I almost puked at the finish…). My friend Joseph had told me he would come race that day if we had a beer afterward, but then he backed out. So what? I was hooked on this mountain running thing!

(And Thursday I took as a day off, since only your best four results from the five-day series are scored in the cup.)

I ended up racing up 6,000 feet in elevation in four days, with an average grade of 9.5%. As the week went on, fewer 55-year-old women and teenagers beat me. If there’s one strength of cross-country skiers, it’s that we are tough and can take the hits in grueling race series.

I was proud to have finished the Cup, especially after sweating all day in my unventilated seventh-floor office in the midst of one of Europe’s worst heat waves ever. Not exactly good race prep.

For those of us with day jobs, maybe we enter races for glory. But in the end we finish them for the satisfaction of doing so. Dropping out is not an option, and crossing the finish line is an accomplishment something that most of our coworkers will never be able to boast about.


first big-girl paper!

In case you missed my facebook/twitter/researchgate/everything blitz, I finally published my first first-authored paper! It is in Oecologia, a good general ecology journal. I’m really happy and proud of myself, and a number of people have told me that this paper you’ll be happiest and most satisfied to publish, ever. I’m certainly enjoying the new addition to my CV.

Here’s a link to the paper, and here’s an abstract:

“Alpine plant communities are predicted to face range shifts and possibly extinctions with climate change. Fine-scale environmental variation such as nutrient availability or snowmelt timing may contribute to the ability of plant species to persist locally; however, variation in nutrient availability in alpine landscapes is largely unmeasured. On three mountains around Davos, Switzerland, we deployed Plant Root Simulator probes around 58 Salix herbacea plants along an elevational and microhabitat gradient to measure nutrient availability during the first 5 weeks of the summer growing season, and used in situ temperature loggers and observational data to determine date of spring snowmelt. We also visited the plants weekly to assess performance, as measured by stem number, fruiting, and herbivory damage. We found a wide snowmelt gradient which determined growing season length, as well as variations of an order of magnitude or more in the accumulation of 12 nutrients between different microhabitats. Higher nutrient availability had negative effects on most shrub performance metrics, for instance decreasing stem number and the proportion of stems producing fruits. High nutrient availability was associated with increased herbivory damage in early-melting microhabitats, but among late-emerging plants this pattern was reversed. We demonstrate that nutrient availability is highly variable in alpine settings, and that it strongly influences performance in an alpine dwarf shrub, sometimes modifying the response of shrubs to snowmelt timing. As the climate warms and human-induced nitrogen deposition continues in the Alps, these factors may contribute to patterns of local plants persistence.”