estimating and reporting the reliability of our cognitive tasks: Introducing the splithalf package

It is important that us researchers report the reliability of our cognitive tasks. In fact, I believe that the estimation and reporting of task reliability should be standard practice, however this tends to be the exception, rather than the rule. The reliability of our tasks influences the confidence with which we can give our results, and in turn, the power to replicate our results (LeBel & Paunonen, 2011). The continued development and improvement of our implicit psychological measures would benefit from regular reporting of task reliability, much as we currently do for self-report measures. i.e. it is common check the cronbach's alphas of the scales that we have used - as well as conduct large validation studies early on in the development of the measure.

I developed a script to allow myself to estimate the reliability of the dot-probe task that I had used in one of my studies. With the assistance of Phil Enock and my supervisor Anne-Wil Kruijt I was able to turn this script into a workable function, DPsplithalf(), which I hope will be fairly straightforward for others to use. This function is contained within the splithalf package, which I am currently asking others to test before I upload it to CRAN. The package also offers several other functions to calculate the split half reliability of tasks. 

The functions include the option to estimate split-half reliability with a monte carlo simulation process repeating a large number (5000 is recommended here) of calculations in order to yield a more stable estimate than other approaches, such as the odd/even numbered trial split. I will write in more detail about this at a later time, but for now it is worth noting that the package offers this, and I sugest that it will be the best to use. 

I wanted to use this post to introduce the package and provide an example of how it may be used. Therefore, I have copied parts of the package vignette, which provides two examples that may be helpful to explain how to use the DPsplithalf function. The first is straightforward, using a simulated data set in which there are no missing data and the variables are named as required by the function. The second is slightly more complex and will hopefully help with compatibility issues. Last I have included a figure to give an idea of the expected run time of the function with increasing numbers of conditions and larger sample sizes. 

If any readers have a dot-probe data set laying around, I would really appreciate it if you take the time to test the package to test its functionality and usability. Any suggestions or input are very welcome, as I want the package to be as user friendly as possible. In the future I also aim to develop a shiny application in order that non R users are able to run these functions online.

you can install the package from github with the following

install.packages("devtools")

library("devtools")

devtools::install_github("sdparsons/splithalf")

Example 1:

First, lets look at the data. This is simulated data contained within the package. It has been designed to be as straightforward as possible for the purposes of this example. You will note that the data contains no missing data, the variables do not need to be renamed, and the congruency is appropriately named (example 2 will cover when this is not the case).

str(DPdata)

## 'data.frame':    3840 obs. of  6 variables:
##  $ subject   : int  1 1 1 1 1 1 1 1 1 1 ...
##  $ blockcode : Factor w/ 2 levels "block1","block2": 1 1 1 1 1 1 1 1 1 1 ...
##  $ trialnum  : int  1 2 3 4 5 6 7 8 9 10 ...
##  $ congruency: Factor w/ 2 levels "Congruent","Incongruent": 1 2 1 2 1 2 1 2 1 2 ...
##  $ latency   : num  23.6 25 24.1 24.7 24.6 ...
##  $ correct   : num  1 1 1 1 1 1 1 1 1 1 ...

head(DPdata)

##   subject blockcode trialnum  congruency  latency correct
## 1       1    block1        1   Congruent 23.56481       1
## 2       1    block1        2 Incongruent 24.98990       1
## 3       1    block1        3   Congruent 24.08452       1
## 4       1    block1        4 Incongruent 24.70067       1
## 5       1    block1        5   Congruent 24.58380       1
## 6       1    block1        6 Incongruent 24.53391       1

sum(is.na(DPdata))

## [1] 0

In this example, we want to calculate the split-half reliability for block 1 and block 2. We want to use the “random” method to process 5000 iterations of randomly generated splits, and return estimates for both blocks. The function will take a little time to run (the time taken while creating this vignette was 23s).

library(splithalf)

## 
## Attaching package: 'splithalf'

## The following object is masked _by_ '.GlobalEnv':
## 
##     DPdata

DPsplithalf(DPdata, conditionlist = c("block1","block2"), halftype = "random", no.iterations = 5000)

## [1] "condition block1 complete"
## [1] "condition block2 complete"
## [1] "Calculating split half estimates"
## [1] "Split half estimates for 5000 random splits"

##   condition  N  splithalf spearmanbrown
## 1    block1 20 -0.9146333     -23.96023
## 2    block2 20 -0.9442398     -38.08540

What you see first is the console output. You will be informed as each condition has been processed and at which point the estimates are being calculated. This is purely so the user can keep track of the progress of the function.

Next, the function output. The data frame returns each condition separately. The N column indicates the number of participants’ data that has been processed, which is important to check as it is an early indication of missing data. The splithalf column is the raw split half estimate, in this case the average of 5000 random splits. The spearmanbrown column returns the spearman-brown corrected reliability estimate. You will quickly note that this estimate is flawed in this example; this is due to the high negative splithalf estimate. This is expected with this randomly simulated data.

The purpose of this example is to provide a more practical example of how to use the DPsplithalf() function with real data. Therefore, I have included snippets of script that will enable users to adapt their data where necessary. We will take the example step-by-step to cover many of the issues that could arise. First, note that the data frame does not actually contain any missing data, however, there is no data for subject 15 in block 2.

sum(is.na(DPdata_missing))

## [1] 0

head(subset(DPdata_missing, DPdata_missing$accuracy == 0))

##      subject  block trialnumber    trialtype responsetime accuracy
## 2785      15 block2           1   Congruent1     24.60937        0
## 2786      15 block2           2 Incongruent2     25.35842        0
## 2787      15 block2           3   Congruent1     25.84493        0
## 2788      15 block2           4 Incongruent2     23.74887        0
## 2789      15 block2           5   Congruent1     24.90278        0
## 2790      15 block2           6 Incongruent2     24.66794        0

From the head() data above we can also see that the variable names do not conform to what is needed to run the function. First, we will rename the Congruent and Incongruent trials. The function requires that a “congruency” variable specifies whether the trial is “Congruent” of “Incongruent” (case sensitive). One such method is to run the following. The code searches in the trialtype variable for strings that contain “Incongruent” and returns “Incongruent” if present, and “Congruent” if not. This simple line of script can be adapted to meet most namings. Note, ensure that the name searched for is not present in other conditions, otherwise these will be included too.

DPdata_missing$congruency <- as.factor(ifelse(grepl("Incongruent", DPdata_missing$trialtype), "Incongruent", "Congruent"))
str(DPdata_missing$congruency)

##  Factor w/ 2 levels "Congruent","Incongruent": 1 2 1 2 1 2 1 2 1 2 ...

Next, at the moment we have 4 different conditions in this task, however we want the same 2. This may happen for example, when you have two stimuli lists that are counterbalanced between participants. The next line of code is a slight variation on the above to return the blockname without the additional “a”. Also note that there are a large number of methods to do this within R. This example is meant to be informative to those with minimal R experience, as well as being applicable across a number of contexts. In short, the code tests whether the names of the block conform to either condition, and return that condition.

DPdata_missing$block <- as.factor(ifelse(DPdata_missing$block  == "block1" | 
                                         DPdata_missing$block  == "block1a",
                                         "block1",
                                  ifelse(DPdata_missing$block  == "block2" |
                                         DPdata_missing$block  == "block2a",
                                        "block2","")))

str(DPdata_missing$block)

##  Factor w/ 2 levels "block1","block2": 1 1 1 1 1 1 1 1 1 1 ...

Now that our congruency and block names are correct, all we need to do is ensure that the variable names are specified within the function. Reminder that the variable names are defaulted to; var.RT = “latency” var.condition = “blockcode” var.participant = “subject” var.correct = “correct” var.trialnum = ‘trialnum’

We will therefore specify; condition, trialnum, RT, correct

example2 <- DPsplithalf(DPdata_missing, conditionlist = c("block1","block2"),        halftype = "random", no.iterations = 5000, var.condition = "block",            var.trialnum = "trialnumber", var.RT = "responsetime", var.correct = "accuracy" )

## [1] "condition block1 complete"
## [1] "condition block2 complete"
## [1] "Calculating split half estimates"
## [1] "the following are participants/conditions with missing data"
##        condition participant
## 170001    block2          15
## [1] "note: these iterations will be removed from the split half\n          reliability calculations, in that condition"

## Warning in DPsplithalf(DPdata_missing, conditionlist = c("block1", "block2"), : Bias indices missing:
##           at least one participant has missing data from at one condition
##           These cases are removed from calculating reliability estimates
##           $omitted contains the missing cases

## [1] "Split half estimates for 5000 random splits"

example2$Estimates

##   condition  N  splithalf spearmanbrown
## 1    block1 20 -0.9256936     -27.82766
## 2    block2 19 -0.9136950     -23.87316

head(example2$omitted)

##        condition participant iteration bias1 bias2
## 170001    block2          15         1   NaN   NaN
## 170002    block2          15         2   NaN   NaN
## 170003    block2          15         3   NaN   NaN
## 170004    block2          15         4   NaN   NaN
## 170005    block2          15         5   NaN   NaN
## 170006    block2          15         6   NaN   NaN

You will notice that we have much more in our output than in example 1. You will see a warning message pointing out that there is missing data. In the console you will see a short message “the following are participants/conditions with missing data”, which is followed by a data frame showing which participants have have missing data from which conditions. The function will still calculate split half reliability. The function output two data frames (I also recommend loading the function into an object to ease this step). The Estimates data frame includes the split-half reliability estimates as described in example 1. This will also highlight which conditions have missing participants in the N column. The omitted data frame is a complete data frame of all omitted iterations. In this example we have one participant in one condition missing and therefore the omitted data frame contains all 5000 iterations of missing data.

function run time

An important question is; how long does this function take to run? The quicker the functions run, the more usable. Given that ideally we will run 5000 simulations as standard, we want the script to run quickly. No-body is keen to run a 2 hour script on 20 participants’ data to get only a single value as an output. Much of the recent development of this package has actually been reducing the run time of the functions. This has been very successful, going from some 45 minutes for roughly 40 participants in 2 conditions, to about 45 seconds. To illustrate the likely run-time of the script I have created the below figure. The first section of the script was used to simulate the data (I then replicated the output in the next section, in order to avoid the need to run the simulation to knit this document). Next is the data table, and the graph showing the run time in seconds across 1,2,3,4,5, and 10 conditions, with sample sizes of 10,20,30,40,50,100,200, and 500. This should give a rough idea of the likely run time of the DPsplithalf function for other data sets (note that the simulated data does not have missing values).

conditionnumber <- c(1,2,3,4,5,10)
samplesize <- c(10,20,30,40,50,100,200,500)
temp <- NA
times <- NULL

for(i in conditionnumber)
{
  for(j in samplesize)
  {
    # ensure the correct number of conditions
    if(i == 1) {
      conlist <- c("block1")
    } else if(i == 2) {
      conlist <- c("block1","block2")
    } else if(i == 3) {
      conlist <- c("block1","block2","block3")
    } else if(i == 4) {
      conlist <- c("block1","block2","block3","block4")
    } else if(i == 5) {
      conlist <- c("block1","block2","block3","block4","block5")
    } else {
      conlist <- c("block1","block2","block3","block4","block5",
                   "block6","block7","block8","block9","block10")
    }

  # generate the simulated data  
   temp <-  data.frame(subject = rep(1:j, each = (96*i)),
                       blockcode = rep(conlist, each = 96, length.out = i*j*96),
                       trialnum = rep(1:96, length.out = i*j*96),
                       congruency = rep(c("Congruent","Incongruent"), length.out = i*j*96),
                       latency = rep(rnorm(100,100,25), length.out = i*j*96),
                       correct = rep(1, length.out = i*j*96) )
  
    # run DPsplithalf
    time[j] <- system.time(DPsplithalf(temp, conditionlist = conlist, 
                                       halftype = "random", no.iterations = 5000))
   
   # save the data
   times <- rbind(times, c(i,j,time[j]))
   
   # keep track of the runs
   print(paste("completed",i,"conditions",j,"participants", sep = " "))
  }
}

times <- read.csv("timestable.csv")
summary(lm(data=times, System.time ~ Conditions + Sample.size))

## 
## Call:
## lm(formula = System.time ~ Conditions + Sample.size, data = times)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -552.79  -55.82    7.85   75.60 1138.02 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) -234.5706    62.8531  -3.732 0.000531 ***
## Conditions    56.7447    11.3265   5.010 8.90e-06 ***
## Sample.size    1.8793     0.2124   8.847 2.09e-11 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 228.4 on 45 degrees of freedom
## Multiple R-squared:  0.6967, Adjusted R-squared:  0.6832 
## F-statistic: 51.68 on 2 and 45 DF,  p-value: 2.2e-12

library(ggplot2)
times$Conditions <- as.factor(times$Conditions)
ggplot2::ggplot(times, aes(x = Sample.size, y = System.time, linetype = Conditions)) +
  geom_line(size = 1)

The figure highlights the run time for 48 simulations of the DPsplithalf function. As you can see, there is a linear increase in the run time as a function of the number of conditions and the sample size. It is worth noting that most research utilizing cognitive tasks such as the Dot-Probe recruit smaller sample sizes, and so for most data sets, the estimation of task reliability should take no more than a few minutes. A linear model also yields the following equation to estimate the amount of time needed for the function to run: -234 + 56.7(Conditions) + 1.9(Sample size). Please note that the operating system and the actual data will have a large effect on the time taken to run the functions.

References

Enock, P. M., Hofmann, S. G., & McNally, R. J. (2014). Attention Bias Modification Training Via Smartphone to Reduce Social Anxiety: A Randomized, Controlled Multi-Session Experiment. Cognitive Therapy and Research, 38(2), 200–216. http://doi.org/10.1007/s10608-014-9606-z*

LeBel, E. P., & Paunonen, S. V. (2011). Sexy But Often Unreliable: The Impact of Unreliability on the Replicability of Experimental Findings With Implicit Measures. Personality and Social Psychology Bulletin, 37(4), 570–583. http://doi.org/10.1177/0146167211400619*

Writing a research grant application as a PhD

Last week, we submitted a grant application to a UK research council. It was an uphill struggle from the beginning, and the last week was a flurry of activity between my supervisor and I to make the revisions that were much needed to polish the application. Now, I have to wait with my fingers crossed that our efforts will not succumb to the ~90% likelihood that we will not receive funding.

As with any long term project, I have had plenty of time to reflect on the experience. So, by way of release, here are some things that I have learned/experienced as a PhD student applying for a research grant. I plan to write a follow up post on the practicalities of writing the grant, resources that were useful, and depending on the success/failure of the application; what to do and what not to do.

Co-applicant or named researcher?

First, I would like to begin with what was, for me, the worst part of the experience. As background, I started a much more negative version of this post at the start of the year entitled "Co-applicant or named researcher - someone else decides". I am unsure if other universities have a similar policy, but what happens here is that the research committee must approve each researcher to have a JeS account of the correct type to submit research grant applications. Without this approval/support from the department, you simply cannot be a co-applicant. In short, I was denied the request to be a co-applicant on the grant application that I had written with my supervisor (note; this was at the time that we had decided on a rough plan together, and I had then worked solidly for about two months in creating the first draft of the complete application). The reason? I did not get one at first. I did receive an automated message from the JeS system to tell me that my request had been denied, but that was about it. After some prompting I was informed in essence that I was too junior (which again is a nice sting for someone hoping to stay in academia). My hope was that the fact that I had led the application across all stages might hold sway over my publication count, but sadly not. All in all, this part of the experience really stung.

The result was that by necessity, I needed to remove my name as co-applicant from each document and continue working on the application. Although I would not be receiving full credit - fully acknowledging the unlikeliness of receiving the funding in any case - I needed to continue to take the lead and work on the proposal. If I had allowed myself the opportunity to slow down for more than a brief period of venting frustrations then the application would not have gotten completed.

So, I learned several lessons pretty quickly;

• You may not get full credit for the work
• But, you must keep going regardless
• Be more experienced, I guess...

There will be setbacks and delays

For me, a setback was changing our targeted research council. Looking back it was certainly the right call, but at the time I had to do quite a bit of rethinking and restructuring - including the loss of two pages from the Case for Support (essentially the background and research plan). For others it might be illness, or changes of deadlines, receiving feedback, or co-applicants disagreements over the application, to name only a few. 

Given the time-course of a grant application, one should expect;

• There is likely to be at least one major delay, as there could be in any project. Plan for this working from the deadline
• You will likely lose a week or so before the end of the final deadline, as the application will have to be submitted to your research institution / department. To illustrate, our deadline was the 31st and we had to submit on the 24th. 

You will lose months of your PhD

The process will differ for everybody, but, writing a grant application will take months of dedicated work. I suspect that PhDs who acquired competitive funding for their doctoral research will have a stronger background in funding applications than I, but I doubt this would save them more than a few weeks. There are so many sections to the complete application, all of which take much longer than expected to polish. A 2,000 character summary of the project objectives sounds like easy going, but it is also essential that it is accurate and reflected in every other document, as well as being exciting, timely, and well written. 

• It will take months, and much longer than expected
• Seriously, take your high estimate for time commitment. Now, make that your 'best case scenario' in which you and any other co-applicants (or in my case - named researcher) work on the application solidly for, and aren't hindered by any delays.

Take help wherever you can

I was lucky enough to have a postdoc and another PhD in my lab read through and comment on the application. My wife also acted as a non-expert to offer some commentary on the documents. Each little bit of feedback is useful in some way. The comments that I found particularly helpful related to the flow of the application. Things that were clear to me were not clear in the writing. This is common. I feel, when you have spent so much time on a document that it blends into a single entity, rather than what should be a concise and flowing train of thought.

On a related note, the research councils have a ton of resources to use and the JeS has its own summaries of what is entailed in each section. What is extremely useful are other's applications, although whether they will be shared is another issue. Use these resources, they are vital in understanding what is expected in each section.

• Use any resource that you can

Writing a research grant application as a PhD

It's hard work. It is long, frustrating, and sometimes rage-inducingly tedious. In the back of your mind there will always be that niggling doubt, not just in yourself; is it the right project? am I writing this section correctly? is this enough information? how the hell can I fit this all into 6 pages? Worst, there is the understanding that it is months of work for only a small chance at gaining the funding. 

But, it is an essential skill to develop as an academic. Perhaps not as early as in your third year of a PhD, but I've always believed in developing skills early - ideally before they are actually needed. So, in way of a summary; It is draining work, with a low success rate. However, if it gets me a postdoc position in the lab I want to stay in, on a project of my own devising, then it is worth it. If not, then it is good experience (repeat to self several hundred times and you might believe it). I've grown fond of the saying (in my own words) "academia, and grant writing in particular is like a pie eating contest in which the prize is more pie". Please give me some more pie. 

Take home messages from the 2017 MQ Science Meeting

Amnesty International, London held the third annual science meeting run by the mental health charity MQ. It saw many mental health researchers and professionals meet and discuss state of the art and groundbreaking research across many interdisciplinary domains. MQ's tag line is "Transforming mental health through research" and what I loved about this event in particular was the focus on the research that can and has been undertaken in order to tackle the growing challenge of mental health. MQ highlights the need for interdisciplinary and collaborative approaches, and science meetings or conferences such as this enable the discussions to happen in order to meet this goal. Through research is particularly why I was drawn to MQ.

Two days of talks, an extended poster session, networking opportunities, and an insightful panel discussion on "What good is a diagnosis?" rounded up the two-day line up. I had the opportunity to get feedback on my poster "A cognitive model of psychological resilience: current research and future directions" which thankfully has given me quite a lot to ponder, as well as some ideas for cognitive task designs. For this post I would like to highlight several take home messages that I especially took to heart during the meeting.

Targeting the mechanisms
Michelle Moulds gave a talk and led the discussion on repetitive negative thinking as a transdiagnostic process in mental illness. I was particularly glad as this aligned well with my current thinking (and background for an upcoming grant application). Repetitive negative thinking is common across disorders and is predictive of comorbidity. Therefore, the idea that interventions target repetitive negative thinking is timely and useful. In relation to my field (discussed more below), this suggests that cognitive interventions should target this underlying process, rather than attempting to address symptoms that lie further downstream. The discussion also led to the mention of some work that suggests that certain patterns of thought are promotive and some destructive, perhaps based on the content (e.g. positive versus negative content). Again, this is a useful distinction that resonates with my work; cognitive processes can be adaptive or maladaptive, depending on the context and content.

We need longitudinal research
I have made this point before, so I don't want to spend too much time on it here. Suffice to say that more longitudinal research is needed and in particular with a developmental focus, in order to understand mental illness. Although the cross-sectional and correlational research that has been undertaken is important, it needs to be situated in a developmental context. There are ethical issues with many potential experimental research designs (e.g. maltreating one group of children in order to compare to a protected group, or similar), however, this is where the interdisciplinary nature of MQ comes into play. Animal models, genetic, and epigenetic approaches are highly valuable, as are natural experiments such as the English and Romanian Adoptee study (link here) discussed by Professor Sonuga-Barke. My take home message from the varied and approaches discussed during the meeting is that there are are many ways in which vastly distinct disciplines can be integrated and provide complementary understanding of mental health. A caveat is that more longitudinal research is still needed in order to understand the development of mental ill-health, in order to develop preventative and resilience based approaches.

We need collaborative and integrative research
Touched on slightly above is that we need to collaborate and conduct multifaceted research in an interdisciplinary fashion. Animals and basic science help us understand the mechanisms underpinning mental health, while therapeutic intervention studies allow us to assess treatment options, and (epi)genetic studies give an additional glimpse into the underpinnings of mental health.
 As it was brilliantly stated during the meeting (and ad-libbed here), it starts with meetings and conferences such as this, which facilitate the discussion between disciplines to share our knowledge and expertise. Ultimately, we are all interested in the same thing, transforming mental health through research. From this community will rise an exceptional body of research. Again, I love the tag line through research and think that it is what we need more of (as well as the funding and support to do so).

More research from a cognitive-experimental information-processing approach is needed
Or, perhaps it might be better to say that this approach should be better represented and communicated. I was excited to meet several researchers that are doing some great work in this field, including Ernst Koster and Colette Hirsch. My supervisor, Elaine Fox is also well known in this field. The cognitive-experimental approach borders with cognitive neuroscience and may provide a linkage between biological and neurological measures, and mental health related behaviour. We typically use computerised tasks to investigate differential responses to emotionally salient stimuli using a range of outcomes, including; behavioural (response times and accuracy rates), psychophysiological (e.g. eye tracking, heart rate), and neural (EEG, fMRI and so on), amongst others. These approaches have provided strong evidence for the causal contribution of automatic attentional biases favouring threatening stimuli towards anxiety symptoms (). In addition, this approach offers a number of paradigms designed to train particular biases and executive control processes. In one discussion during the MQ meeting it was commented that working memory training may help individuals with ADHD-related working memory impairments, but unfortunately there aren't such paradigms available. In fact, there are several lines of research tackling just that, and with some success. For example, improvements in working memory capacity have been shown to improve trait anxiety (Sari, Koster, Pourtois, & Derakshan, 2016). There are a growing number of studies from Derakshan and Koster which highlight the benefits of executive control training on mental health symptomology. Building on this existing research with novel paradigms and integrated methodologies (e.g. incorporating EEG and eye-tracking measures) has the potential to extend our understanding of the cognitive basis of mental health and resilience to mental illness. In addition, cognitive interventions may prove to be a useful clinical aid or preventative tool transforming mental health.

Final thoughts
The MQ science meeting was successful in bringing together experts from around the globe, from a multitude of disciplines, with the intention to foster collaborative and interdisciplinary mental health research, It was amazing to see such a range of science brought to the table to tackle the growing issue of mental health, both on a societal and individual level. I am looking forward to the next meeting, but in the mean time, I look forward to the funding call announcements and submitting my own proposals to fulfil the gap in current research investigating the cognitive-affective aspect of repetitive negative thinking and mental health in adolescence, and the development of resilience in this particularly vulnerable period.

A rebuttal: What does it mean to be a successful academic? And how to not suck at achieving it

I came across this blog post on twitter a few days ago titled "What Does It Mean to Be a Successful Academic? And How Not to Suck at Achieving It". Sounds great, and as a junior academic crawling to the end of my PhD looking (and hoping) for that elusive postdoc position, it felt like a timely read. The post has been shared a bit and does have some positive messages that I agree with. I confess however, the more I read the more the post failed to resonate with me, so here goes with trying to reconcile that. Note: I am trying to be balanced, but these issues are hitting home.

The first message is to enjoy the work, which I wholeheartedly agree with. The example of slaving away 80-90 hours a week to get that perfect paper or self-nominated award, etc, sounds extremely unappealing. I also like the mindset that what defines success is a personal thing and driving to be "the best" in a field isn't necessarily healthy (nor conducive to good science, which is the marker of success that I would like to drive towards).

The question, is it worth it? is raised and it is an important side of the coin to balance. While not mentioned explicitly, the theme of life first and academic achievement second agrees with my mindset. One Postdoc in my research lab is honest about this and conveys the refreshing attitude that while she loves her job and research, if and when necessary she will leave academia she will do so and be happy about it. Life first, academia second. I think that this is the way it should be.

Where I think that the message of the post was derailed for me is the mention of one of the benefits of tenure as not having to worry about having the most publications or highest impact factor etc. This is perhaps where the idea of personal achievement comes in. My idea of achievement at this stage is to do enough to stay in academia without an employment gap once my studentship funding runs dry. This introduces the thing that the post fails to capture, competition. At the early researcher stage, you are fighting for these positions tooth and nail. That means if you can get an extra publication, publish in higher impact journals (which I realise is nearly entirely BS, but is still sadly important), acquire research funding, and so on, then you pretty much have to play that game just to get the position. As a junior academic you are also susceptible to Departmental/University procedures which can leave you at a disadvantage (or at least not receiving full credit for grant applications due to being too junior, as I found to my own dismay recently). When competition and a lack of positions drives what is the most basic aspect to being an academic, having an academic position, the whole notion of the post fails. We junior academics cannot not worry about it and be content with a reasonable amount of output but not enough to stand out from the crowd. This left me with a take home message (completely unintended by the author I am sure), that having tenure means that you can stop worrying and enjoy life. But, what about everybody else?

Recently we had a discussion with our director of graduate studies who gave the average completion time of PhDs in our department at 44 months. For those like me lucky enough to have funding, it will run out after 3 years. So in terms of being successful and not sucking at achieving it, the best case scenario is that PhD candidates are finding extra funding to finish their PhDs. Realistically, they are likely being strategic and submitting their theses at a later date in order to get out those few extra papers and to get the extra time a few years later when applying for fellowships. Worst case scenario, it means that too many PhD candidates are paying their way through the latter stages in order to be competitive in this under-funded environment.

I'll try to bring this full circle and my apologies for the rant. Again, given present worries about employment, the post hit home. The importance of personal context, desired quality of life, and definition of success are stressed as key factors to balance in this equation. I agree that these are factors to balance in order to have a fulfilling life. However, when your definition of success is actually being able to stay in academia (or get into academia in the first place) then the milestone moves too far away from that set in that particular post. It all comes down to personal circumstance, goals, and life balance. In my case, without the benefit of tenure, the achievements that may be seen as arbitrary may also be the very ones that enable me to do what I love and stay in research.