How to deal with feedback / comments on your writing without smashing your computer up

The Problem

My job involves a lot of writing and one of the most frustrating tasks I encounter is making changes to pieces of writing based upon feedback / comments from other people (much as I appreciate their advice of course!). I think there are a few reasons for this. Firstly, we need a little introduction / reminder to the concept of the ‘next action’:

As the ‘Getting Things Done ®’ author David Allen notes repeatedly, people’s to do lists are often vague and don’t represent real actions i.e. you read the ‘to do’ item, and you still don’t know what … to do. For example, suppose you have a project to retile the bathroom, and on your to do list is just ‘Bathroom’. You read this and, since it’s not an actual ‘action’, your mind naturally feels averse to it and you procrastinate – repeat daily until you’re left wondering how you’ve done nothing on the bathroom in a month. The ‘next action’ might actually have been to call a builder to get a quote, or perhaps to research online some DIY stores which have the kinds of tiles you need. If you actually had ‘Errand: go to DIY store’ on your to do list instead of ‘Bathroom’ you would have been much more likely to actually do it.

So, even when people write to do lists for themselves, often they aren’t very well composed and feel like a vague list of things you’d like to achieve with no concrete actions for how to achieve them. Now, when we send our work off to someone for comments, what we are effectively doing is asking them to write a to do list to get the write up to the acceptable standard required. If we often can’t write good to do lists for ourselves, what do we expect when we let someone else write one for us (when they aren’t going to be the ones who actually have to do the things)? In my experience even the most thoughtful reviewer of my work will only have given concrete action steps such as ‘This figure needs a caption’ in a small proportion of their comments, and generally only for the simplest ones. More often you will get things like ‘This section feels out of place here’, or ‘I think this could flow better’ and you’re left thinking ‘OK, but how do I actually fix that?’ or perhaps just inarticulate frustration. Unfortunately, I am exactly the same when I provide reviews for other people – coming up with concrete actions is time consuming and difficult – it’s the writer’s job, and this isn’t going to change.

Making this situation even worse is that a lot of comments are not confined to a small portion of the write up – a piece of writing is a large, complex and interwoven thing with different sections dependent upon each other, and so often comments require either a grand restructuring or at least have ramifications for multiple sections, which can in itself be overwhelming.

A final set of issues is that commenting functions tend to be set up to be natural and easy for the commenter, but little thought seems to have been given to the person who has to enact them i.e. they lack the features of to do list programs (i.e. being able to tag, sort and organise them). There are often also an overwhelming number of comments, and furthermore, in their natural format, you cannot see them on a single screen (they’re spread out over many pages), and so your brain struggles to process them as a whole, and can go into panic mode about how much work there is to do and what the nature / structure of that work is.

The Solution

OK, so, how do we fix this? The main way I am going to propose draws quite heavily on the ‘next action’ concept in Getting Things Done®, and is not far from how you would sort out an overflowing messy to do list.

The first thing is to go through all the comments and do any that, by lucky happenstance, happen to be both written in such a way that when you read it, your brain immediately knows what to do (like ‘add a caption to this graph’) and will take less than a few minutes. Just do these now, ignoring all the other comments. If you reaaally don’t wanna, fine, just move to the next step, but the reason we do this is because the sheer number of comments can often be overwhelming in and of itself, and so getting all these small ones out of the way not only provides small wins for your motivation, but gives you less comments to sort in the next step.

The next thing to do is to extract all the comments which are either structural (i.e. refer to structural changes needed in the write up), or relate to more than one section. We have to do this because these comments will tend to interact, contradict or be redundant with each other, so they need to be processed and integrated as a whole. I suggest you take these to whatever tool you like to use for brainstorming, and effectively use these as ‘notes’ and completely rewrite your own set of structural ‘actions’ based upon these. If you find comments during this that relate to a different portion of the write up, but only to that portion (e.g. the commenter thought of a point later on but couldn’t be bothered to go to the right place to make it), delete it from where it is, and move it to the right place (i.e. remake it as a new comment), but don’t include it in these extracted structural comments.

What you should now have remaining are small and medium sized tasks which were not phrased very helpfully by the commenter, such that an obvious action did not occur to you. What you now need to do is ‘reply’ to each of these comments within the document (you’re not actually replying to the commenter, don’t worry, this is just for your purposes) with the ‘next action’ required to actually make the change that the commenter wants to happen (or at least, progress towards that goal – it may require multiple action steps with the first being e.g. ‘Read paper Y’). If you really want to know a lot more depth about next actions I have to recommend that you read the Getting Things Done® book (I recommend this wholeheartedly either way), but the test for whether you’ve devised a good next action, is that when you read it, your brain will immediately feel ready to jump into action (it knows what to DO), and you will feel no ‘resistance’ (actually you still might feel some if it’s an undesirable task e.g. speaking to an unpleasant boss, but you won’t get that same mental block as you do when you have a poorly defined action). Remember that ’email to ask further clarification from X on this’ is a perfectly reasonable action if you really don’t know what they want. Store these up and email them all at once.

Finally, depending on how many of these actions you have, you may want to extract them to an external to do list, where you can sort them by length of time, difficulty, energy required etc. If you do this, I recommend leaving the original comments in place as you might forget what section certain actions relate to otherwise.

OK, that’s all. Enjoy your peaceful mind.

Read More

Incurably Sceptical: Can Super Recognisers Detect Happy or Fearful emotions by Sniffing Underarm Sweat?

Welcome to the second in my ‘Incurably Sceptical’ series (see first post here). In this section I pick a paper from the cognitive psychology literature that appears interesting based on abstract alone. We then pick apart the author’s aims, methodology, analysis and interpretation, having perhaps just a little fun at their expense but hopefully also learning a few useful things about scientific method along the way. This time we will be looking at a paper entitled ‘Testing for Individual Differences in the Identification of Chemosignals for Fear and Happy: Phenotypic Super-Detectors, Detectors and Non-Detectors.’ [Link] Broadly, the aim of this paper was to examine the extent to which people can detect a person’s mood (fearful or happy) by smelling their under-arm sweat (stay tuned for more on the protocol for extraction).

Super Detector (/ recogniser) research is a popular trend at the moment, both academically and in the media. The idea is that there are some individuals in the population who, for whatever reason, have a ‘super’ ability for e.g. detecting flavours in wine / coffee, recognising faces, detecting minute changes in pitch / tone etc (in general, having an extremely heightened ability to detect similarities between two stimuli or patterns based on one or other of the senses). There have been many articles lately about super detectors being used by the police and private companies for all sorts of wonderful things (see: Are you a Super Recogniser? and ‘The Super-Recognisers of Scotland Yard’). Even the higher-quality reporting on the subject has raised my scepto-sense before – see for example this paper where a group of people with an average 93% accuracy for facial recognition (vs 80% in the general population) apparently deserve the title ‘Super Recognisers’. ‘Slightly Better Recogniser’ might be more appropriate. This is inevitable of course. When being ‘on trend’ increases your likelihood of getting published, the application of sensational category labels like ‘Super Detector’ to small group differences is to be expected. So, I would certainly describe myself as in a sceptical frame of mind when I first read the title of this paper. The abstract didn’t improve things with its mention of ‘implications for the further study of genetic differences’ despite there clearly being no actual genetic analyses in the study. Further, ‘dual processing’, another trendy term, was thrown in despite a lack of clear relevance. In short, this paper appeared to me, based on the abstract, to be tenuously ticking all the boxes publishers like to see, and when a paper’s doing that, I tend to worry that low quality work is being masked underneath. However, the abstract also said that “mood odors had been collected from 14 male donors during a mood induction task” and that 41 females had been asked to identify the mood odor chemosignals … so obviously I read on.


So, yes, onto this extraction method. Normally, I would paraphrase a method, but I enjoyed the tone of the write up of this one so much that below I reproduce (nearly) the whole section on extraction:

In this study the mood odors were collected from 14 healthy male undergraduate nonsmokers. For a 7-day period prNoseior to the sample collection, the donors only used the provided odor-free deodorant and cleansing products. The donors were instructed to shower (using the soap provided) the morning of sample collection approximately 6 hours prior to sample collection. They were also given a list of prohibited “spicy” and other odorous foods and did not eat them during the 24 hours prior to the collection.

Axillary samples were collected during two video mood inductions, one day apart. The fear mood and happy mood induction videos were 12-minute standardized videos … The videos were shown twice to the subjects for a 24-minute induction. The videos have multiple facial displays for fear (or happy). There is no narrative theme. This reduces the likelihood that repetition of the 12 minute video would decrease the impact of the video.

Samples were collected onto cleaned Kerlix8 brand sterile gauze. Prior to mood induction, donors were given 2 pairs of gauze strips (each strip 3cm x 8cm) in separate plastic enclosed bags labeled “right” and “left” arm. They placed one pair in each left/right axilla. At 12 minutes into the mood induction, the film was paused and donors removed one pair (1 left and 1 right) of axillary pads. Donors placed each pad into its labeled plastic zipper bags. All air was forced from the bag prior to sealing. The second pair of pads was collected in the same manner after 24 minutes. All samples were placed in a minus 80C° freezer within 2 hours of collection.

So, yes, an unusually invasive and controlling set of requirements for these healthy male undergraduates. They don’t report the incentive for them to take part in the study, or if they were paid more or less than the people who had to smell their sweat – tough call that one. In terms of the smelling protocol, “Participants (detectors) were tested individually in dedicated testing rooms approximately 8’x8′” (I’m not sure why they included the room size here, but perhaps if you know a lot about smelling, this is quite important for understanding the … diffusion dynamics or something). Then:

On each trial the experimenter placed five identical sample jars from one set of donors on a plastic tray on the table, shuffled them, and presented the tray of jars to the detector. S/he was instructed to sniff the jars as many times as necessary and in any order. The detector identified the odors by setting each jar on its label [e.g. fear, happy, control] on a place-mat.

I imagine this scene a bit like a gross version of the ball and cup trick (‘Keep your eyes on the fear-sweat jar’). The offer of unlimited sniffs was very generous though. Anyway … despite its amusing elements, on reading the methodology I was struck by how well controlled it was e.g. “To avoid position effects, half of the detectors had fear labels on the left side of the place mat and half of the detectors had them on the right side of the place mat.” There were lots of neat little controls built into the study like this to ensure the results weren’t biased and overall I was impressed by the attention to detail.

So, they extracted sweat during fear-inducing or happy-inducing videos, then got people to sniff fear, happy, and control (no video) sweat to see if they could correctly label them. Simple enough. Now, what did they find? “The first analysis (rtt) showed that the population was phenotypically heterogeneous, not homogeneous, in identification accuracy.” This sentence annoyed me, I must admit. The use of the word phenotypically implies an important distinction is being made between the participants’ genotype (their set of genes) and their phenotype (their body). But there’s no genetic testing in this paper, so the distinction is pointless – the word can just be deleted entirely without affecting the meaning of the sentence. And heterogeneous? All that means is that every single individual in the sample weren’t all equally good at smelling – the addition of these words seem just to serve to add a ‘sciencey’ sounding feel to the paper. If you’re wondering why I’ve gone on about this for so long, well yes, it’s a pet peeve of mine. Flowery, jargon-rich scientific

Percentage of Super Detectors, Detectors and Non-Detectors accurately identifying the jars on each of 15 trials.

writing usually hides a lack of competence and knowledge, rather than demonstrating it. It also serves to alienate lay people and even scientists from other disciplines. It is exactly the kind of writing I expected from reading the abstract, with its unnecessary use of trendy terms. In truth it actually isn’t a bad paper underneath, despite my expectations, so I all the more wish they could have stuck to a more concise, less show-offy (that’s not jargon, just a made up word, by the way) reporting style.

I think what they were actually trying to convey with this sentence was that their participants’ smelling ability, rather than being a smooth spectrum from rubbish to good, was broken up into well-defined groups. Indeed, around 49% were deemed to be super-detectors, who had around a 75% accuracy rating by the final trial. 33% were just ‘detector’s (around 40% accuracy on the final trial) and around 18% were ‘non-detectors’ with 0% accuracy. Now, as I briefly outlined earlier, this concept of super detectors rests on the idea that there is a proportion of the population who have an unusually heightened ability. Any definition you look up of ‘super’ is likely to include the words ‘particularly’, ‘especially’, ‘unusually’, etc. This makes it a peculiar term to apply to the largest group (half the sample!) This is the majority, not some niche elite … and here again we arrive at issues with, not the underlying paper itself (the statistical analysis is actually as far as I can tell excellent) but with sensationalism and dressing-up in the write up. These authors used the term ‘super-detectors’, despite the ludicrous fact that their ‘super’ group was half the sample. The only reason for this can be that it is an eye catching term and increases their chances of getting published. Sigh. There are no 100% objective scientists. They are all just regular people who need to further their careers. 15 year old me would be very depressed.

Read More

Why WorkFlowy and Latex make the perfect partnership for Academic Writing.


I have just handed in my thesis. Yes, it was tough, but not as tough as it could have been. Why? Mostly because of a little web app called WorkFlowy which isn’t supposed to be for writing at all, let alone for a piece of writing as big and formal as a doctoral thesis. But it works. By god, it works. This post is firstly going to try convince you to adopt a combination of WorkFlowy and Latex for your academic writing, and then guide you through setting them both up with tips, tricks, add-ons and advice galore. This is going to be a long post, and is not necessarily devised to be read in order. Below is a table of contents to give you an idea of the layout. If you know nothing about WorkFlowy or Latex, I do suggest you start from the beginning. If you use both of these but want to know more about the best way to integrate them and use them to write a thesis, just skip to any section that grabs your attention. Throughout the post there will also be links to shared WorkFlowy lists and one particularly important one containing a load of Latex code and tips, so look out for those.

Here’s what’s to come:

  • The Power of WorkFlowy
  • Latex
  • WorkFlowy – Latex’s Best Friend
  • Writing Academic Papers in WorkFlowy
    • Tagging
  • Writing Latex in WorkFlowy
    • Phrase Express
    • Keeping your .bib file in WorkFlowy, a drop down list of references and citing papers
    • Miscellaneous tips
  • Comments, thoughts, ideas …
  • Keyboard shortcuts, Navigating and Bookmarks
  • When you are struggling
  • Compiling
  • Conclusion

The Power of WorkFlowy
WorkFlowy is an ‘outliner’, and it is at first sight a simple-looking program. When you first open a WorkFlowy account you are pretty much just presented with a single bullet point, like below, which can be a bit ‘Err, OK’.

Figure 1. The Humble Beginnings of your grand WorkFlowy document.

But this reductionism is in fact WorkFlowy’s greatest strength. Hidden behind this simplicity, it has a few core features that combine to make it a phenomenally powerful tool for a surprisingly vast range of activities:

  • Every ‘item’ in WorkFlowy is either a bullet point or a note attached to that bullet point. Bullet points are therefore simultaneously content and ‘folders’ for other bullet points.
  • You can nest bullet points within other bullet points to infinite levels (see below)
  • Figure 2. A Basic WorkFlowy Structure. It’s bullet points all the way down.

The best advert I can give is for you to just try it out yourself. Here is a link to a ‘shared’ list on my account which you can play around with. It is a ‘demo’ list so it will revert back to its original form the next time you click on it (or anyone else does), so feel free to go wild.

Link 1. A demo WorkFlowy list to play around with.

  • You can ‘zoom in’ to bullet points by clicking on them to make that bullet point the ‘header’ and everything nested under it the only content on the screen – you can keeping zooming in as deeply as you like. When zoomed in you can see the ‘breadcrumb’ navigation at the top showing all the ‘levels’ above your current. Click on any of these to return to those levels / zoom out. Alternatively, alt + left will zoom you out, and alt + right will zoom you in.

Figure 3. WorkFlowy, zoomed mode!

  • WorkFlowy has a full tagging system with intelligent search function and logical operators using # and @ tags.
  • It is entirely plain text with the sole exception of bold and italic functions and so extremely lightweight and fast.
  • It has a raft of keyboard shortcuts to help you get around at rapid speed.
  • You can customise almost everything. Because it is web-based and simple, add-ons are super easy to make, and there are a vast range of excellent third party CSS additions which add colour, functions and a load of extra features – as you have access to the CSS code these are easy to customise to your heart’s content. These provide further functionality while keeping your actual document lightweight. For example you can add colour, like below:

Figure 4. WorkFlowy, in technicolour!

There are endless debates on various blogs and forums about whether hierarchical (e.g. OneNote) or tag-based (e.g. Evernote – if you’re doing it right) structures are best for digital organisation. In truth (as always) they both have their strengths, and are useful in different circumstances. The ideal of course is to have both at your disposal when needed. With the invention of infinite zoomable bullets, WorkFlowy has in my opinion created the perfect hierarchical system (I may be wrong but I honestly don’t really see how it can be improved upon other than with further ease of navigation) – navigating around something like One Note after a few days with WorkFlowy is hell itself – the differentiations between ‘different levels’ like ‘Notebook’, ‘Page’, ‘Sub-page’ etc have come to seem to me so pointless, dated and even quaint. Why make the distinction? Just allow the user to decide. With its version of a tagging system WorkFlowy has some advantages and some disadvantages over Evernote’s approach, but overall the same power to completely construct almost any organisation system you want that tagging systems afford. If I could make an analogy, I would describe WorkFlowy as a ‘sandbox’ digital organizer. Just as Minecraft removes the constraints on how you ‘should’ play a computer game and encourages you to explore your own way of doing things, so too does WorkFlowy remove the constraints on how you should organize your life. Once you absorb WorkFlowy into your life, other programs come to feel horribly restrictive. After around a year of use I have slowly come to use it as calendar, project and task manager, journal, life logger, finance and budgeting tool, and yes, thesis writer. None of these things came as templates or anything like it, I have ‘built’ them myself (with a lot of help from the WorkFlowy blogger Frank Degenaar’s great book), There is no compromise here – I am very aware of the popular products for these functions and WorkFlowy surpasses all of them if only because of sheer customisability and of course perfect integration / a single search box for my entire life. But this post is about thesis writing – I do intend to detail my life organisation approach with WorkFlowy at some point but that will have to wait for a later post (I should probably do some science posts in between to maintain the charade of this as a science blog too!).

Now, pricing. WorkFlowy uses a freemium model, and the free version is absolutely fine to get a feel for the program and even for casual to medium usage. The main restriction on a free account is the number of items (bullet points) you can create per month: 250 with a basic free account, but, magically, 500, if you sign up via this link: This is actually less of a restriction for thesis writing than general task-type use, as you would tend to be making less, but longer (i.e. paragraphs) items, than a typical to do list for example. So I think you can definitely get a very good feel for everything in this post entirely for free. I suspect however, that once you get going with WorkFlowy, you will shortly be hooked, and the ~£5 a month for the level of life-organisation and peace of mind it provides.


Latex is of course much more well-known than WorkFlowy and is used throughout the world by academics to write their papers, so I am not going to spend much time introducing it. Basically, it is a relatively simple coding language for producing academic papers (and books etc.) It has a relatively steep learning curve (it’s really not that steep, but compared to a classic word processor…), which is presumably what puts some people off. However, not for you. Below is a WorkFlowy shared link with all the instructions you need to get set up with Latex, and all the snippets of code I (and therefore hopefully you) needed (/ will need) for your thesis or academic paper. This is a live and on-going list which I will also be sharing through other means, and as it says in that link, if there is anything academic-related you want to know (or perhaps even a tip you have that should be added) please let me know and I will endeavour to include it.

Link 2. A guide to using Latex (particularly but not exclusively with WorkFlowy).

WorkFlowy – Latex’s Best Friend

One thing a lot of people love about Latex is that you can write it in any text program – even in a .txt file. It is therefore lightweight (unlike Microsoft Word), unlikely to crash (unlike Microsoft Word) and it doesn’t intentionally format things like tables or figures in insane ways just to irritate you (I have a very healthy distaste for Microsoft Word). The other thing of course is that with all the packages out there (Latex is open source) you can format / layout a write up any way you like, with confidence, and, at the flick of a switch (function), have it formatted in a completely different way (say, for a different journal, or to transform a part of a thesis into a paper, as I am doing now). Now, there are some great Latex editors out there, like Tex Studio, which provide some nice features to help you write a big piece of work, such as a section navigation pane. However, none of them comes close to the power of WorkFlowy for helping you focus on exactly what you want, whether that is a single small section, full screen, with no distractions, or multiple sections from different parts of your report at the same time. And none of them has a full tagging structure to keep track of the state of every section of your work, or the ability to make endless notes on any section of work, or to have \subsubsubsubsubsubsubsubsubsubsubsubsections which don’t even have to show up in the final output, or allow you to see the structure of your document not only as an outline, but inherently, in the main body of the text, or to have keyboard shortcuts to take you back and forth between different sections, or the ability to edit perfectly well on a mobile device, without any fuss or worry. So. We are going to see how by combining WorkFlowy and Latex we can overcome the weakness of both (a lack of formal formatting capabilities in WorkFlowy’s case, and a lack of … well, WorkFlowy features, in Latex’s case) and harness each other’s strengths.

Writing Academic Papers in WorkFlowy

As I’ve said, WorkFlowy is an outliner: a set of nested bullet points. These nested bullets can be infinitely deep and each bullet point can also have an attached ‘note’ (shift + enter to edit the note of a bullet). When I write serious things with WorkFlowy the bullets become the section headings and the notes contain my prose. You can see a real example from my thesis below.

Figure 5. A depiction of a series of sections of my thesis.

Some might prefer to just use bullets and sub bullets for both section headings and content, and you could, but I find my approach easier for formatting (more freedom within a note), and simply for taking in the structure of my document visually at a glance (because bullets and notes look different). Now before you get writing in WorkFlowy you are going to need a few add-ons. To get started you need to install the ‘Stylish’ extension (available at the least for chrome, Firefox and Safari with a quick internet search) which allows you to make visual user-end modifications to web-pages. The first and perhaps most important of these for writing is an add-on that allows you to have all the notes attached to your bullets ‘open’ or ‘expanded’. This option is not a native feature of WorkFlowy unfortunately, but this add on works perfectly so do not worry. To get this add-on, once you have stylish installed, open WorkFlowy, click on the ‘Stylish’ button that has now appeared on your browser bar and click (right at the bottom) ‘Write Style for’ Copy and paste the below code in, give it a name (e.g. ‘Open All Notes’) and save.

div.notes div.content {display: block !important; height: auto !important; overflow: visible !important;}

Now click on the Stylish button again and there will be a check box with the name you used. You can choose to flick the ‘Open All Notes’ style on or off. Flick ‘Open All Notes’ off to get an overview of your document and for quicker navigation, or flick it back on to have a full-prose format (like in my thesis screenshot above). Don’t forget if you really want to focus on a specific section to use the ‘zoom’ functionality to make it full screen.


WorkFlowy has two types of tags, the @ tag, and the # tag. You can of course use them how you like – my preference is to use @ tags for structure e.g. @Thesis, @Chapter1, @Introduction, @Method and # tags to indicate the status of things such as #todo, #reviseddraft, #finaldraft # wip (work in progress, to allow you to easily move between sections of your document you are currently engaged with – kudos to Frank for this invention). This nice thing about maintaining a rigid distinction like this is that sometimes you want two tags called the same thing but with different functions. For example, you might want to use the @Introduction tag for structural searches, but you might also want to tag a note, comment or thought (see further down for more on this) with #introduction to bring all thoughts (wherever they are) about your introduction together with a search. When you come to write each day you can use your tags to decide what to work on. Finding sections that are marked #todo and zooming in on them is far less daunting and easy to get into first thing in the morning than a huge word document with no obvious place to start. It is tricks like these that help ease you into writing each day and curb procrastination. The advantage of having structure tags is firstly for speed of navigation but secondly that you can easily see all of a certain type of section, for example all the discussion sections (if you have multiple experiments) with a simple search, all in a single pane. If you have several experiments with their own small discussions, and then a general discussion, this can be a great way to see if they flow / if you have repeated yourself, etc. If there are certain unrelated sections of your document you want to see all at once, just make up a new unique tag on the spot – at one point I wanted to look at all the sections which were to do with #NestedSetsvsNatFreqs (don’t ask) – so I simply tagged them all with this then searched for that and bingo, there they all are, neat and organised. Things like this, which require so much effort in other programs are child’s play in WorkFlowy.

Writing Latex in Workflowy

Phrase Express

Now, I want to briefly introduce another application into the mix (and again thank Frank for the introduction to it). Phrase Express is one of many, but I believe a particularly good / easy to use, keyboard shortcut / macro program. It is perfectly functional in its free version and it’s up to you if you think you get enough value from it to support them (I certainly get enough). It is also extremely easy to use and can be incredibly powerful. One problem with writing in Latex is human error. If you write out all the code necessary for an entire thesis, there will be mistakes, a lot of them, and they can be hard to find (however, see the section in the Latex link on debugging). I recommend putting every single Latex command (yes even the simple ones) into Phrase Express. This not only allows you to access them with a keyboard shortcut (increasing speed for large things like figures), but also ensures you never, ever make a mistake. This will save you so much frustration at the end. Shift and alt and f provides me instantly with all the code needed for a figure. Shift and alt and 1 turns any selected text into a perfectly formatted chapter heading (and so on for shift and alt and 2, 3, 4, etc). Shift and alt and p opens up a parentheses citation (more on this shortly) and shift and alt and t, a text citation. It really is wonderful once you get it going. Here’s a link to the phrase express download page:

Keeping your .bib file in WorkFlowy, a drop down list of references and citing papers.

The one thing that completely sold me on Latex was the citation management and the one thing some people might have raised an eyebrow at in leaving something like Tex Studio is the ability to open up a citation command and get a drop down list of all the papers in your .bib file. You are right that this is of course not an innate function of WorkFlowy and would be sorely missed. However, the exact same function can be achieved through @ tags, which also come with a drop down menu. I must confess to being quite excited right now: this is the best thing in this post. Firstly, I would wholly recommend using WorkFlowy as a place to keep your bibtex database, even if you didn’t use WorkFlowy for your writing at all. I recommend a setup like the image below.

Figure 6. My .bib file in WorkFlowy.

Use a blank bullet point, with the bibtex entry in the note. Of course, if you wanted to arrange your bibtex entries in some hierarchical format according to subject, this would be easy to achieve, and I probably haven’t explored everything you could do with this. However, right now we are going to take this to the next level. For each bibtex reference ‘code’ e.g. Turner2015, put a space and an @ tag in front of it (you need the space – see image above). So Turner2015 would just become @Turner2015. It has just become a tag. When you come to compile, Latex has no issue with either the space or the @ sign, so don’t worry about that. However, you now have two great new features. Firstly, you can search your .bib file in an instant with a drop down menu of codes from the search box (say, if there’s an error with a particular reference). Secondly, when you want to cite a paper from your .bib file in your document, when you type \citep{ @ a drop down list of tags appears, with all your .bib codes, just like in Tex Studio or similar. This admittedly only apes that same functionality, and doesn’t better it, but the ability to manage your actual database in WorkFlowy is far better I think than even the professional citation management systems can offer. You can utilise WorkFlowy’s hierarchical and full tagging system to sort entries by topic, you can easy add additional information such as links to the appropriate pdf or, perhaps, to your Evernote summary page on that paper (See my post on a system for using Evernote for academic literature reviews). When you want to export your .bib file it takes four clicks: alt and click on the title (copy all – alternatively click anywhere and press ctrl a twice), copy, open target .bib file, paste. Done.

Now, for actually citing papers while writing I have a set of phrase express macros which make life far easier.

First, to get a paranthetical cite just use ‘/citep[][]{ @’ without apostrophes and for one which gives the text cite ‘\citet{ @’. For these you literally just copy that text into phrase express and when you press the keyboard shortcut it will just spit out the text.

Then, I have one which tidies up the previous citation (removes the spaces) and adds another citation (if you are citing multiple authors), like this: \citep[][]{@Gigerenzer1996, @

This is a bit trickier. To get this working, copy the following into phrase express:  x{#CTRL {#LEFT -count 2}}{#LEFT -count 2}{#DEL}{#CTRL {#RIGHT -count 2}}{#DEL}{#BKSP}{#BKSP}, @

Then we have another macro that tidies up and closes a citation off (when you are finished citing), like this: \citep[][]{@Gigerenzer1995,@Macchi2000}

To get this working, copy the following into phrase express:  x{#CTRL {#LEFT -count 2}}{#LEFT -count 2}{#DEL}{#CTRL {#RIGHT -count 2}}{#DEL}{#BKSP}{#BKSP}}

I do recommend you explore Phrase Express. While the above may look complicated in written form this can all be achieved by using the drop down menus found in the ‘Macro’ menu – it can all be done through the far more friendly button-and-menu-based user interface rather than pasting ugly bits of code and really is quite easy to build your own macros through trial and error.

For those less familiar with what I am on about in this section, below you can see the before and after for both ‘paranthetical’ citations and ‘text’ citations below:



Miscellaneous tips

I would now like just to give a range of miscellaneous tips for writing with WorkFlowy and latex.

  • WorkFlowy is a web app, so you are perfectly able to have two instances of the web browser up, split screen, and further to have as many tabs open on those two browsers as you like. Too many can get confusing, but for serious writing I often find this extremely useful.
  • The % Sign. Get in to the habit of putting a % sign after every section title (it can even be part of your phraseexpress macro). That way, if you want to add tags after section titles, like #firstdraft these won’t show up in Latex and ruin your compile.
  • WorkFlowy has intelligent search functions. You can use the OR operator to good effect for debugging your latex document. For example the search { OR } will highlight ALL curly brackets, making it super easy to spot any opener without a closer.
  • Search & Replace (Chrome). For the eternally-useful ‘search and replace’ function, there is a great chrome app called exactly that here. However there is a little (admittedly slightly annoying) trick you need to know to use it with WorkFlowy:
    • Once you have done your replacing, you need to click in every note that you changed (not on every change, just in every note, so if you have multiple changes in one note, just 1 click), otherwise when you refresh the page, the changes will revert – annoying I know, but it’s a 30 second job really.
    • Pro Tip: To make this a bit easier, first search in WorkFlowy for the thing you are going to replace. This will bring up only the notes with them in. THEN do the search and replace thing, and you won’t have to scroll through your whole document looking for the notes you have to click in – with this additional step, it’s really not so bad. An added advantage is that even after replacing the words, they will remain highlighted until you click on them, so it is easy to see where to click.
  • The WorkFlowy app on both android and iphone is fantastic. You can do serious work in this app on the go, on your thesis, seriously (it can even be a nice mental change). This is a tiny bullet point in a large post, but this is no small benefit to using WorkFlowy – the ability to zoom in to the section you want to work on takes on even greater power on a small screen. No more scrolling through endless reams of tiny text and feeling overwhelmed before giving up and checking Facebook. Use a tag search to go straight to sections which are still in #firstdraft or #todo status, edit them on the go (editing is a task I often intentionally left for phone time as it doesn’t need too much typing), change them to #reviseddraft, move on to the next. There is no better irritation-free text editor on mobile than WorkFlowy, I guarantee you.
    • For more advanced computer-like functionality on iphone and android, check out the excellent third-party HandyFlowy app (no offline mode though and relatively data intensive).
  • Here are some more great add-ons:
    • The highlighter tool. WorkFlowy has no highlight function, but do not fear. To create your own, make a new stylish style and enter the below code. By the way, this code turns underline into a highlighter, so if you love underlining, you’ll have to make a choice. I suppose you could alter the code to make bold or italic a highlighter instead, but unless you’re unusual they are probably more often used than underline.
      • span.contentUnderline {     background-color: #FFFF00 !important;     text-decoration: none !important; }
    • Customisable coloured tags (code below, just copy and paste it in to a new stylish style). You can modify these to your hearts content. Just change the tag name in the code and choose the colour hex code (the #FFA500 thing). Just internet search for hex colour codes.
      • .contentTag[title*=”#firstdraft”] { color: #FFA500 !important; } .contentTag[title*=”#reviseddraft”] { color: #FFD700 !important; } .contentTag[title*=”#finaldraft”] { color: #00ff00 !important; } .contentTag[title*=”#comment”] { color: #FFD700 !important; } .contentTag[title*=”#todo”] { color: #FF0000 !important; }
    • Rawbytz and the WorkFlowy Count
      bookmarklet / add-on:
      • ‘Rawbytz’, another key figure in the WorkFlowy community, has created a range of extremely useful ‘bookmarklets’ / browser add-ons which add extra functionality to WorkFlowy: see his webpage for a full list here.
      • If you want to log how long you spend on your write up, you can add time tags such a #1h #50m, etc. – these need the third party app WorkFlowy Count (See Frank’s post on this bookmarklet here) to be recognised – but if you want, the feature is there. If you do a bit of work just quickly throw the amount of time you spent in the section header (after a % sign, remember), and at the end you can click on the bookmarklet and it will give a pop up window which instantly adds them all up – a voila – you can even see which sections took you the longest by zooming into that section before pressing the button.
    • Tag Index Bookmarklet / add-on:
      • Another one by Rawbytz. This gives you a paste-able list of all the tags in the current list. Handy if you’ve gone wild with tags and forgotten them all: See Frank’s post on this here.
    • Collaborating & Tracking Changes. WorkFlowy is an incredibly intriguing collaboration platform, and I am only just beginning to explore the potential of this. Hover over the bullet point of a list and go to share – give someone the link that pops up and they can view / edit the list live along with you. The best thing is they don’t even need to have a WorkFlowy account, so you don’t have to convince your supervisor or collaborator to sign up to WorkFlowy or install anything at all in order to check over / comment on your work.
      • Because WorkFlowy is such a DIY program, you need to devise a collaboration ‘system’ or ‘code’ yourself: for a very well-considered one which has been through considerable testing, check out Frank’s offering here.
      • Also check out Frank’s system for tracking changes here.
        • By the way, as with all of these add-ons, if Frank’s specific choice of bold-italic-underline combinations don’t work for you (personally, I found Frank’s to be a bit too complicated to ask a WorkFlowy-naive supervisor to do) – you can easily change the code to suit your needs. I have had some success with simply asking people to underline text they want to remove, italicise text they want to add and bold any in-line comments). As we use these very rarely in academic writing, there isn’t much issue around clashing with content you actually wanted bold etc. (other than headings but you can just ignore these).
          • If you do want to change the style for this add on just click on the stylish button and go to the edit button next to it. At the top of each block of text you will see somethings saying e.g. ‘span.contentUnderline.contentBold’. That means the formatting beneath will apply when text is both underlined and bold. Delete / change that to whatever you want such as just ‘span.contentBold’ – now the formatting changes beneath will apply to anything bold.
  • Placeholders. Tags are great if you need a placeholder as you can search for them later. Common examples when I was writing up were @figure, @table, @reference, etc. When I was thrashing something out and didn’t want to take the time to insert a figure, OR I wanted to refer to a figure or table but it didn’t exist to refer to yet, I used these. When I came to the point where I wanted to fill these in, I didn’t have to trawl through the whole document to try and spot them, but just had to carry out a simple search – and of course, I didn’t miss any.

Comments, thoughts, ideas …

While writing, one has many ideas which are not formal enough to include in the main text. There is no specific comment ‘function’ in WorkFlowy. However, what there is is better. Firstly, when writing in WorkFlowy I recommend breaking your document down into very small sections as a general rule – this makes it way easier to understand your structure, and when you really want to hash out a paragraph, being able to look only at this is just wonderful. Now, if you want to make a comment on the specific section you are working on, make another bullet point nested under it, start the bullet with a % so it will be ignored by any Latex compiler if you accidentally leave it in, use a tag like #comment and say what you need to say, then minimise it so it doesn’t get in the way during normal writing. Add any tags that make sense for you like #introduction #discussion if the thought relates to those sections. Avoid going overboard with tags as if you forget they exist, you can’t search for them and they serve no function. You can now easily search your document in an instant for every section which has outstanding comments. If you want to make inline comments, I firstly recommend you use the Coloured tags add on from the link above to give the #comment tag some colour that works for you (mine is yellow). Use the highlighter to highlight sections of the text which you want to refer to then either add a #comment nested bullet or in the body of the text add a comment and put your comment in between two square brackets (or whatever works for you of course). You might also want to highlight your comment, who knows. You can do the same thing with anything: a chunk of text from a pdf you think might be relevant, some scrap notes you wrote, just nest it under the relevant section and give it a #comment or #note or #todo tag, or whatever. If you have an idea about a different section, don’t bother navigating there – just tag it with #introduction or whatever and run a search for these later. Or, have a ‘thoughts’ section somewhere, with a keyboard shortcut (see below) to navigate there, then once you’re done, just press the back button on your browser to go back to exactly where you were.

Additional note, 01/12/16: I recently found a trick with comments in WorkFlowy which I thought I would add here. If you want to add the comment ‘this needs revision’, consider formatting it like this: #comment-this-needs-revision . The entire comment is now a single tag. Now, if you have configured stylish to make the #comment a certain colour, this will also be that colour, so the whole thing really pops out. Secondly, if you search for #comment, this and any others like it will show up in the search. Finally, if you want to get rid of the comment, you can hold down alt and left click on it with the mouse (alt and left click ‘explodes’ tags in WorkFlowy). So there you have it: searchable, coloured, one-click ‘explodable’ comments. The only downside is pressing – instead of space, but I don’t find this too bad personally.

Keyboard shortcuts, Navigating and Bookmarks

There are lots of extensions for every browser which allow you to assign keyboard shortcuts to bookmarks. One I use at the moment for chrome is called speed dial. What I like about this is it lets you have your bookmarks in a folder so keeps your bar neat. One very important and wonderful feature of WorkFlowy is that every bullet point has its own unique URL. When you have WorkFlowy open every word of your document is ‘loaded’ (even this doesn’t take long because it’s all just simple text). You can therefore type the URL of a bullet point (or list) into the address bar, press enter, and it will open instantaneously, without loading, even offline. This means that you can save WorkFlowy lists to your bookmarks bar, assign keyboard shortcuts and navigate back and forth between these different sections at the press of a button and without waiting for it to load – you can also re-assign these shortcuts extremely easily anytime you like. Even better, these bookmarks can be saved searches, so if you search for #comment, then bookmark the URL, every time you click that URL it will run that search from the same place you ran it when you bookmarked, not from your current location. This is a wonderful thing. I recommend for example a bookmark which displays the major sections of your document (this might be a search from the ‘top level’ of your document for “@Introduction OR @Method OR @Results OR @Discussion”. This will make it extremely easy for you to navigate to a new section whenever you like. A “#comment OR #todo OR #firstdraft” search from the top level might also be a good idea for when you want to know what to do next. Another nice search is the lastchanged:1 search. This shows every bullet you have altered in the last minute. The 1 can be changed to a different number and you can even use 1h or 1d etc to search for hours and days.

When you are struggling

When I am struggling with writing a section, the best approach for me (and I think for many others) is to break it down – and at breaking a section of text down to help you understand it’s structure, all programs bow to WorkFlowy. Just cut-and-paste (by the way if you ever use the mouse to cut, copy and paste, stop it right now and learn ctrl + x, ctrl + c, and ctrl + v) to break it up into smaller and smaller bullet-point-headed sections until you have a grasp on the argument being made. Use the actual bullet point to summarize each paragraph and the note to contain the actual text. A useful guide I teach my students is to use a single sentence to summarise a paragraph (write this sentence in the bullet point). Now, just read the bullet point sentences and you will be able to tell far more easily if the argument you’ve made makes sense, flows effectively, or where the logical flaws are, than if you tried to read the whole section in full. Maybe you need to delete a whole section. A neat way to do this without losing the section is to use ctrl + enter which in WorkFlowy ‘completes’ the bullet point. Use ctrl + o to toggle whether completed sections are visible or invisible. If you turn it to invisible you can ‘try’ deleting a section and easily bring it back if you regret it. Maybe you need to reorganize. Dragging different sections around by clicking and holding on the bullet point makes this child’s play. Maybe you need to tag different paragraphs according to how vital they are to the argument being made / how well written they are. It is times like this that I think WorkFlowy shows its greatest power. It makes the frustrating enjoyable, and the difficult easy. It makes writing fun.


Compiling from WorkFlowy is as easy as two copy + pastes. Open up your entire document (I recommend including every single word in WorkFlowy including the whole pre-amble, /begin{document}, etc.) by going to the top level and double clicking on the header (this ‘expands’ / ‘opens’ all nested lists, ensuring everything is copied. Alt and click on the title (or click anywhere and press ctrl a twice) to select it all, and copy and paste it over (I personally used Tex Studio for this compile but sharelatex is becoming very popular and has just introduced a track changes feature which is currently unique among Latex compilers. Then, do the same for your .bib file. Compile. Done. Well, actually you’ll undoubtedly have some errors to sort out, but that’s Latex life.


Without Latex, WorkFlowy wouldn’t be great for writing a thesis. It would still be nice for getting a first draft because it helps your mind flow and helps you understand the structure, argument flow, and nature of your document, which is the key to good writing. With Latex however, WorkFlowy is a thesis powerhouse, all the way up to the final compile and beyond. Reviewers come back with a raft of comments? No problem. Insert them all as sub-bullets under the appropriate section, tag them all up as #todo and perhaps according to how difficult they will be to deal with (allowing you to deal with the hardest when you are at your most cognitively able – first thing in the morning for me), deal with each in turn by zooming in (avoiding being overwhelmed), break difficult sections down, reorganize, tag, rewrite. It’s all just so much easier. I beg you, for your own sanity, never write another paper in anything but WorkFlowy with Latex again.

Read More

Can Statistics and Law ever learn to get along?

R v Adams

In 1996 a jury heard forensic testimony that a ‘match’ had been found between Denis Adams’ DNA and a sample found at the scene of a crime where a woman had reported being assaulted and raped. The probability of this match occurring by chance was described by the forensic expert as being ‘1 in 200 million’. In Adams’ defence his lawyers related to the jury that the victim herself in fact failed to pick the defendant out of a line up and even explicitly stated that Adams did not match her description of her assailant. Further, Adams girlfriend testified that he was with her on the night the incident took place. Despite this, the jury convicted Adams of the crime. The defence, convinced that the jury had overly weighted the DNA evidence in their deliberations, immediately launched an appeal. Unsure of the correct way to combine the three pieces of evidence, they recruited a statistical expert, Peter Donnelly (Donnelly, 2005) to undertake a ‘Bayesian’ analysis of the case.

It was resolved by all parties involved that the statistical calculations must be undertaken by the jurors themselves. Donnelly, in combination with the statistical experts from the prosecution, devised a questionnaire to encourage the jurors to quantify the various pieces of evidence presented in the case. For example, in regards to the failure of the victim to identify Adams, the jurors were asked to provide numbers to the questions ‘If he were the attacker, what’s the chance that the victim would say her attacker didn’t look anything like him?” and “If he wasn’t her attacker, what’s the chance she would say this?”. Once the jurors had given numerical estimates for the value of each piece of evidence, they were then guided in how to combine these using Bayesian techniques to arrive at a figure representing the value of all three pieces of evidence combined.

However, the attempt to guide the jurors and the judge through this process was described by Donnelly as rife with misunderstanding, mishaps and general difficulty, some of which Donnelly elucidates:

‘The episode had some amusing sidelines. It was suggested that it would be helpful to supply the jury (and judge) with basic calculators. Although the total cost was well under £100, this request was so unusual that it seemed to require clearance personally from the Lord Chancellor. Then, during my evidence, we walked the jury through a numerical example—the barrister would suggest token numbers in answer to the questions, and the jury and I entered them in the calculators which were eventually supplied. They seemed to have no difficulty in following this, but at an early stage in the calculation, when I said something to the effect that: “Your calculator should now show the value 31.6,” and the jurors all nodded, the judge rather plaintively said: “But mine shows zero.”‘ Donnelly (2007)

The appeal was eventually rejected, with the appeal judge scathing of the statistical approach used. As a result of his experiences, Donnelly remains unconvinced that such an approach is a feasible future for the presentation of Bayesian reasoning in legal cases.

But can there be a future for statistics in the court room? Is there another way? And what even is Bayesian inference anyway?


‘What even is Bayesian Inference anyway?’

Bayesian inference is the mathematically-accurate method of updating a ‘prior’ probabilistic belief in a hypothesis (such as Adams being the attacker) in the light of new evidence (such as the DNA evidence, the alibi, and the line-up identification failure) to arrive at a ‘posterior’, or updated belief level in that hypothesis.

It might be clear that this general concept, of updating one’s beliefs in ‘something’ in the light of new information, is hardly one restricted to the court room, and indeed some believe this fundamental belief-updating process, and therefore Bayesian inference, is central to almost all human endeavours (McGrayne, 2011; Link, 2009; Gelman et al. 2014).

Bayes’ formula for undertaking this inference was published over 250 years ago (Bayes & Price, 1763). A picture of Reverend Bayes next to his famous formula can be seen below, however I don’t want to get bogged down in the algebra – there are many (many) thorough explanations of it elsewhere. Suffice to say at this point that what you get out of the formula (highlighted red below and known as the ‘posterior’) is the updated belief level, and to calculate that you combine the prior (green: the old belief level) with the new information / evidence (blue). Hopefully that makes some intuitive sense.

Bayes’ theorem has been extensively validated and is no longer in any doubt as the correct approach in probability-updating situations amongst the statistical community: as Fenton, Neil and Hsu (2014) stated:

‘The application of Bayes’ theorem to probabilities is akin to the application of addition or multiplication to numbers: probabilities are either correctly combined by this rule, or they are combined incorrectly by other means.’ Fenton, Neil & Hsu (2014)

So, if the numbers going into the formula are correct, or correspond to reality, then the number coming out will also be correct. But here of course, lies almost all of the contention: the conversion of non-quantified beliefs (Adams’ guilt; your chance of catching a bus; a patient’s probability of having a given disease; how much your friend likes you; a football team’s chance of winning a match) into the quantified ones the formula requires. Nowhere is this conversion currently more contentious than in the legal realm. However there are work-arounds: one can calculate probability ‘distributions’, for example, taking into account multiple feasible valuations of each piece of evidence (e.g. those most in favour of the prosecution and those most in favour of the defence). For example, while there is no access to the original figures calculated by the jurors in the Adams trial, a Bayesian post-analysis of the case by Dawid (2002) suggested that the probability distribution of guilt taking into account the three pieces of evidence may be as low as 0.36 or as high as 0.98. He believed this analysis demonstrated that there was room for ‘reasonable doubt’. Perhaps this also demonstrates that the techniques can be informative to trials like this.


‘Trial by Mathematics’

There are many opponents to the use of Bayesian inference in court cases, and many of them point to a now-classic paper by Professor Laurence Tribe (1971), entitled ‘Trial by Mathematics: Precision and Ritual in the Legal Process’ in Harvard Law Review. Tribe begins the paper with an implicit comparison of modern attempts to ‘mathematize’ the legal process with those from the middle ages:

‘The system of legal proof that replaced trial by battle in Continental Europe during the Middle Ages reflected a starkly numerical jurisprudence. The law typically specified how many uncontradicted witnesses were required to establish various categories of propositions, and defined precisely how many witnesses of a particular class or gender were needed to cancel the testimony of a single witness of a more elevated order. So it was that medieval law, nurtured by the abstractions of scholasticism, sought in mathematical precision an escape from the perils of irrational and subjective judgment.” Tribe, 1971

Tribe’s implied point here is: this was tried before, and it is as bad an idea now as it was back then. One of Tribe’s main functional arguments for this (apart from some compelling moral arguments), is that statistical evidence will be far more salient, or attractive to the jury, than the non-mathematical evidence that they will always be asked to combine it with, because statistical evidence exudes an “aura of precision”. Tribe argued throughout his paper against an article in the very same journal issue by two authors named Finkelstein and Fairley (1971), who were proposing the use of Bayesian inference in legal trials for the first time (presumably the journal had approached Tribe for his views prior to publication instead of this being some wonderful coincidence). Finkelstein and Fairley were proposing a system somewhat similar to that employed in R v Adams above, where the jurors convert their beliefs in numerical values. Tribe makes the point that:

Even assuming with Finkelstein and Fairley that the accuracy of trial outcomes could be somewhat enhanced if all crucial variables could be quantified precisely and analyzed with the aid of Bayes’ Theorem, it simply does not follow that trial accuracy will be enhanced if some of the important variables are quantified and subjected to Bayesian analysis, leaving the softer ones – those to which meaningful numbers are hardest to attach – in an impressionistic limbo. On the contrary, the excessive weight that will thereby be given to those factors that can most easily be treated mathematically indicated that, on balance, more mistakes may well be made with partial quantification than with no quantification.” Tribe, 1971.

I hold some sympathy for Tribe’s views that the legal process might be better if mathematics were kept out of it entirely, particularly at the time it was written. However, unfortunately for Tribe’s modern proponents, while they stand with their shoulders at the door of the courtroom, using Tribe’s arguments to keep the statisticians out, they have forgotten to look behind them. And behind them the real nightmare that Tribe hoped never to see has crept up over the last three decades: legal professionals with no statistical training misusing mathematical evidence, overweighting it, and leading to numerous miscarriages of justice (e.g. Forrest, 2003; Mehlum, 2009; Donnelly, 2007)

The problem grew largely due to the rise of the use of DNA evidence in court cases from the early 1990s. When forensic teams report a DNA match in court and they want to get across the importance of the evidence, they generally report the probability of a random person from the population matching the sample found at the crime scene (the ‘random match probability’, as it’s known). The most notorious error in legal practise, single-handedly responsible for a swathe of miscarriages of justice is intimately entwined with this figure and is known as the Prosecutor’s Fallacy. To demonstrate, imagine you are a juror on a murder trial and you are told there is only one piece of evidence against the defendant: he matches a DNA sample found on the victim’s body which could only have been left by the murderer. You are then told the chance of this match occurring by chance (the RMP) is extremely low: only 1 in 2 million people have this DNA type. Now, what do you think is the chance that this person is innocent? If the answer that pops into your head is also ‘1 in 2 million’ I’m afraid you’ve just committed the fallacy. Why is this a fallacy? Well imagine the murder happened in London (with a population of about 8 million) and we determine that anyone in London could feasibly have done it in the time frame. How many matches for this DNA sample would we expect? Four. We already know the defendant matches, so he is another 1 – so our best estimate of how many matches there are in London is now 5. Since we have no other evidence against the defendant, the best we can say is that he is one of these 5 people, one of which must have committed the crime, so he has a 1/5 chance of being the assailant, or a 4/5 chance of innocence.

Now 4 / 5 is a very big difference to 1 / 2million. The mistake in reasoning here is to ignore the population size, or, the ‘prior’ chance of guilt (which the population gives us). Before we did the DNA match, what was the defendant’s chance of guilt? 1 in 4 million – he had no more chance of being guilty than anyone else in London. So when we combine this with the DNA figure of 1 in 2 million we end up with 4 matches. But what if we were talking about an island with only 50 people on it? How many matches would we expect here? Not even 1 (0.000025 in fact). So if we were talking about this island and the DNA match occurred it would be extremely likely he had committed the crime: the bigger the population, the smaller the prior and the less convincing the DNA evidence. As we saw above with Bayes’ formula, the new, ‘posterior’ belief level has to be a combination of both the prior and the new information (the DNA match) – the mistake in the Prosecutor’s Fallacy is to entirely ignore the prior, or the population size, and focus entirely on the new information.

In this simplistic example we didn’t include any other evidence, and while there are cases where the prosecution rested entirely on DNA evidence (the Adams case we began with for example), it is not often the case. However, unfortunately, exactly as Tribe predicted, cases such as R v Adams as well as a swathe of research studies have now shown that people vastly overweight DNA evidence presented as a ‘random match probability’, typically due to the prosecutor’s fallacy (e.g. Thompson & Schuman, 1987; Koehler Chia & Lindsey, 1995; Schklar & Diamond, 1999; Kaye et al, 2007).


‘Is there another way?’

Fenton, Neil and Hsu (2014) see trials like R v Adams as proof that any legal process which includes jurors or lawyers attempting to calculate the statistics behind a trial as doomed to failure. A recent experiment I conducted with them confirmed this view: we presented 100 general-population participants with a Bayesian legal problem with only a single piece of ‘match’ evidence, including all the variables (such as the possibility for error during forensic testing) they would have to take into account to accurately calculate the value of the evidence. Not a single person was able to get the correct result – and most trials include more than 1 piece of evidence.

So what other option is there? Fenton and Neil (2011) argued that if the jurors and the lawyers can’t do the maths themselves, then they are just going to have to trust validated computer programs to do it for them. While it would always be up to people to determine the numbers that go into the formula, once that is done, the computer program, running validated, mathematically-factual algorithms, should be trusted to produce the correct output. This, they argue, is comparable to the way we trust calculators to undertake large multiplications or divisions. What they argue for, in short, is a greater role for statistics in law.

While this approach may appear very much the polar opposite of Tribe’s more conservative views of keeping statistics entirely out of the court room, and we can’t be sure of his views on this, I think it is actually much more in the spirit of his classic rebuttal – his major fear was the misuse of statistics, particularly through overweighting, and that is exactly what is happening now. We are living in a half-way house, with untrained legal professionals (occasionally but not systematically with professional assistance) presenting statistics to untrained jurors, and expecting them to understand the calculations. This is, not the best, but the worst of both worlds. And perhaps unfortunately, we can no longer return to Tribe’s non-mathematical utopia. DNA is here to stay, and with it, comes the random match probability. The only way out of the mess, it seems, is forward, not backward.


Bayesian Inference. William Link (2009). Elsevier Ltd.

Bertsch McGrayne (2011). Yale University Press.

Dawid, A. P. (2002). Bayes’s theorem and weighing evidence by juries. In Bayes’s Theorem: Proceedings of the British Academy. R. Swinburne. Oxford, Oxford University Press. 113: 71-90.

Donnelly, P. (2007). Appealing statistics. Medicine, Science, and the Law, 47, 14–17. doi:10.1258/rsmmsl.47.1.14

Fenton, N., & Neil, M. (2011). Avoiding Probabilistic Reasoning Fallacies in Legal Practice using Bayesian Networks, (June), 1–44.

Fenton, N., Neil, M., & Hsu, A. (2014). Calculating and understanding the value of any type of match evidence when there are potential testing errors. Artificial Intelligence and Law, 22(September), 1–28. doi:10.1007/s10506-013-9147-x

Finkelstein & Fairley (1971). A Bayesian Approach to Identification Evidence, 83 Harvard Law Review, 489

Forrest, a R. (2003). Sally Clark–a lesson for us all. Science & Justice: Journal of the Forensic Science Society, 43, 63–64. doi:10.1016/S1355-0306(03)71744-4

Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2014). Bayesian data analysis (Vol. 2). Boca Raton, FL, USA: Chapman & Hall/CRC.

Kaye, D. H., Hans, V. P., Dann, B. M., Farley, E., & Albertson, S. (2007). Statistics in the Jury Box: How Jurors Respond to Mitochondrial DNA Match Probabilities. Journal of Empirical Legal Studies, 4(4), 797–834. doi:10.1111/j.1740-1461.2007.00107.x

Koehler, J., Chia, A., & Lindsey, S. (1995). The random match probability (RMP) in DNA evidence: Irrelevant and prejudicial? Jurimetrics Journal, 201–220. Retrieved from

Mehlum, H. (2009). The Island Problem Revisited. The American Statistician, 63(3), 269–273. doi:10.1198/tast.2009.08107

Schklar, J., & Diamond, S. S. (1999). Juror Reactions to DNA Evidence: Errors and Expectancies. Law and Human Behavior, 23(APRIL 1999), 159–184.

The Theory That Would Not Die: How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy. Sharon Bertsch McGrayne. Yale University Press.

Thompson, W. C., & Schumann, E. L. (1987). Interpretation of Statistical Evidence in Criminal Trials: The Prosecutor’s Fallacy and the Defense Attorney’s Fallacy. Law and Human Behavior, 11(3), 167–187. doi:10.2307/1393631

Tribe, L. H. (1971). Trial by Mathematics: Precision and Ritual in the Legal Process. Harvard Law Review, 84(6), 1329–1393. doi:10.2307/1339610


Read More

Is our feeling of ‘agency’ over an event inherently rewarding?

Today I would like to introduce a new section to this blog: ‘Incurably sceptical’. In this section I rifle through the recent cognitive psychology literature and pick out a paper which looks interesting to me based on the abstract alone. I then proceed to examine the authors’ aims, methodology, analysis and interpretation. Hopefully along the way we will not only learn a little about the topic of the paper, but, in appraising it with a critical eye, perhaps also derive some lessons about the scientific method. Maybe we will even have some fun … Importantly, these are not ‘bad’ papers. Indeed, unless papers I find interesting are more likely to be bad, they should be representative of the standard of papers being published in the main cognitive psychology journals at the moment.



This time we will be looking at a paper titled ‘I control therefore I do: Judgments of agency influence action selection’1. The paper aimed to investigate whether a person’s feeling of agency over an effect made them more likely to engage in the behaviour which produced that effect – in other words, the paper sought to determine if a feeling of agency is in itself rewarding.

There are purportedly several facets to a person’s ‘sense of agency’. One such facet we will focus on is the mental belief that one is the intentional source of an outcome. For example, if you decide to put a dirty mug in the dish washer, and you then do so, you might hold the belief that ‘I intentionally moved that mug’. In the words of Haggard & Tsakaris: “As we perform actions in our daily lives, we have a coherent experience of a seemingly simple fluent flow from our thoughts, to our body movements, to the effects produced in the world. I want to have something to eat, I go to the kitchen, I eat a piece of bread. We have a single experience of agency – of control over these events.”2

This experience of agency, not only of simple movements of a hand, but also of more complex outcomes in the world is a growing area of study in a wide range of disciplines. It has been noted as an important concept in moral responsibility in law2, hypothesized as a core component of one’s experience of consciousness3 and a lack of agency has been implicated as one potential factor leading to auditory hallucinations in schizophrenia4. Work has also shown that individuals and corporations considered ‘harmful’ are actually judged to possess less agency5. Given that agency is very closely linked to blame6, it therefore may also have ramifications for the apportionment of blame in the wake of social disasters such as the banking crisis (see Blame The Banks). We can also be tricked into illusions of agency, such as in the notorious ‘mirror-hand illusion’7.


Figure 1. Some beliefs about agency are perhaps more illusory than others …

The Paper

The present study sought to determine if this sense of agency over events, effects or outcomes, is itself rewarding – do we seek and enjoy this sense of agency or control as an end in itself?

To test this hypothesis, the researchers placed their participants in front of a computer screen and gave them four buttons to press. They were instructed to press one of the four buttons every time a red dot appeared on the screen. They were also instructed to “take care that the sequence of responses they generate will be as random as possible”, i.e. ‘try to press all the buttons equally often’. That was the entirety of their instructions (well actually there was the occasional blue triangle, but we’ll get into that later). Now if you are suspicious at this point that there must be something ‘more’ going on in this experiment, I don’t blame you, and I personally find it hard to believe that the participants would have been convinced that this was the entirety of the experiment. This can be a problem particularly if your participants figure out the real aim of the experiment and even worse if they figure out what your hypothesis is – they might intentionally try to prove it (common enough to be known as ‘demand characteristics’) or disprove it (not common enough to have its own name – I suppose it would take a real jerk to want to do this) – either of which completely undermine the vital experimental assumption of participant naivety.

Of course, the experiment wasn’t just studying how good people are at constructing random sequences (we’ve known for a long time that we suck at it8, if you’re interested). Participants actually found themselves unknowingly in one of three conditions. In each of the three conditions, with varying probability, each of the four buttons was set to cause an ‘effect’: sometimes, when they pressed the buttons, the little red dot would turn into a little white dot before promptly disappearing. In the first, ‘High Probability’ condition, all four buttons had a 90% chance of triggering this white-dot effect. In the ‘No effect’ condition, the effect could never happen – these poor chaps really were just pressing random buttons for no reason. Finally, in the ‘Key Specific’ condition, the four buttons varied in their likelihood of producing the white-dot effect (90%; 60%; 30%; 0%). The idea behind this method was to variably instil this ‘sense of agency’ in the participants – for them to feel to varying extents like they were ‘causing’ this white dot to appear. The researchers assumed (reasonably, I suppose) that if the participants found this sense of agency rewarding or pleasurable in some way, they would press the button that produced that effect more frequently. Perhaps causing a white dot to appear doesn’t sound particularly rewarding to you, but I guess that was the point – they don’t state this explicitly but the researchers may have wanted to eliminate the potential confound (alternative explanation of an effect) that the participants were pressing the buttons with a higher probability of producing the effect not for the sense of agency it provided, but instead for the sheer enjoyment of the stimulus it provided – if the buttons produced biscuits instead of white dots, for example, no one would hesitate to complain that the desire for biscuits was driving the button pressing, rather than the enjoyment for some abstract ‘sense of agency’.  However, I think I am not sure this issue is entirely dealt with. While a white dot is about as unstimulating an effect as I can imagine, I think we have to consider just how dull the existence of these poor button-mashing people was during the course of this experiment – this is perhaps one of the most boring experiments I have ever come across. The presence of a white dot instead of a red dot may well have seemed like nirvana itself. If we do find the hypothesized effect, perhaps the participants really just want to bring forth the holy white dot merely to revel in the brief glory of its existence and don’t care one wit whether they are the cause or not – perhaps the variety from the monotony of pressing four buttons for no reason was reward enough.


Figure 2. How I imagine this experiment.

Anyway, on to the results. We are firstly told in the results section that “To increase statistical power and the accuracy of parameter estimation, the following statistical analyses were conducted on the combined data from a small preliminary experiment (N = 29) which included only the Key Specific condition”. Personally, this rings alarm bells. They are not the alarm bells of outright fraud, but the slightly quieter but more insidious bells of ‘Researcher Degrees of Freedom’. Researcher degrees of freedom are the branching set of choices that experimenters are able to take throughout the entire process of designing, undertaking and analysing an experiment (e.g. figure 3, below) which may alter the likelihood of getting a ‘significant’ (ideally an indication that a result is not due to random fluctuations) result at the end. These include things like which of a number of designs to use, when to stop collecting participants, which statistical analysis to use, which outcome measures to focus on, and so on. With each of these decisions, the researcher will often know that one choice is more likely to lead to a significant result, and it takes a high level of commitment to scientific integrity to make a fully objective decision. In this paper, the researchers were at some point faced with a decision as to whether or not include these extra 29 participants from the preliminary study, and it is a very common decision experimenters face – I’ve faced it myself. Now, if the experimenters had absolutely no idea what the results were of either that preliminary sample or the main sample then there would be absolutely no issue in combining them – the authors would be entirely correct that it would simply boost statistical ‘power’ (more participants = better, in general). The problem comes when the experimenter knows whether the results from that preliminary study are in the direction of their hypothesis or not. We now know enough about human psychology and the subtle unconscious biases that influence our choices to know that on average, if one choice (e.g. including the data) makes a desired result more likely, and the other makes it less likely, then, all else being equal, the person is more likely to choose the former. I can tell you by personal experience that a researcher in this position will miraculously discover that a large number of perfectly sciencey-sounding reasons favouring including the data will spring to the person’s mind (such as ‘bigger sample sizes are better’, for example) – and if you think that scientists do not ‘want’ a particular result, that they are entirely objective, paper-producing automatons, who don’t care about furthering their career or producing highly-cited papers to get grant funding, well then you probably haven’t met many.

Figure 3. Researcher Degrees of Freedom: The worrying reality behind many significant findings in psychology?

Anyway, let us assume that no biases influenced the decision of whether to include that data. The researchers found that within the ‘Key Specific’ condition, where each key had a different chance of producing the white-dot effect, the buttons which produced it with greater frequency were pressed more often (actually, this effect was only significant for the 90% button, but hey, let’s move on). The researchers also found that participants in this ‘Key Specific’ condition were less ‘random’ in their key presses than in either of the other two conditions, where all the buttons had the same probability. This suggests that overall, the participants in this condition were drawn from their requested task of pressing the buttons randomly by the desire to produce the white dot effect. The researchers also found that reaction times in the ‘High Probability’ condition were greater than in the ‘No Effect’ condition. The conclusion again: people must be pressing faster because they are enjoying the sense of agency so much. The researchers also ruled out general engagement with the task as a ‘confound’ (alternative explanation of the effect). Remember that blue triangle tit bit I teased you with earlier? Well the researchers planted several trials throughout the experiment where these showed up instead of a red dot, and participants were told at the start that when this happened they should press the space bar, instead of one of the four normal buttons. This was supposed to be a measure of ‘engagement’ with the task. Apparently, by this measure, participants in the ‘High Probability’ condition were no more engaged than those in the ‘No Effect’ condition, but they still reacted faster.



So what can we draw from this experiment? Overall it seems that when people are able to cause an effect, even when explicitly asked to do something which runs counter to this (press the buttons randomly), they can’t seem to quite help themselves but press the buttons that cause that effect, and they are even quite good at becoming attuned to which buttons cause the effect more frequently. So are we all power-hungry maniacs who can’t even follow simple instructions when there is the temptation to cause some effect in the world (even one as mild as producing a white dot?) Well I think we should be careful before jumping to this conclusion. Firstly, there was little real incentive for people to pursue the randomness goal given them. Perhaps if the researchers had paid them according to how ‘random’ they were then this power-hungry idea might be more compelling. Further as I said before, there is good reason to think the participants might have second-guessed the researchers’ intentions with the white dots, and might have tried to achieve more white dots, suspecting this was the researcher’s true aim: the deception used in the task may not have been dastardly enough. Further, the effect was not very large – as mentioned, in the key specific condition, only the 90% button was actually pressed significantly more often – the 60% and 30% ones were no different to the 0% one (although this makes some sense: if you want the holy white dot, and you’ve figured out the frequencies, I guess you would just press the 90% one all the time, and ignore the 60% and 30% buttons equally as much as the 0% button). We also have the slight worry about the researcher degree of freedom involved in choosing to add the preliminary study’s data in to the present study – did the present data show a significant effect all on its own? If not, did the researchers know this before they added in the extra 29 people? Finally, even if all of these issues weren’t present, we still have the interpretation issue that the participants may have been pressing the button for the effect itself, for pure love of the white dot, instead of for the love of being the cause of the white dot. This is perhaps made less likely by the already discussed fact that the production of a white dot in itself shouldn’t be that rewarding, but it really can’t be ruled out on the present data. It is admittedly quite hard to think of an experiment where you could experimentally distinguish these two: the love of the white dot itself, and the love of causing the white dot – if you can think of a good way, you might have a paper on your hands.



1. Karsh, N., & Eitam, B. (2015). I control therefore I do: Judgments of agency influence action selection. Cognition, 138, 122–131. doi:10.1016/j.cognition.2015.02.002

2. Haggard, P., & Tsakiris, M. (2009). The Experience of Agency, 18(4), 242–246.

3. Schwabe, L., & Blanke, O. (2007). Cognitive neuroscience of ownership and agency. Consciousness and Cognition, 16(3), 661–666. doi:10.1016/j.concog.2007.07.007

4. Pareés, I., Brown, H., Nuruki, A., Adams, R. a, Davare, M., Bhatia, K. P., … Edwards, M. J. (2014). Loss of sensory attenuation in patients with functional (psychogenic) movement disorders. Brain : A Journal of Neurology, 137(Pt 11), 2916–21. doi:10.1093/brain/awu237

5. Khamitov, M., Rotman, J. D., & Piazza, J. (2016). Perceiving the agency of harmful agents: A test of dehumanization versus moral typecasting accounts. Cognition, 146, 33–47. doi:10.1016/j.cognition.2015.09.009

6. Shaver, K. (2012). The attribution of blame: Causality, responsibility, and blameworthiness. Springer Science & Business Media.

7. Tajima, D., Mizuno, T., Kume, Y., & Yoshida, T. (2015). The mirror illusion: does proprioceptive drift go hand in hand with sense of agency? Frontiers in Psychology, 6(February), 200. doi:10.3389/fpsyg.2015.00200

8. Wagenaar, W. A. (1972). Generation of random sequences by human subjects: A critical survey of literature. Psychological Bulletin77(1), 65.

Read More

The Right Way to use Evernote for Academic Literature Reviews

In this post I will be taking a rather major departure from the normal topics I cover to discuss something very dear to my heart: digital productivity! I will be presenting a system for digitally capturing, organizing and retrieving the key information from academic papers to ensure that you have the information you need exactly when you need it without tiresome searches.

I am in the 3rd year of my PhD now and I have spent far more time than I probably should have in developing digital systems to help my productivity. This has slowly led to a fairly sophisticated digital organization system, a part of which I want to share with you today. The main reason for this is that while I have stolen almost every digital productivity system I use from someone else, I actually developed this portion of it myself, with a lot of trial and error, and I think it finally works very well. Two final things to say are that I use Evernote on Windows but as far as I am aware everything I suggest here should translate over to mac (at least functionally if not button->button). Also I will be assuming you are already a user of Evernote so I won’t be introducing basic concepts – there are so many good guides elsewhere it isn’t even worth giving a link! – if you are new to the software and I suggest you read those first.

So, if you’re ready, let’s get going.

Firstly, you might be tempted to store your paper pdf’s in Evernote – don’t. Mendeley’s auto meta-data population (i.e. bringing in abstract, author, year, journal etc) and intelligent search function make it far more powerful for this.

No, Evernote is for your personal notes on those papers.

The rest of the post will be separated into three broad sections corresponding to the vital steps of any information processing system:

  1. Capturing
  2. Organizing
  3. Retrieval

There will also be some bonus material at the end in ‘Extra Tips’.


Step 1: Create New Tag / Notebook

So when you start taking notes on a new paper, make a new tag for that paper. You could also make a new notebook for each paper (I used to do this) but there is a 250 notebook limit which doesn’t take long to hit in some fields (Ps. If you are interested in using a tag-only system in Evernote see the final section ‘Extra Tips’). I recommend naming the tag / notebook by ‘Year: Authors’ e.g. ‘2015: Jim Bob & Frank’ (oh by the way if you use tags you can’t include commas between authors names – Evernote will get very upset if you try). The reason the year needs to come first is so that once you have multiple tags / notebooks for your various papers in a list they will be automatically sorted by year.

Step 2: Go through paper taking screenshots / notes

The next step is just to start transferring the important stuff from the paper you are reading into notes in Evernote. For speed, I highly recommend screenshotting. Especially for the graphs in a paper, you are going to be want to be taking screenshots, but I am not talking about those old-fashioned whole-screen print-screen things. Find out how to take screenshots of only a portion of the screen. On my computer it is shift + win + s but I think I may have set that up in the far distant past so you might need to look up how to do it on yours.

Step 3: Paste them into notes – single ‘idea’

Paste these screenshots into a note in your tag / notebook and give it a meaningful title. Don’t be tempted to put loads in a single note – try to keep each one to a single ‘concept’ or ‘idea’ – it will make it easier to understand what is in each at a glance later, and will ensure you don’t overlook things ‘further down’ the note. Feel free of course to add your own notes above or below summarizing what is going on.

Step 4: Don’t forget the annotation tools!

Don’t forget the awesome attachment ‘Skitch’ which you might have to download separately but which allows you to add really eye-catching annotations to the notes you bring in very easily.

Step 5: Screenshot text in too!

I personally screenshot text in as well as graphics. You can of course copy it in as text, but I find this makes a nice division for me between the notes I have brought in from the paper and the notes I then make on those notes.
Also on a lot of pdf’s when you copy text over some of it becomes corrupted or is formatted weirdly etc.
I tend to highlight important sections within the paper itself before screenshotting so that I have those highlights in Mendeley too if I ever want to flick through the paper there.


Now that we have got our stuff into Evernote it is time to organize it. These leads us to …

Step 6: Create (no ordinary) Table of Contents Note

and …

Step 7: Make it into a personalized summary note

So you make a table of contents note by selecting all your notes on that paper and clicking the ‘Table of Contents’ button that comes up. This creates a new note with green ‘links’ to all your other notes in it. But we won’t just leave it like that – oh no no. This will be our ‘summary’ note and it is where the real magic begins. Firstly title this note with exactly the same title as you gave the notebook / tag you used to group all your notes on this paper i.e. ‘Year: Authors’. Next at the top of the note give a brief description of the paper – like a personalized abstract – a few lines which will work with however your mind works to immediately remind you what paper this was from the dark recesses of your memory when you come across it again half a year down the line. Secondly add a little more detail to your ‘links’ to the other notes so that when you scan this ‘summary’ note it is clear what is in them. Now …

Step 8: Tag it with ‘PaperSummary’ (non-negotiable) and any other tags needed

Next, we tag this note only. Unless you have any particular reason, you don’t need to tag any of your other notes. Firstly this would take ages so you probably wouldn’t keep up the practise in the long run, and secondly all the ‘lesser notes’ are already linked in this note, so there is no need really.

Firstly we add the tag ‘PaperSummary’ or something similar. This is non-negotiable and you will see the benefit shortly. Then add other tags to describe the paper itself. When I first started this I added 10+ tags to each of my paper summaries, and soon found out that I was only actually using 2-3 of them. Think about the most important sub-categories of your academic life. For example, I am interested in the presentation of risk, and I am interested in which papers look at visual presentation vs verbal presentation so I always tag my summaries as ‘visual’ or ‘verbal’ – I am also interested in whether papers are about ‘Frequentist’ or ‘Bayesian’ statistics so I use those tags also. In some ways it is better to over-tag than under-tag, but also, if tagging becomes too much of a pain then there is a tendency to just stop doing it at all, and then your system really will fall apart. If you are really in a rush then you could add a ‘ToSort’ tag which you would need to review regularly to tag them up properly.


Ok so you have all your paper notes in Evernote, with summary notes which are tagged up. What did we do all that for?

Step 9: Filtering

Once you get beyond 10-20 papers organized in the way I have outlined above, you will really start to see the benefit. Now search for the tag ‘PaperSummary’ and all your summaries from all your papers will come up, in order of when they were published (because we put the Year at the start of the name of each one). Now depending on what you are looking for you can begin to sub-divide these by your descriptive tags – for example I might, as I said above be writing a paper about visual presentation, so I only want to look at those during my research – or perhaps I only want to look at those who looked at visual presentation of risk and at Bayesian statistics, or visual presentation and frequentist, or any other combination! Furthermore once you have found the papers you are interested in you have a personalized an organized summary with links to all the important information on that paper. I struggle to imagine an information retrieval system which could bring you what you want more quickly than this with current technology – if you know of one please tell me and I will put off my real work for another couple of weeks to implement it. This system is also helpful if I want to find a paper but can’t remember exactly who it was or the year – I probably know if it was, say visual or verbal risk presentation, and Bayesian or Frequentist statistics, so if I sub-divide by these then I have a much smaller list to scour through than if I were to search through all my papers one by one. You also of course have a powerful search function if you remember any snippets of text that might be in it. Further you don’t have to explore your papers through these summaries. You can just go to all your paper notebooks / tags (whichever one you chose) and search through them by year (or search for author names) to look at all your notes for that paper. If you click on notebooks/tags and search for a particular author’s name only also, it should bring up every paper they were author on, whether first, second, third, etc … lots of options!

Finally, another great thing you can do at this point is, if you are writing a literature review for a paper or an article you could make a new tag called ‘NewLitReview’ and then as you go through all your summary papers, filtering etc, and then tag the ones you think will be important to the write up with this tag so you can refer back to them anytime and have them all in one place. Once you are done with this review, just delete the tag!

If that isn’t enough to wet your appetite, wait until you see ..

Step 10: Areas Overviews!

So say you are writing a paper on visual presentation of risk, and you look at all your notes on this area and there are still loads – 30, say – way too many to get your head around simply by reading through one by one. How can Evernote help here? Summaries of your summaries, that’s how! Select all the papers you are interested in and make another Table of Contents note.

Step 11: Area Overviews continued …

So as you can see in this link I have a bunch of green titles: these are links to the paper summaries of each of those papers – just like the paper summaries above had links to the other notes about that specific paper. I then have a brief summary of each paper in this area written below the link for that paper – I tend to put them in chronological order (which should happen by default if your list of papers was already in year order, which it should be if you followed the steps above) – and this allows me to really easily scan through and get an intuitive feel for the area very quickly – sometimes if possible I just put the main graph from each paper and one sentence like ‘No effect’ ‘Mild Effect’ etc …. This is often the first stage in preparing to write a literature review (or part of one) for me. At the top of this note it might be nice to write a brief summary of the area as a whole also e.g. ‘Mixed findings, some found a difference between the groups, some didn’t’. Now this is as high as I have ever gone in terms of ‘Summaries of summaries’ but in theory, depending on your area it might be helpful to keep going, so I might have another even-higher-level summary note with links to the ‘Visual’ and ‘Non-Visual’ areas of my field, to compare them perhaps, or compare the ‘Frequentist’ and ‘Bayesian’ areas…. In theory you could have one master summary note just called ‘Statistics’ which would define your entire field and then have links to these four areas, which would have links to …. You get the idea.

Ok I hope all of this has been helpful to you. If you’re thinking ‘But what about the actual write up?’ check out my companion post here

If you still aren’t satisfied, read on for some extra tips and tricks.

Some Final Tips

When you don’t have time…

Firstly, if I ever come across a paper but I am in way too much of a rush to go through the whole process above, I just make a tag / notebook for the paper, make a blank summary note immediately (you can copy the note links in later – you don’t have to do it by making a table of contents note), tag it as ‘PaperSummary’ and ‘ToRead’ and then just screenshot the Title and Abstract over. Done. Once you are quick this takes maybe 30 seconds and if you stick religiously to the rule that you will always do at least this for any relevant paper you come across, it ensures no papers get lost or forgotten about. I guess if you are out and about with just your phone you could even just make the summary note with the year, author, PaperSummary and ToRead tags, and that would suffice to remind you in the future to look into the paper – but I don’t really come across many papers when I am out and about to be honest …

Reference Notes

Another thing which can be handy is to create a note for every paper and copy over the text from the reference section from that paper. Tag all of these notes ‘References’ or whatever. Then you can search within just these notes for a given paper to easily see who has cited them.

Finally, if you go the full-tag route … [serious nerdiness alert]

So I am one of those nutty Evernote users who only has three notebooks. One for Inbox i.e. if I don’t have time to sort a note right now, it just goes in inbox and I sort it later. I also have one main notebook (Cabinet) where almost everything eventually goes. The entirety of my organization beyond this point is achieved through tags.

Tags are in theory better than notebooks because notes can belong to multiple tags but only one notebook, and everyone knows the anxiety created by grey areas between two hard-categories (e.g. is this bioengineering note really about Biology or Technology? – well both dangit!). Tags are soft categories – no anxiety needed, just tag them as both! Also you can only have 250 notebooks max, which didn’t take me that long to hit when making a new one for every paper I read.

Anyway, if you do want to go the full tag route, I recommend a system broadly like these:

You don’t have to do it exactly the same but the key point to take away is to use punctuation marks to create different ‘types’ of tags. I use the ! sign for tags relating to areas of my life (e.g. !Personal, !PhD, etc.), the @ sign to refer to tags relating to areas of knowledge (e.g. @Psychology, @Biology) and the . (full stop sign) to refer to tags relating to the type of note it is (e.g. .Article, .Paper, .Idea). I also use the $ sign for tags regarding important people (e.g. $JosephThomas). This system is helpful when you are both tagging your notes and also when searching your massive ‘Cabinet’. For instance if you click on Cabinet then click on tags, and type in ! it will bring up a drop down list of all your ‘Areas of Life’ because they are the only tags that begin with ‘!’. Then once you have chosen an area of life, e.g. ‘!PhD’, you could type in the @ sign, and it will bring another drop down list of all the areas of knowledge within PhD (it will only show tags with at least 1 note within your current search area), and so on … So this system stops you feeling like you are just lost in a sea of a hundred tags and allows you to feel like you understand the ‘structure’ of your note system a little better.

A system like this is also far more minimalistic / clean than a full-notebook system as you can have ‘.Idea’ notes about ‘@Biology’ or .Idea notes about your ‘!Writing’ or even about ‘$JosephThomas’. With a notebook system you would have to have a sub-notebook within the Biology notebook for ideas on that, another one within Writing for ideas on that, another one within JosephThomas … once you give in to the full-tag system, this kind of messy organization will send a shiver down your spine.


[06/05/16 Note: I believe the below has now been incorporated natively into Evernote version 6.0 so you shouldn’t have to go through the hassle of changing regedit files yourself – I’ll leave it in place a while just in case it’s helpful for someone]

However ..

However, if you do go the full tag route, there is one more really important thing you will have to do. What drove me nuts when I first did this, was that if I clicked on a tag, then made a new note, the new note would not be tagged with the tag that I was ‘in’. This is different to notebooks. If you are currently ‘in’ a notebook and make a new note, it will be stored in that notebook. This lack of function with tags led to waay too many of my notes being lost in the ether due to me forgetting to tag them with the current tag. However, there is a fix. It requires a little fiddling around in the windows systems – but don’t worry it isn’t too hard.

Firstly close Evernote. Now to go to the ‘Run’ application (you should be able to just search for it from the start menu) then type in ‘regedit’ and click ok. When the window pops up go to HKey_CURRENT_USER/Software/Evernote/ Now click on the Evernote folder and scroll down the list of … whatever they are … until you get to the one called ‘SetNewNoteTags’. Double click on this and change the ‘0’ value to ‘1’. Don’t worry, you won’t break Evernote!

NOW, click on a tag, any tag. Make a new note. Fill in the title of the note and press enter. Magically, the current tag will be assigned to the note. AHHHH. No more stressful losses of notes. So when we make a tag for a paper, and click on that tag and start making notes on the paper, they will all automatically be assigned to that tag – goodbye notebooks!

Read More

The Cognitive Psychology of Moral Discrepancy  


During the Second World War, the French existentialist philosopher Jean Paul Sartre, under the occupation of the German army, co-founded the underground resistance group Socialisme et Liberte. He also contributed to several illegal newspapers and magazines, writing articles in opposition to the invaders. However, he also famously accepted a lecturing position which had been taken from a Jew following the ban of their teaching in the country by the Nazis.

Jean Paul Sartre
Jean Paul Sartre

Further, after
approaching several people about joining Socialisme et Liberte and meeting indecision and uncertainty, the group shortly dissolved, and Sartre took no further part in active resistance. Sartre’s philosophy espoused the value of freedom and the moral duty of human beings, and given this, there has been much debate regarding whether Sartre’s actions, or lack of action, during this period were consistent with his professed beliefs.

The study of this relationship between ‘espoused’ moral values and actual behaviour has a long history and also continues to this day. Espoused Theory (developed principally by Chris Argyris e.g.  Argyris and Schön [1974]) states that:

When someone is asked how he would behave under certain circumstances, the answer he usually gives is his espoused theory of action for that situation. This is the theory of action to which he gives allegiance, and which, upon request, he communicates to others. However, the theory that actually governs his actions is this theory-in-use [actual behaviour]. (Argyris and Schön 1974: 6-7)

Jonathan Haidt (2001) goes further: espoused moral beliefs and actual behaviours are governed by completely different mental systems. In Haidt’s “Social Intuitist Model” (see paper: ‘The Emotional Dog and its Rational Tail’) the vast majority of real moral judgments / behaviours in-the-moment are made by one’s intuitive reaction to the situation, rather than through step-by-step reasoning. Reasoning, Haidt states, is generally only used in order to make after-the-fact justifications for moral decisions that have already been made intuitively or indeed to explain one’s moral beliefs to others in a theoretical context.

So what is the reason for this discrepancy? Both Espoused Theory and the Social Intuitist Model provide little explanatory theory outside of the proposal that the two phenomena are governed by different ‘theories’ or ‘mental systems’. Why do these systems behave differently? One possibility comes from research on cognitive biases. Firstly, check out the two versions of the ‘disease’ problem below:


Disease Problem: V1

Imagine you are in charge of the health department for a country experiencing a national disease outbreak. You have quarantined all the affected cases, 600 people in total. Your advisor presents you with the only two treatments available. You are told that treatment A will definitely save 200 lives, while treatment B has a 33% chance of saving all 600, but a 66% possibility of saving no one.

Which treatment option do you choose?



Disease Problem: V2

The situation is the same; 600 people quarantined. However in regards to the treatments, you are now told that treatment A will definitely kill 400 people, while Treatment B has a 33% chance that no people will die, but a 66% chance that all 600 people will die.

Now which treatment do you choose?


While the decision to be made in each version of these two problems is precisely equal, it has been consistently shown that the majority of people opt for treatment A in the ‘lives saved’ framing version (V1) but the same majority opt for treatment B in the ‘deaths’ framing version (V2). This effect has been found in many other experiments with related problems and the general consensus is that when faced with ‘gains’ people tend to choose the safe / certain option, while when faced with ‘losses’ people tend to choose the risky option – even when the two decisions are precisely equal.

This insight, known as ‘Loss Aversion’ led to Tversky and Kahneman’s 1979 ‘Prospect Theory’, a cornerstone of modern behavioural economics. In their 1974 paper (‘Judgment under Uncertainty: Heuristics and Biases’) they

System 1 vs System 2 [Illustration by David Plunkert, via The New York Times]
System 1 vs System 2 [Illustration by David Plunkert, via The New York Times]
proposed that people are susceptible to a wide variety of other cognitive biases also (including ‘anchoring‘, the ‘base rate fallacy‘, the ‘conjunction fallacy‘ and many others). Further, in Kahneman’s best-selling 2011 book ‘Thinking Fast and Slow’ he lays out

his belief that these biases are inherently due to the design of the mental ‘System 1’ (Haidt’s ‘Intuitive’ system) and can be overcome by greater use and education of the mental ‘System 2’ (Haidt’s ‘Reasoning’ system). In Kahneman’s model, both systems have their virtues and vices: System 1 makes decisions quickly and can handle a large amount of complexity, but it makes mistakes. System 2 is slower but more methodical and so makes less mistakes. In the moment, System 2 will often be too slow to determine how to behave so we rely predominantly on System 1.

So, perhaps we have the best intentions but are simply incapable of carrying them out in the moment due to the cognitive limitations of System 1?


An Experimental Test

In a recent paper, Schwitzgebel and Cushman (2015) wanted to test whether the degree of theoretical knowledge of moral situations would affect this in-the-moment decision making. To examine this, the authors decided to compare philosophers (people with philosophy degrees) to “similarly-educated” non-philosophers on the two disease problems. They also took data on the level of expertise in philosophy as well as whether ‘ethics’ was their area of speciality.




The study firstly replicated previous results, with a large majority of participants choosing the risky option when faced with ‘deaths’, and far less choosing the risky option when faced with ‘lives saved’. Furthermore, the effect size was the same for non-philosophers (83% vs 43%) and for philosophers (79% vs 32%) and no difference was seen even for philosophers with specialization in ethics.

Another Approach

This all fits with Espoused Theory, the Social Intuitist Model and the Cognitive Biases approach. Philosophers are trained to deal with ethical problems slowly and precisely (using ‘System 2’ in Kahneman’s language), but when faced with problems like the disease scenarios, their System 1 is just as vulnerable to the framing effect as anyone else.

But can this approach explain all moral discrepancy? Does it even explain the story we began with? Can Sartre’s actions during the war really be put down to cognitive biases and framing effects? He certainly would have had time to consider whether to disband his resistance group as well as whether to take the lecturing post. Can we really class these as in-the-moment, intuitive decisions? Professor Schwitzgebel (of Schwitzgebel and Cushman) has another theory. He has spent a large amount of his life’s work conducting empirical studies on the moral behaviours of Professors of Ethics in particular to determine if they are any kinder, fairer or more moral than other people.

angel_devilOver the years Professor Schwitzgebel and colleagues have looked at a vast range of behaviours including donating to charity, responding to student emails, organ and blood donation, frequency they call their mothers, eating meat, theft of library books, etc etc. The overall finding? No difference. Professors of philosophy studying ethics were no worse or better on these range of behaviours than other people.

Further, especially in regards to eating meat and giving to charity, the ethics professors were significantly different to other groups in their espoused belief about how morally bad eating meat was (they thought it was worse) and how much of one’s salary should be given to charity (they thought it should be more). But when it came to actual
behaviour? No difference.

So why the discrepancy here? The cognitive biases approach doesn’t seem any more relevant here than in Sartre’s case – there are no clear ‘framing’ effects, and people have all the time they need to make these decisions. From all his studies and interviews Professor Schwitzgebel believes one fact clearly shines through: morally, he says, people just want to be about as good as the other people around them. Studying ethics will change your idea of what an ‘ideal person’ is – but it won’t change your desire to be that ideal person – you will still just aim to be about average and no amount of theoretical expertise will change this fact. So it seems that even when we have time to employ our ‘System 2’ and really think about our behaviour, ‘good enough’ is good enough and we shouldn’t expect those with a large amount of training in ethical philosophy or even those who profess these beliefs, like Sartre, to stand by them in practise. Schwitzgebel calls this ‘Cheeseburger Ethics’ and you can find out why by reading his excellent post here:

God Speed!


Haidt, J. (2001). The Emotional Dog and Its Rational Tail: A Social Intuitionist Approach to Moral Judgment. Psychological Review, 108(4), 814–834. doi:10.1037//0033-295X.

Kahneman, D. (2011). Thinking, fast and slow. Macmillan.

Kahneman, D., & Tversky, A. (1979). Prospect theory: an analysis of decision under risk. Econometrica, 47(2), 263–292. Retrieved from

Meyer, M. W., Argyris, C., & Schon, D. a. (1976). Theory in Practice: Increasing Professional Effectiveness. Contemporary Sociology (Vol. 5). doi:10.2307/2062989

Schwitzgebel, E., & Cushman, F. (2015). Philosophers’ biased judgments persist despite training, expertise and reflection. Cognition, 141, 127–137. doi:10.1016/j.cognition.2015.04.015

Tversky, A, & Kahneman, D. (1974). Judgment under Uncertainty: Heuristics and Biases. Science (New York, N.Y.), 185(4157), 1124–1131. doi:10.1126/science.185.4157.1124






Read More

‘That Facebook Study’

Let’s do a test of selective memory. Do you remember ‘that Facebook study’ from last year? It was really creepy and ethically dubious right? Do you remember what the study was actually about? No?


Gustav Le Bon

Well let me tell you. It was about ‘emotional contagion’. This is the theory that the spread of emotions in a social network (on or off-line) is essentially replicative, like the spread of a virus. This ‘epidemiological’ approach can be traced back to Gustav Le Bon’s 1896 work ‘Psychologie des Foules’  or ‘The Psychology of Crowds’. Le Bon’s work was motivated by the French elite who were becoming increasingly afraid of emotional contagion in rioting masses and its potential effects on social order. Le Bon believed that the spread of emotions in crowds could be seen like the spread of germs and that this effect deprived them of their capacity to act individually and rationally.


The emotional spread in the modern contagion model (see Hatfield et al, 1994) is thought to occur not directly, but through two steps:

The eighteenth century ruling french elite feared emotional contagion in rioting mobs.

1. The observer mimics the behaviour of the individual experiencing the emotion (not necessarily in its entirety, but e.g. by tensing one’s stomach in response to fear, screwing up one’s face in response to disgust etc.)

2. The mimicked behaviour causes the observer to experience the same emotion.

If your brain is immediately coming up with counter examples to this model, don’t worry, you aren’t alone (see the recent paper by Dezecache et al. [2015] for a discussion of its limitations). The model holds fairly well for things like disgust and fear or anger at an out-group (like the ruling French elite), but what about interpersonal emotions like envy? This doesn’t usually trigger envy in an onlooker, and certainly not in the person being envied. So, it seems fair to say:


Ben Goldacre’s 2014 book ‘I think you’ll find it’s a bit more complicated than that’

Moving onto the Facebook study itself, what were they actually trying to do? The study was titled ‘Experimental Evidence of Massive-Scale Emotional Contagion through Social Networks’. So they were trying to test this emotional contagion theory on positives emotions in the biggest data set ever, with all the power of technology at their fingertips. They classified posts as ’emotionally positive’ if they had at least one positive word in them and no negative, (I don’t blame your brain if it is again coming up with issues with this method, but it is at least in accordance with previous work). They then reduced the amount of these positive posts in some people’s news feeds and not others, selected randomly (you remembered that bit didn’t you?).


So what did they find? By reducing the frequency of these ‘positive’ posts in a person’s news feed they were able to decrease the amount of positive posts that person then produced themselves. This was good enough to prove the emotional contagion theory for these authors. To quote the paper: “The results show emotional contagion.” (Kramer, Guillory & Hancock, 2014).


But hold on there Core Data Science Team, Facebook, Inc., are you entirely sure you have thoroughly examined your reasoning process here? In psychology we like to think about ‘confounds’ when interpreting our findings. These are things which explain a finding other than what you are claiming is the explanation. So, is there anything that could explain this change other than ‘emotional contagion’? Well, I can think of a few. What if the original ‘positive’ post provides information of some event which affects the person who sees the post e.g. “Oh my God I am so Happy ­-Insert Popular Band- is coming to town!” which would make someone in the same town who likes said popular band but didn’t know they were coming more happy and more likely to post positive things, possibly about the same band. Or what if the post directly mentions other individuals e.g.“I can’t wait to see Jim, Bob and Frank this weekend, I hope you are ready for me, it’s going to be great fun!!!”. It would probably make Jim, Bob or indeed Frank quite happy to know their friend was looking forward to coming to see them.


In both of these cases positive emotions are spreading through the social network, but it has nothing to do with behavioural mimicry and isn’t behaving like the spread of germs. It is spreading through revelation of a mutually happy event in the first example, and social bonding in the second. So while the Facebook study was able to show that positive emotions spread, it really wasn’t able to say anything about why and unfortunately emotional contagion is a ‘why’ theory. This is a good example of one of the problems with big data experiments. No data set in the world can make up for a flawed experimental method (not even if it’s REALLY BIG). And methodology tends to get more sloppy with really big samples. Here is a link to a nice article by Tim Harford, of BBC R4’s ‘More or Less’ on this topic.


Now for our second test of selective memory. If you were one of the people who did actually remember what the study was about, do you remember what was the difference between the groups who got the ‘less happy news feed’ and those in the control group? What would your guess be? 10% more positive posts? 20%? You can see the answer in the graph below from their paper. Looks pretty impressive huh? Now look at the scale. Yep. 0.1%. When positives posts were reduced in the person’s news feed, they produced, on average, 0.1% less positive words in their own posts. In the Psychology field we call this … a very, very, very small difference.

Graph reproduced from Kramer, Guillory & Hancock (2014)

So, the next time someone asks you what you think about ‘that Facebook study’ you can reply “Yea, that was so dodgy! Emotional Contagion is an overly simplistic model, their method was confounded and anyway they only found a 0.1% change in positive post frequency”.



Dezecache, G., Jacob, P., & Grèzes, J. (2015). Emotional contagion: its scope and limits. Trends in Cognitive Sciences, (APRIL). doi:10.1016/j.tics.2015.03.011

Goldacre, B. (2014). I think you’ll find it’s a bit more complicated than that. Harper Collins.

Hatfield, E. et al. (1994). Emotional Contagion. Cambridge University Press.

Kramer, D. I., Guillory, J. E. & Hancock, J. T. (2014). Experimental evidence of massive scale emotional contagion through social networks. Proceedings of the National Academy of Sciences of the United States of America, 111(29), 1073. doi:10.1073/pnas.1412469111

Le Bon, G. (1896). Psychologie des Foules, Macmillan.

Read More

Chess and The Clash of Civilizations

Several years ago I read The Player of Games by the late Iain Banks. In the far distant future, Jernau Morat Gurgeh, one of the greatest game players in his galactic civilization ‘The Culture’ is invited to travel to the distant and rival ‘Empire of Azad’ to play the most complex game ever created. The Azad use the game as the principal method of determining who their next emperor will be, and hold regular tournaments for this purpose. PlayerofGamesIt is to one of these tournaments that Gurgeh is invited, and is expected to be defeated easily. However as the story unfolds it becomes clear that Gurgeh’s vastly different cultural background to all the Azad players is deeply apparent in his approach to the game. This makes him unpredictable to them and gives him an edge in his play. As Gurgeh faces the reigning emperor in the final match, so much so do their cultures influence their styles of play that the game becomes a proxy for the war between The Culture and the Azad Empire.


So when I recently came across a paper entitled ‘Civilization differences in Chess Experts’ by Chassy and Gobet (2015) my mind immediately recalled The Player of Games and I couldn’t help but read on.


Chassy and Gobet examined the first move made by chess experts from across the globe. By far the two most frequent in expert play are e4 (king’s pawn forward two spaces) and d4 (queen’s pawn forward two spaces). This is because an important initial principle in chess is to control the centre of the board with your pieces. On, these two moves are played in around 80% of all games. Therefore all other first moves were lumped together in a third category.


As it transpires, taking into account wins, draws and losses, e4 is a ‘riskier’ move than d4 at this expert level of chess. With d4 the game is slightly more likely to end in a draw, whereas with e4 one has a slightly greater chance of both victory and loss. Importantly, neither is clearly ‘better’ than the other, e4 is just slightly riskier and d4 slightly more conservative. The third category (all other moves lumped together) was intermediary, being slightly less risky than e4 but slightly more risky than d4.


The authors then divided the world up according to Huntington’s (1996) classic text The Clash of Civilizations and the Remaking of World Order. Huntington thought that in the post-cold war world, the primary source of global conflict would be people’s cultural and religious differences rather than specifically territorial boundaries (what a fool…). He divided the world up based on these cultural / religious differences. These included: Western, Orthodox (Russia and the eastern bloc), Islamic, African, Latin American, Sinic (Chinese and neighbouring countries), Hindu, Buddhist and Japanese. In the current paper, ‘Japanese’ had to be removed by the authors as they didn’t have enough chess games to analyse and ‘Jewish’ was added.

WorldMapChess is played globally and further, has the same rules all over the world and has a single global rating system (the elo rating system). The authors were therefore able to extract data from games played between experts across the globe and compare them meaningfully. The results can be seen below.


Cold Hard Boring Reality

As much as I have enjoyed conflating real world research with fiction in this article so far, seeing the actual data forces me to put my researcher hat back on (it’s a very serious hat – no frilly bits at all).


Firstly, I actually have a fairly serious issue with this paper. It is assumed that e4 is chosen on average slightly higher by some cultures ‘because’ it is a risky move and further that this choice of risky move somehow says something about that culture. The paper says that “the level of risk-taking varies significantly across cultures” and “we discuss which psychological factors might underpin these civilization differences”. But this is only valid if players know that e4 is riskier and if they play it because it is riskier. But we really can’t be sure that that is why the move is chosen. Different cultures may have different opening strategy habits, passed down from teacher to student for generations which have more to do with tradition than optimization or risk. Or there might be another reason for the differences. We really can’t be sure so we can’t confidently take the leap of inference from move choice to general approach to risk.


The second thing to say is that these differences are very small and they are averages. If you pit two players from America and Russia against each other you aren’t going to see a Rocky IV style clash of cultures, let alone anything on the scale of the Gurgeh-Azad game. In fact if they are representative of the Western and Orthodox regions as a whole the most probable outcome on this data is that they would both play d4 when they are white. You also won’t find any validation for classic stereotypes in this data.


And that’s the real bucket of cold water. We humans are all just too boringly similar to each other. I suppose we will have to wait until we meet a neighbouring chess-playing galactic empire before we can get some really interesting data.



Chassy, P., & Gobet, F. (2015). Risk taking in adversarial situations: Civilization differences in chess experts. Cognition, 141, 36-40.
Huntington, S. P. (1996). The class of civilizations and the remaking of world order. Penguin Books India.



Read More

David Marr, Cognitive Science and the Middle Path

David Marr published Vision in 1982, and the work continues to influence research in cognitive science today. So much so in fact that Topics in Cognitive Science has published a special edition ‘Thirty Years after Marr’s Vision’ including articles on the applications and relevance of Marr’s work to the modern cognitive scientist.


The Tri-Level Hypothesis

David Marr (1982) proposed in his ‘Tri-Level Hypothesis’ that when seeking to explain the behaviour of any information-processing system such as the brain or its parts, there are three distinct levels of analysis, each of which must be understood: the computational, the algorithmic and the implementation levels. The computational level represents what the system does (the problems it solves e.g. producing colour vision). The algorithmic level includes the representations and processes used to solve these problems and the MarrTriLevelimplementation level is the way in which the system is physically implemented (e.g. in the brain, the specific wiring and connections between neurons in the system). The Tri-Level hypothesis has been reformulated several times in the subsequent 30 years (e.g. Anderson, 1990; Newell, 1982; Pylyshyn, 1984) and remains a core tenet of cognitive science.


Reductionism and Vagueness

In an introduction to Topics in Cognitive Science’s special issue, Peebles and Cooper (2015) contend that the middle level, the algorithmic, is too often being ignored in modern times, with ‘reductionist neuroscience approaches’ focusing entirely at the implementation level and vague ‘Bayesian approaches’ focusing overly at the computational level. While this latter approach may indeed succeed in solving a problem which the brain solves, little or nothing is learned of how the brain itself actually solves the problem. Noting Marr’s insistence on the necessity of understanding at all three levels, the authors therefore urge greater focus on theories of cognitive architecture, which operate at the middle algorithmic level and decompose and explain the system through the interaction of functional cognitive components.


The Encroachment of Neuroscience

However, Bickle (2015), in the same issue, argues against this view. Peebles and Cooper’s attack, particularly on reductionist neuroscience, echoes Marr’s original attack on the inability of reductionists of his time to explain vision using electrophysiological cellular readings (e.g. Barlow, 1972). Bickle argues that while Marr’s original attack on reductionism was justified, it (and by extension, Peebles and Cooper’s) is no longer tenable. A swathe of new techniques and tools such as cortical microstimulation have allowed neuroscientists to begin constructing causal-mechanistic explanations of the brain including the dynamic interaction of parts and their organization as well as explanations of how these interactions ‘solve the problem’ of interest. While reductionist approaches in Marr’s time were merely descriptive (and clearly operated only at the implementation level) modern neuroscience theories are therefore genuinely explanatory and appear to encroach on the algorithmic level. A causal-mechanistic neuroscientific explanation of a system is indeed different from the kind of explanation given by cognitive science and advocated by Peebles and Cooper, but is not clearly inferior, Bickle contends. Further, the interaction between, or equivalence of, these ‘higher level’ neuroscientific explanations of the brain system and the more traditional cognitive explanations at the algorithmic level, is not fully understood and will need further work. Marr did MarrQuotenot anticipate this encroachment of neuroscience on the algorithmic level, Bickle states, and it is not clear what he would have made of it.


Synergistic Working

In the same issue, Love (2015) argues for greater cooperation between those working at different levels and proposes that findings at one level might be used to test theories at another. In fact there is already at least one good example of this synergistic working between neuroscience and cognitive psychology, and can be found in Smith, Kosslyn and Barselou (2007, p16). It began before and ended after, Marr’s 1982 work, and was in fact also within his own field of vision. In the 1970s there were two competing theories of how mental images (e.g. imagining a square) were represented in the mind. Pylyshyn (1973) claimed they were represented conceptually, similar to language (a mental image of a square would be represented simply as the concept ‘square’). However Kosslyn and Pomerantz (1977) believed that such images were actually ‘depicted’ in the mind, geometrically mapping point for point with a real image (a mental image of a square would literally be represented by four joined equal-length lines with right angles between them). For over a decade this debate continued with neither side able to disprove the other on cognitive evidence alone. However in the late 90s’ advances in neuroscience allowed careful examination of the area of the brain underpinning mental imagery. They were found to be represented in the primary visual cortex of the brain ‘topologically’. Mental imagery literally produced a ‘picture’ of activation on the surface of the cortex, which, while it would not be recognizable to a naïve viewer as the original image, corresponded to the size and orientation of the imagined image (see figure 2, below), and which mapped one to one with the mental experience of imagining the object (Klein et al, 2004; Kosslyn et al, 1995; Kosslyn & Thompson, 2003). This provided strong evidence for the ‘Depiction’ theory, and demonstrated the potential value that multi-level working could provide.


Figure 2. ‘A picture on the brain’: two sets of fMRI images from Klein et al (2004) demonstrating (a) the two stimuli used (b) the unique cortical activation for the horizontal image and (c) the unique cortical activation for the vertical image. Activation is shown for both direct perception of the images and for subsequent ‘mental imagery’.



It has been over 30 years since the publication of David Marr’s Vision and the work still remains central to cognitive science. In this time the work has been revised and the clear distinction between the three levels apparent in Marr’s time has become somewhat blurred by the encroachment of neuroscience on the algorithmic level previously monopolised by cognitive science. Further, in developments that would have surely pleased Marr, synergistic working between levels has produced advancements in understanding of the function of the brain. Finally, the Tri-Level Hypothesis still shows the capacity to provoke debate, even within a single publication, and it is perhaps this capacity which will ensure its centrality is maintained for the next thirty years.



Anderson, J. R. (1990). The adaptive character of thought. Hillsdale, NJ: Lawrence Erlbaum Associates.

Barlow, H. B. (1972). Single units and sensation: A neuron doctrine for perceptual psychology. Perception, 1, 371–394.

Bickle, J. (2015), Marr and Reductionism. Topics in Cognitive Science, 7: 299–311. doi: 10.1111/tops.12134

Klein, I., Dubois, J., Mangin, J. F., Kherif, F., Flandin, G., Poline, J. B., … & Le Bihan, D. (2004). Retinotopic organization of visual mental images as revealed by functional magnetic resonance imaging. Cognitive Brain Research, 22(1), 26-31.

Kosslyn, S. M., & Thompson, W. L. (2003). When is early visual cortex activated during visual mental imagery?. Psychological bulletin, 129(5), 723.

Kosslyn, S. M., Thompson, W. L., Kim, I. J., & Alpert, N. M. (1995). Topographical representations of mental images in primary visual cortex. Nature, 378(6556), 496-498.

Kosslyn, S. M., & Pomerantz, J. R. (1977). Imagery, propositions, and the form of internal representations. Cognitive Psychology, 9(1), 52-76.

Love, B. C. (2015), The Algorithmic Level Is the Bridge Between Computation and Brain. Topics in Cognitive Science, 7: 230–242. doi: 10.1111/tops.12131

Marr, D. (1982). Vision: A computational investigation into the human representation and processing of visual information. New York, NY: Henry Holt and Co.

Newell, A. (1982). The knowledge level. Artificial Intelligence , 18(1), 87–127.

Peebles, D. and Cooper, R. P. (2015), Thirty Years After Marr’s Vision: Levels of Analysis in Cognitive Science. Topics in Cognitive Science, 7: 187–190. doi: 10.1111/tops.12137

Pylyshyn, Z. W. (1973). What the mind’s eye tells the mind’s brain: A critique of mental imagery. Psychological bulletin, 80(1), 1.

Pylyshyn, Z. W. (1984). Computation and cognition: Toward and foundation for cognitive science. Cambridge, MA: MIT Press.

Smith, E. E., Kosslyn, S. M., & Barsalou, L. W. (2007). Cognitive psychology: Mind and brain. Upper Saddle River, NJ: Pearson Prentice Hall.















Read More