Cognitive reflection and cognitive reflection-like items.
Below are a list of items from the academic literature.
Original CRT (3 – item Frederick, 2005)
- A bat and a ball cost £1.10 in total. The bat costs a dollar more than the ball. How much does the ball cost? (Intuitive answer 10 pence; correct answer 5 pence).
- If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets? (Intuitive answer 100 minutes; correct answer 5 minutes).
- In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half the lake? (Intuitive answer 24 days; correct answer 47 days).
4 –items from Toplak (2014).
- If John can drink one barrel of water in 6 days, and Mary can drink one barrel of water in 12 days, how long would it take them to drink one barrel of water together? (Intuitive answer 9; correct answer 4).
- A man buys a pig for £60, sells it for £70, buys it back for £80, and sells it finally for £90. How much has he made? (Intuitive answer £10; correct answer £20).
- Simon decided to invest £8,000 in the stock market one day early in 2008. Six months after he invested, on July 17, the stocks he had purchased were down 50%. Fortunately for Simon, from July 17 to October 17, the stocks he had purchased went up 75%. At this point, Simon has: a. broken even in the stock market. b. is ahead of where he began. c. has lost money. (Intuitive answer b; correct answer c value is £7000).
- Jerry received both the 15th highest and the 15th lowest mark in the class. How many students are in the class? (Intuitive answer 30; correct answer 29).
2016 CRT (Thomson & Oppenheimer, 2016)
- If you’re running a race and you pass the person in second place, what place are you in? (Intuitive answer 1st; correct answer 2nd).
- A farmer had 15 sheep and all but 8 died. How many are left? (Intuitive answer 7; correct answer 8).
- Emily’s father had three daughters. The first two are named April and May. What is the third daughter’s name? (Intuitive answer June; correct answer Emily).
- How many cubic feet of dirt are there in a hole that 3’ deep x 3’ wide x 3’ long? (Intuitive answer 27; correct answer none).
Decoy / control items.
- A cargo hold ship had 500 crates of oranges. At the ship’s first stop, 100 crates were unloaded. At the second stop, 200 more were unloaded. How many crates or oranges were left after the second stop? (answer 200 crates)
- Sara, Emma, and Sophia embark on a river trip. Each of them brings one supply item for the trip: a kayak, a cooler of sandwiches, and a bag of apples. Sara brought the apples and Emma didn’t bring anything edible. What did Sophia bring? (answer cooler of sandwiches)
- An expedition on a mountain climbing trip was traveling with eleven horse packs. Each horse can carry only three packs. How many horses does the expedition need?(answer 4 horses)
- A mechanic shop had five silver cars, two red cars, and one blue car in the garage. During the day, three silver cars and one red car were picked up, and one black car was dropped off. How many silver cars were in the garage at the end of the day?(answer two silver cars)
2-items used for boosting in Shtulman & McCallum (2014).
- A house contains a living room and a den that are perfectly square. The living room has 4 times the square footage of the den. If the walls in the den are 10 feet long, how long are the walls in the living room? (Intuitive answer 40; correct answer 20)
- A store owner reduced the price of a $100 pair of shoes by another 10 percent. How much do the shoes cost now? (Intuitive answer 80; correct answer 81)
Baron et al., (2015)
Arithmetic items with lures.
- If it takes 2 nurses 2 minutes to measure the blood pressure of 2 patients, how long would it take 200 nurses to measure the blood pressure of 200 patients?
- Soup and salad cost $5.50 in total. The soup costs a dollar more than the salad. How much does the salad cost?
- Sally is making sun tea. Every hour, the concentration of the tea doubles. IF it takes 6 hours for the tea to be ready, how long would it take for the tea to reach half of the final concentration? (Finucane & Gullion, 2010)
- Jack is looking at Anne but Anne is looking George. Jack is married but George is not. Is a married person looking at an unmarried person? a. Yes b. No c. Cannot be determined (Toplak & Stanovich, 2002)
- Ann’s father has a total of five daughters: Lala, Lele, Lili, Lolo and__. What is the name of the fifth daughter?
- On the side of a boat hangs a ladder with six rungs. Each rung is one foot from the next one, and the bottom rung is resting on the surface of the water. The tide rises at a rate of one foot an hour. How long will it take the water to reach the top rung? a. 5 hours b. 6 hours c. never.
Primi et al., (2015) Cognitive reflection test – long scale.
- If three elves can wrap three toys in an hour, how many elves are needed to wrap six toys in 2 hours? (Intuitive answer 6 elves: correct answer 3 elves)
- In an athletics team, tall members are three times more likely to win a medal than short members. This year the team had won 60 medals so far. How many of these have been won by short athletes? (Intuitive answer 20 medals; correct answer 15 medals)
- If you flipped a fair coin three times, what is the probability that it would land ‘Heads’ at least once? _____ percent
- A car and a bus are on a collision course, driving toward each other. The car is going 70 miles an hour. The bus is going 80 miles an hour. How far apart are they one minute before they collide? ___ miles
- Ellen and Kim are running around a track. They run equally fast but Ellen started later. When Ellen has run 5 laps, Kim has run 15 laps. When Ellen had run 30 laps, how many has Kim run? ___ laps
- An ice cream vendor sells 2/3 of her stock of ice creams on sunny days and 1/3 of her stock on cloudy days. Yesterday, it was a sunny day, and she sold 300 ice creams. Today is a cloudy day. How many can she expect to sell?
- In a class, there are 42 children. There are 12 more girls than boys. How many girls are there in the class?
Ackerman (2014) items – translated from Hebrew
- A frog fell into a hole 30 meters deep. Every day it climbs up 3 m, but during the night it slides 2 m back down. How many days will it take the frog to climb out of the hole?(Intuitive answer 30 days; correct answer 28 days)
- Apple mash is comprised of 99% water and 1% apple solids. I left 100 kg mash in the sun and some of the water evaporated. Now the water is 98% of the mash. What is the mash weight? (Intuitive answer 99; correct answer 50)
- If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5% what is the chance that a person found to have a positive result actually has the disease, assuming that you know nothing about the person’s signs or symptoms?(Intuitive answer 95; correct answer 2)
- Every day, a bakery sells 400 cookies. When the manager is not there, 20% of the cookies made that day are eaten by the staff. How many additional cookies should be made on the manager’s day off to ensure that 400 cookies can be sold? (Intuitive answers 80, 500; correct answers 100)
- Steve was standing in a long line. To amuse himself he counted the people waiting, and saw that he stood 38th from the beginning and 56th from the end of the line. How many people are stood in the line? (Intuitive answers 94 or 92; correct answers 93)
- Ants are walking in a line. A bad-mannered ant cuts in front of the ant waling second. What is the rude ant’s place in the line? (Intuitive answer 1st; correct answers 2nd)
Tremoliere & De Neys (2014) modified congruent and incongruent versions of the bat-and-ball.
- A Ferrari and a Ford together cost $190,000. The Ferrari costs $100,000 more than the Ford. How much does the Ford cost? (Intuitive answer $90,000: correct answer $45,000)
- A Rolls-Royce and a Ferrari together cost $190,000. The Rolls-Royce costs $100,000 more than the Ferrari. How much does the Ferrari cost? (No intuitive answer; correct answer $45,000)
De Neys et al., (2013) control version of the bat-and-ball
- A magazine and a banana together cost $2.90. The magazine costs $2. How much does the banana cost? (No intuitive answer; correct answer 90 cents)
Oldrati (2016) booster items.
- A rope ladder hangs over the side of a boat with the bottom rung on the surface of the water. The rope ladder has 6 runs that are 30 cm apart from each other. The tide rises 70 cm. How many rungs will stick out of the water at high tide? (Intuitive answer 3 rungs; correct answer 6 rungs)
- There are 12 one-cent stamps in a dozen. How many two-cent stamps are there in a dozen? (Intuitive answer 6 stamps; correct answer 12 stamps)
- A farmer makes 4 piles of hay in one corner of a field and other 5 piles in another corner. If he merges them how many piles will he have? (Intuitive answer 9 piles; correct answer 1 pile)
- You are participating in a run. You overtake the second runner in the last meters before the finish line. In what position did you finish? (Intuitive answer first position; correct answer second position)
- 25 soldiers are standing in a row 3 m from each other. How long is the row? (Intuitive answer 75 m; correct answer 72 m)
- A snail starts climbing up a five-meter-high wall in the morning. During day it climbs 2 m and during the night it slips back 1 m. How many days will it take the snail to reach the top of the wall? (Intuitive answer 5 days; correct answer 4 days)
- A brick weighs 1 kg plus half a brick. How much does half a brick weigh? (Intuitive answer .5 kg; correct answer 1 kg)
- There are 5 white and 5 black socks in Franco’s drawer. Franco’s room is in the dark. How many socks should Franco take out of the drawer to be sure that he gets a matching pair? (Intuitive answer It cannot be determined; correct answer 3 socks)
- You go to bed at eight. You set your old analogue alarm clock to wake you up at nine. How many hours of sleep will you get? (Intuitive answer 13 h; correct answer 1 h)
- One month has 28 days. How man of the 11 months left have 30 days? (Intuitive answer 4 months; correct answer 11 months)
If you use any of these please reference the original source paper (cited in brackets above the items) and if possible this blog.
In APA format this blog is cited as…
Edgcumbe, D (2017). Cognitive reflection and cognitive reflection-like items [Blog post]. Retrieved from https://absolutedecisionsblog.wordpress.com/2017/01/08/cognitive-reflection-and-cognitive-reflection-like-items/
Frederick, S. (2005). Cognitive reflection and decision making. The Journal of Economic Perspectives. 19(4), 25-42.
Toplak, M., West, R. & Stanovich, K. (2014). Assessing miserly information processing: An expansion of the Cognitive Reflection Test. Thinking & Reasoning, 20(2), 147-168.
Thomson, K. & Oppenheimer D. (2016). Investigating an alternate form of the cognitive reflection test. Judgment and Decision Making, 11(1), 99.
Shtulman, A. & McCallum, K. (2014). Cognitive reflection predicts science understanding. In Proceedings of the 35th Annual Conference of Cognitive Science Society (pp. 2937-2942).
Baron, J., Scott, S., Fincher, K. & Metz, S. (2015). Why does the Cognitive Reflection Test (sometimes) predict utilitarian moral judgment (and other things)? Journal of Applied Research in Memory and Cognition, 4(3), 265-284.
Toplak, M. & Stanovich, K. (2002). The domain specificity and generality of disjunctive reasoning: Searching for a generalizable critical thinking skills. Journal of Educational Psychology, 94(1), 197.
Primi, C., Morsanyi, K., Chiesi, F., Donati, M. & Hamilton, J. (2015). The development and testing of a new version of the cognitive reflection test applying item response theory (IRT). Journal of Behavioral Decision Making,
Finucane, M. & Gullion, C. (2010). Developing a tool for measuring the decision-making competence of older adults. Psychology and aging, 25(2), 271.
Ackerman, R. (2014). The diminishing criterion model for metacognitive regulation of time investment. Journal of Experimental Psychology: General, 143(3), 1349.
Tremoliere, B., De Neys, W. & Bonnefon, J. (2014). The grim reasoner: Analytical reasoning under mortality salience. Thinking & Reasoning, 20(3), 333-351.
De Neys, W., Rossi, S. & Houde, O. (2013). Bats, balls, and substitution sensitivity: Cognitive misers are no happy fools. Psychonomic Bulletin & Review, 20(2), 269-273.
Oldrati, V., Patricelli, J., Colombo, B. & Antonietti, A. (2016). The role of dorsolateral prefrontal cortex in inhibition mechanism: A study on cognitive reflection test and similar through neuromodulation. Neuropsychologia, 91, 499-508.
Cognitive reflection test