Evaluating understanding – Page 4 – bioliteracy / biofundamentals

Is a little science a dangerous thing?

Is the popularization of science encouraging a growing disrespect for scientific expertise?

Do we need to reform science education so that students are better able to detect scientific BS?

It is common wisdom that popularizing science by exposing the public to scientific ideas is an unalloyed good, bringing benefits to both those exposed and to society at large. Many such efforts are engaging and entertaining, often taking the form of compelling images with quick cuts between excited sound bites from a range of “experts.” A number of science-centered programs, such PBS’s NOVA series, are particularly adept and/or addicted to this style. Such presentations introduce viewers to natural wonders, and often provide scientific-sounding, albeit often superficial and incomplete, explanations – they appeal to the gee-whiz and inspirational, with “mind-blowing” descriptions of how old, large, and weird the natural world appears to be. But there are darker sides to such efforts. Here I focus on one, the idea that a rigorous, realistic understanding of the scientific enterprise and its conclusions, is easy to achieve, a presumption that leads to unrealistic science education standards, and the inability to judge when scientific pronouncements are distorted or unsupported, as well as anti-scientific personal and public policy positions.That accurate thinking about scientific topics is easy to achieve is an unspoken assumption that informs much of our educational, entertainment, and scientific research system. This idea is captured in the recent NYT best seller “Astrophysics for people in a hurry” – an oxymoronic presumption. Is it possible for people “in a hurry” to seriously consider the observations and logic behind the conclusions of modern astrophysics? Can they understand the strengths and weaknesses of those conclusions? Is a superficial familiarity with the words used the same as understanding their meaning and possible significance? Is acceptance understanding? Does such a cavalier attitude to science encourage unrealistic conclusions about how science works and what is known with certainty versus what remains speculation? Are the conclusions of modern science actually easy to grasp?
The idea that introducing children to science will lead to an accurate grasp the underlying concepts involved, their appropriate application, and their limitations is not well supported [1]; often students leave formal education with a fragile and inaccurate understanding – a lesson made explicit in Matt Schneps and Phil Sadler’s Private Universe videos. The feeling that one understands a topic, that science is in some sense easy, undermines respect for those who actually do understand a topic, a situation discussed in detail in Tom Nichols “The Death of Expertise.” Under-estimating how hard it can be to accurately understand a scientific topic can lead to unrealistic science standards in schools, and often the trivialization of science education into recognizing words rather than understanding the concepts they are meant to convey.

The fact is, scientific thinking about most topics is difficult to achieve and maintain – that is what editors, reviewers, and other scientists, who attempt to test and extend the observations of others, are for – together they keep science real and honest. Until an observation has been repeated or confirmed by others, it can best be regarded as an interesting possibility, rather than a scientifically established fact. Moreover, until a plausible mechanism explaining the observation has been established, it remains a serious possibility that the entire phenomena will vanish, more or less quietly (think cold fusion). The disappearing physiological effects of “power posing” comes to mind. Nevertheless the incentives to support even disproven results can be formidable, particularly when there is money to be made and egos on the line.

While power-posing might be helpful to some, even though physiologically useless, there are more dangerous pseudo-scientific scams out there. The gullible may buy into “raw water” (see: Raw water: promises health, delivers diarrhea) but the persistent, and in some groups growing, anti-vaccination movement continues to cause real damage to children (see Thousands of cheerleaders exposed to mumps). One can ask oneself, why haven’t professional science groups, such as the American Association for the Advancement of Science (AAAS), not called for a boycott of NETFLIX, given that NETFLIX continues to distribute the anti-scientific, anti-vaccination program VAXXED [2]? And how do Oprah Winfrey and Donald Trump [link: Oprah Spreads Pseudoscience and Trump and the anti-vaccine movement] avoid universal ridicule for giving credence to ignorant non-sense, and for disparaging the hard fought expertise of the biomedical community? A failure to accept well established expertise goes along way to understanding the situation. Instead of an appreciation for what we do and do not know about the causes of autism (see: Genetics and Autism Risk & Autism and infection), there are desperate parents who turn to a range of “therapies” promoted by anti-experts. The tragic case of parents trying to cure autism by forcing children to drink bleach (see link) illustrates the seriousness of the situation.

So why do a large percentage of the public ignore the conclusions of disciplinary experts? I would argue that an important driver is the way that science is taught and popularized [3]. Beyond the obvious fact that a range of politicians and capitalists (in both the West and the East) actively distain expertise that does not support their ideological or pecuniary positions [4], I would claim that the way we teach science, often focussing on facts rather than processes, largely ignoring the historical progression by which knowledge is established, and the various forms of critical analyses to which scientific conclusions are subjected to, combines with the way science is popularized, erodes respect for disciplinary expertise. Often our education systems fail to convey how difficult it is to attain real disciplinary expertise, in particular the ability to clearly articulate where ideas and conclusions come from and what they do and do not imply. Such expertise is more than a degree, it is a record of rigorous and productive study and useful contributions, and a critical and objective state of mind. Science standards are often heavy on facts, and weak on critical analyses of those ideas and observations that are relevant to a particular process. As Carl Sagan might say, we have failed to train students on how to critically evaluate claims, how to detect baloney (or BS in less polite terms)[5].

In the area of popularizing scientific ideas, we have allowed hype and over-simplification to capture the flag. To quote from a article by David Berlinski [link: Godzooks], we are continuously bombarded with a range of pronouncements about new scientific observations or conclusions and there is often a “willingness to believe what some scientists say without wondering whether what they say is true”, or even what it actually means. No longer is the in-depth, and often difficult and tentative explanation conveyed, rather the focus is on the flashy conclusion (independent of its plausibility). Self proclaimed experts pontificate on topics that are often well beyond their areas of training and demonstrated proficiency – many is the physicist who speaks not only about the completely speculative multiverse, but on free will and ethical beliefs. Complex and often irreconcilable conflicts between organisms, such as those between mother and fetus (see: War in the womb), male and female (in sexually dimorphic species), and individual liberties and social order, are ignored instead of explicitly recognized, and their origins understood. At the same time, there are real pressures acting on scientific researchers (and the institutions they work for) and the purveyors of news to exaggerate the significance and broader implications of their “stories” so as to acquire grants, academic and personal prestige, and clicks. Such distortions serve to erode respect for scientific expertise (and objectivity).

So where are the scientific referees, the individuals that are tasked to enforce the rules of the game; to call a player out of bounds when they leave the playing field (their area of expertise) or to call a foul when rules are broken or bent, such as the fabrication, misreporting, suppression, or over-interpretation of data, as in the case of the anti-vaccinator Wakefield. Who is responsible for maintaining the integrity of the game? Pointing out the fact that many alternative medicine advocates are talking meaningless blather (see: On skepticism & pseudo-profundity)? Where are the referees who can show these charlatans the “red card” and eject them from the game?

Clearly there are no such referees. Instead it is necessary to train as large a percentage of the population as possible to be their own science referees – that is, to understand how science works, and to identify baloney when it is slung at them. When a science popularizer, whether for well meaning or self-serving reasons, steps beyond their expertise, we need to call them out of
bounds! And when scientists run up against the constraints of the scientific process, as appears to occur periodically with theoretical physicists, and the occasional neuroscientist (see: Feuding physicists and The Soul of Science) we need to recognize the foul committed. If our educational system could help develop in students a better understanding of the rules of the scientific game, and why these rules are essential to scientific progress, perhaps we can help re-establish both an appreciation of rigorous scientific expertise, as well as a respect for what is that scientists struggle to do.

Footnotes and references:

And is it clearly understood that they have nothing to say as to what is right or wrong.
Similarly, many PBS stations broadcast pseudoscientific infomercials: for example see Shame on PBS, Brain Scam, and the Deepak Chopra’s anti-scientific Brain, Mind, Body, Connection, currently playing on my local PBS station. Holocaust deniers and slavery apologists are confronted much more aggressively.
As an example, the idea that new neurons are “born” in the adult hippocampus, up to now established orthodoxy, has recently been called into question: see Study Finds No Neurogenesis in Adult Humans’ Hippocampi
Here is a particular disturbing example: By rewriting history, Hindu nationalists lay claim to India
Pennycook, G., J. A. Cheyne, N. Barr, D. J. Koehler and J. A. Fugelsang (2015). “On the reception and detection of pseudo-profound bullshit.” Judgment and Decision Making 10(6): 549.

Making education matter in higher education

It may seem self-evident that providing an effective education, the type of educational experiences that lead to a useful bachelors degree and serve as the foundation for life-long learning and growth, should be a prime aspirational driver of Colleges and Universities (1). We might even expect that various academic departments would compete with one another to excel in the quality and effectiveness of their educational outcomes; they certainly compete to enhance their research reputations, a competition that is, at least in part, responsible for the retention of faculty, even those who stray from an ethical path. Institutions compete to lure research stars away from one another, often offering substantial pay raises and research support (“Recruiting or academic poaching?”). Yet, my own experience is that a department’s performance in undergraduate educational outcomes never figures when departments compete for institutional resources, such as supporting students, hiring new faculty, or obtaining necessary technical resources (2).

I know of no example (and would be glad to hear of any) of a University hiring a professor based primarily on their effectiveness as an instructor (3).

In my last post, I suggested that increasing the emphasis on measures of departments’ educational effectiveness could help rebalance the importance of educational and research reputations, and perhaps incentivize institutions to be more consistent in enforcing ethical rules involving research malpractice and the abuse of students, both sexual and professional. Imagine if administrators (Deans and Provosts and such) were to withhold resources from departments that are performing below acceptable and competitive norms in terms of undergraduate educational outcomes?

Outsourced teaching: motives, means and impacts

Sadly, as it is, and particularly in many science departments, undergraduate educational outcomes have little if any impact on the perceived status of a department, as articulated by campus administrators. The result is that faculty are not incentivized to, and so rarely seriously consider the effectiveness of their department’s course requirements, a discussion that would of necessity include evaluating whether a course’s learning goals are coherent and realistic, whether the course is delivered effectively, whether it engages students (or is deemed irrelevant), and whether students’ achieve the desired learning outcomes, in terms of knowledge and skills achieved, including the ability to apply that knowledge effectively to new situations. Departments, particularly research focussed (dependent) departments, often have faculty with low teaching loads, a situation that incentivizes the “outsourcing” of key aspects of their educational responsibilities. Such outsourcing comes in two distinct forms, the first is requiring majors to take courses offered by other departments, even if such courses are not well designed, delivered, or (in the worst cases) relevant to the major. A classic example is to require molecular biology students to take macroscopic physics or conventional calculus courses, without regard to whether the materials presented in these courses is ever used within the major or the discipline. Expecting a student majoring in the life sciences to embrace a course that (often rightly) seems irrelevant to their discipline can alienate a student, and poses an unnecessary obstacle to student success, rather than providing students with needed knowledge and skills. Generally, the incentives necessary to generate a relevant course, for example, a molecular level physics course that would engage molecular biology students, are simply not there. A version of this situation is to require courses that are poorly designed or delivered (general chemistry is often used as the poster child for such a course). These are courses that have high failure rates, sometimes justified in terms of “necessary rigor” when in fact better course design could (and has) resulted in lower failure rates and improved learning outcomes. In addition, there are perverse incentives associated with requiring “weed out” courses offered by other departments, as they reduce the number of courses a department’s faculty needs to teach, and can lead to fewer students proceeding into upper division courses.

The second type of outsourcing involves excusing tenure track faculty from teaching introductory courses, and having them replaced by lower paid instructors or lecturers. Independently of whether instructors, lecturers, or tenure track professors make for better teaching, replacing faculty with instructors sends an implicit message to students. At the same time, the freedom of instructors/lecturers to adopt an effective (socratic) approach to teaching is often severely constrained; common exams can force classes to move in lock step, independently of whether that pace is optimal for student engagement and learning. Generally, instructors/lecturers do not have the freedom to adjust what they teach, to modify the emphasis and time they spend on specific topics in response to their students’ needs. How an instructor instructs their students suffers when teachers do not have the freedom to customize their interactions with students in response to where they are intellectually. This is particularly detrimental in the case of underrepresented or underprepared students. Generally, a flexible and adaptive approach to instruction (including ancillary classes on how to cope with college: see An alternative to remedial college classes gets results) can address many issues, and bring the majority of students to a level of competence, whereas tracking students into remedial classes can succeed in driving them out of a major or college (see Colleges Reinvent Classes to Keep More Students in Science and Redesigning a Large-Enrollment Introductory Biology Course and Does Remediation Work for All Students? )

How to address this imbalance, how can we reset the pecking order so that effective educational efforts actually matter to a department?

My (modest) suggestion is to base departmental rewards on objective measures of educational effectiveness. And by rewards I mean both at the level of individuals (salary and status) as well as support for graduate students, faculty positions, start up funds, etc. What if, for example, faculty in departments that excel at educating their students received a teaching bonus, or if the number of graduate students within a department supported by the institution was determined not by the number of classes these graduate students taught (courses that might not be particularly effective or engaging) but rather by a departments’ undergraduate educational effectiveness, as measured by retention, time to degree, and learning outcomes (see below)? The result could well be a drive within a department to improve course and curricular effectiveness to maximize education-linked rewards. Given that laboratory courses, the courses most often taught by science graduate students, are multi-hour schedule disrupting events, of limited demonstrable educational effectiveness, that complicate student course scheduling, removing requirements for lab courses deemed unnecessary (or generating more effective versions), would be actively rewarded (of course, sanctions for continuing to offer ineffective courses would also be useful, but politically more problematic.)

A similar situation applies when a biology department requires its majors to take 5 credit hour physics or chemistry courses. Currently it is “easy” for a department to require its students to take such courses without critically evaluating whether they are “worth it”, educationally. Imagine how a department’s choices of required courses would change if the impact of high failure rates (which I would argue is a proxy for poorly designed and delivered courses) directly impacted the rewards reaped by a department. There would be an incentive to look critically at such courses, to determine whether they are necessary and if so, well designed and delivered. Departments would serve their own interests if they invested in the development of courses that better served their disciplinary goals, courses likely to engage their students’ interests.

So how do we measure a department’s educational efficacy?

There are three obvious metrics: i) retention of students as majors (or in the case of “service courses” for non-majors, whether students master what it is the course claims to teach); ii) time to degree (and by that I mean the percentage of students who graduate in 4 years, rather than the 6 year time point reported in response to federal regulations (six year graduation rate | background on graduation rates); and iii) objective measures of student learning outcomes attained and skills achieved. The first two are easy, Universities already know these numbers. Moreover they are directly influenced by degree requirements – requiring students to take boring and/or apparently irrelevant courses serves to drive a subset of students out of a major. By making courses relevant and engaging, more students can be retained in a degree program. At the same time, thoughtful course design can help students pass through even the most rigorous (difficult) of such courses. The third, learning outcomes, is significantly more challenging to measure, since universal metrics are (largely) missing or superficial. A few disciplines, such as chemistry, support standardized assessments, although one could argue with what such assessments measure. Nevertheless, meaningful outcomes measures are necessary, in much the same way that Law and Medical boards and the Fundamentals of Engineering exam serve to help insure (although they do not guarantee) the competence of practitioners. One could imagine using parts of standardized exams, such as discipline specific GRE exams, to generate outcomes metrics, although more informative assessment instruments would clearly be preferable. The initiative in this area could be taken by professional societies, college consortia (such as the AAU), and research foundations, as a critical driver for education reform, increased effectiveness, and improved cost-benefit outcomes, something that could help address the growing income inequality in our country and make success in higher education an important factor contributing to an institution’s reputation.

A footnote or two…

1. My comments are primarily focused on research universities, since that is where my experience lies; these are, of course, the majority of the largest universities (in a student population sense).

2. Although my experience is limited, having spent my professorial career at a single institution, conversations with others leads me to conclude that it is not unique.

3. The one obvious exception would be the hiring of coaches of sports teams, since their success in teaching (coaching) is more directly discernible and impactful on institutional finances and reputation).

minor edits – 16 March 2020

Reverse Dunning-Kruger effects and science education

The Dunning-Kruger (DK) effect is the well-established phenomenon that people tend to over estimate their understanding of a particular topic or their skill at a particular task, often to a dramatic degree [link][link]. We see examples of the DK effect throughout society; the current administration (unfortunately) and the nutritional supplements / homeopathy section of Whole Foods spring to mind as examples. But there is a less well-recognized “reverse DK” effect, namely the tendency of instructors, and a range of other public communicators, to over-estimate what the people they are talking to are prepared to understand, appreciate, and accurately apply. The efforts of science communicators and instructors can be entertaining but the failure to recognize and address the reverse DK effect results in ineffective educational efforts. These efforts can themselves help generate the illusion of understanding in students and the broader public (discussed here). While a confused understanding of the intricacies of cosmology or particle physics can be relatively harmless in their social and personal implications, similar misunderstandings become personally and publicly significant when topics such as vaccination, alternative medical treatments, and climate change are in play.

There are two synergistic aspects to the reverse DK effect that directly impact science instruction: the need to understand what one’s audience does not understand together with the need to clearly articulate the conceptual underpinnings needed to understand the subject to be taught. This is in part because modern science has, at its core, become increasingly counter-intuitive over the last approximately 100 years or so, a situation that can cause serious confusions that educators must address directly and explicitly. The first reverse DK effect involves the extent to which the instructor (and by implication the course and textbook designer) has an accurate appreciation of what students think or think they know, what ideas they have previously been exposed to, and what they actually understand about the implications of those ideas. Are they prepared to learn a subject or does the instructor first have to acknowledge and address conceptual confusions and build or rebuild base concepts? While the best way to discover what students think is arguably a Socratic discussion, this only rarely occurs for a range of practical reasons. In its place, a number of concept inventory-type testing instruments have been generated to reveal whether various pre-identified common confusions exist in students’ thinking. Knowing the results of such assessments BEFORE instruction can help customize how the instructor structures the learning environment and content to be presented and whether the instructor gives students the space to work with these ideas to develop a more accurate and nuanced understanding of a topic. Of course, this implies that instructors have the flexibility to adjust the pace and focus of their classroom activities. Do they take the time needed to address student issues or do they feel pressured to plow through the prescribed course content, come hell, high water, or cascading student befuddlement.

A complementary aspect of the reverse DK effect, well-illustrated in the “why magnets attract” interview with the physicist Richard Feynman, is that the instructor, course designer, or textbook author(s) needs to have a deep and accurate appreciation of the underlying core knowledge necessary to understand the topic they are teaching. Such a robust conceptual understanding makes it possible to convey the complexities involved in a particular process and explicitly values appreciating a topic rather than memorizing it. It focuses on the general, rather than the idiosyncratic. A classic example from many an introductory biology course is the difference between expecting students to remember the steps in glycolysis or the Krebs cycle reaction system, as opposed to the general principles that underlie the non-equilibrium reaction networks involved in all biological functions, a reaction network based on coupled chemical reactions and governed by the behaviors of thermodynamically favorable and unfavorable reactions. Without a explicit discussion of these topics, all too often students are required to memorize names without understanding the underlying rationale driving the processes involved; that is, why the system behaves as it does. Instructors also give false “rubber band” analogies or heuristics to explain complex phenomena (see Feynman video 6:18 minutes in). A similar situation occurs when considering how molecules come to associate and dissociate from one another, for example in the process of regulating gene expression or repairing mutations in DNA. Most textbooks simply do not discuss the physiochemical processes involved in binding specificity, association, and dissociation rates, such as the energy changes associated with molecular interactions and thermal collisions (don’t believe me? look for yourself!). But these factors are essential for a student to understand the dynamics of gene expression [link], as well as the specificity of modern methods involved in genetic engineering, such as restriction enzymes, polymerase chain reaction, and CRISPR CAS9-mediated mutagenesis. By focusing on the underlying processes involved we can avoid their trivialization and enable students to apply basic principles to a broad range of situations. We can understand exactly why CRISPR CAS9-directed mutagenesis can be targeted to a single site within a multibillion-base pair genome.

Of course, as in the case of recognizing and responding to student misunderstandings and knowledge gaps, a thoughtful consideration of underlying processes takes course time, time that trades the development of a working understanding of core processes and principles for broader “coverage” of frequently disconnected facts, the memorization and regurgitation of which has been privileged over understanding why those facts are worth knowing. If our goal is for students to emerge from a course with an accurate understanding of the basic processes involved rather than a superficial familiarity with a plethora of unrelated facts, however, a Socratic interaction with the topic is essential. What assumptions are being made, where do they come from, how do they constrain the system, and what are their implications? Do we understand why the system behaves the way it does? In this light, it is a serious educational mystery that many molecular biology / biochemistry curricula fail to introduce students to the range of selective and non-selective evolutionary mechanisms (including social and sexual selection – see link), that is, the processes that have shaped modern organisms.

Both aspects of the reverse DK effect impact educational outcomes. Overcoming the reverse DK effect depends on educational institutions committing to effective and engaging course design, measured in terms of retention, time to degree, and a robust inquiry into actual student learning. Such an institutional dedication to effective course design and delivery is necessary to empower instructors and course designers. These individuals bring a deep understanding of the topics taught and their conceptual foundations and historic development to their students AND must have the flexibility and authority to alter the pace (and design) of a course or a curriculum when they discover that their students lack the pre-existing expertise necessary for learning or that the course materials (textbooks) do not present or emphasize necessary ideas. Radiation-kills-in-Boulder Unfortunately, all too often instructors, particularly in introductory level college science courses, are not the masters of their ships; that is, they are not rewarded for generating more effective course materials. An emphasis on course “coverage” over learning, whether through peer-pressure, institutional apathy, or both, generates unnecessary obstacles to both student engagement and content mastery. To reverse the effects of the reverse DK effect, we need to encourage instructors, course designers, and departments to see the presentation of core disciplinary observations and concepts as the intellectually challenging and valuable endeavor that it is. In its absence, there are serious (and growing) pressures to trivialize or obscure the educational experience – leading to the socially- and personally-damaging growth of fake knowledge.

empty images holders removed, new image added – 17 December 2020

The trivialization of science education

It’s time for universities to accept their role in scientific illiteracy.

There is a growing problem with scientific illiteracy, and its close relative, scientific over-confidence. While understanding science, by which most people seem to mean technological skills, or even the ability to program a device (1), is purported to be a critical competitive factor in our society, we see a parallel explosion of pseudo-scientific beliefs, often religiously held. Advocates of a gluten-free paleo-diet battle it out with orthodox vegans for a position on the Mount Rushmore of self-righteousness, at the same time astronomers and astrophysicists rebrand themselves as astrobiologists (a currently imaginary discipline) while a subset of theoretical physicists, and the occasional evolutionary biologist, claim to have rendered ethicists and philosophers obsolete (oh, if it were only so). There are many reasons for this situation, most of which are probably innate to the human condition. Our roots are in the vitamin C-requiring Haplorhini (dry nose) primate family, we were not evolved to think scientifically, and scientific thinking does not come easy for most of us, or for any of us over long periods of time (2). The fact that the sciences are referred to as disciplines reflects this fact, it requires constant vigilance, self-reflection, and the critical skepticism of knowledgeable colleagues to build coherent, predictive, and empirically validated models of the Universe (and ourselves). In point of fact, it is amazing that our models of the Universe have become so accurate, particularly as they are counter-intuitive and often seem incredible, using the true meaning of the word.

Many social institutions claim to be in the business of developing and supporting scientific literacy and disciplinary expertise, most obviously colleges and universities. Unfortunately, there are several reasons to question the general efficacy of their efforts and several factors that have led to this failure. There is the general tendency (although exactly how wide-spread is unclear, I cannot find appropriate statistics on this question) of requiring non-science students to take one, two, or more “natural science” courses, often with associated laboratory sections, as a way to “enhance literacy and knowledge of one or more scientific disciplines, and enhance those reasoning and observing skills that are necessary to evaluate issues with scientific content” (source).

That such a requirement will “enable students to understand the current state of knowledge in at least one scientific discipline, with specific reference to important past discoveries and the directions of current development; to gain experience in scientific observation and measurement, in organizing and quantifying results, in drawing conclusions from data, and in understanding the uncertainties and limitations of the results; and to acquire sufficient general scientific vocabulary and methodology to find additional information about scientific issues, to evaluate it critically, and to make informed decisions” (source) suggests a rather serious level of faculty/institutional distain or apathy for observable learning outcomes, devotional levels of wishful thinking, or simple hubris. To my knowledge there is no objective evidence to support the premise that such requirements achieve these outcomes – which renders the benefits of such requirements problematic, to say the least (link).

On the other hand, such requirements have clear and measurable costs; going beyond the simple burden of added and potentially ineffective or off-putting course credit hours. The frequent requirement for multi-hour laboratory courses impacts the ability of students to schedule courses. It would be an interesting study to examine how, independently of benefit, such laboratory course requirements impact students’ retention and time to degree, that is, bluntly put, costs to students and their families.

Now, if there were objective evidence that taking such courses improved students’ understanding of a specific disciplinary science and its application, perhaps the benefit would warrant the cost. But one can be forgiven if one assumes a less charitable driver, that is, science departments’ self-interest in using laboratory and other non-major course requirements as means to support graduate students. Clearly there is a need for objective metrics for scientific, that is disciplinary, literacy and learning outcomes.

And this brings up another cause for concern. Recently, there has been a movement within the science education research community to attempt to quantify learning in terms of what are known as “forced choice testing instruments;” that is, tests that rely on true/false and multiple-choice questions, an actively anti-Socratic strategy. In some cases, these tests claim to be research based. As one involved in the development of such a testing instrument (the Biology Concepts Instrument or BCI), it is clear to me that such tests can serve a useful role in helping to identify areas in which student understanding is weak or confused [example], but whether they can provide an accurate or, at the end of the day, meaningful measure of whether students have developed an accurate working understanding of complex concepts and the broader meaning of observations is problematic at best.

Establishing such a level of understanding relies on Socratic, that is, dynamic and adaptive evaluations: can the learner clearly explain, either to other experts or to other students, the source and implications of their assumptions? This is the gold standard for monitoring disciplinary understanding. It is being increasingly side-lined by those who rely on forced choice tests to evaluate learning outcomes and to support their favorite pedagogical strategies (examples available upon request). In point of fact, it is often difficult to discern, in most science education research studies, what students have come to master, what exactly they know, what they can explain and what they can do with their knowledge. Rather unfortunately, this is not a problem restricted to non-majors taking science course requirements; majors can also graduate with a fragmented and partially, or totally, incoherent understanding of key ideas and their empirical foundations.

So what are the common features of a functional understanding of a particular scientific discipline, or more accurately, a sub-discipline? A few ideas seem relevant. A proficient needs to be realistic about their own understanding. We need to teach disciplinary (and general) humility – no one actually understands all aspects of most scientific processes. This is a point made by Fernbach & Sloman in their recent essay, “Why We Believe Obvious Untruths.” Humility about our understanding has a number of beneficial aspects. It helps keep us skeptical when faced with, and asked to accept, sweeping generalizations.

Such skepticism is part of a broader perspective, common among working scientists, namely the ability to distinguish the obvious from the unlikely, the implausible, and the impossible. When considering a scientific claim, the first criterion is whether there is a plausible mechanism that can be called upon to explain it, or does it violate some well-established “law of nature”. Claims of “zero waste” processes butt up against the laws of thermodynamics.

Going further, we need to consider how the observation or conclusions fits with other well established principles, which means that we have to be aware of these principles, as well as acknowledging that we are not universal experts in all aspects of science. A molecular biologist may recognize that quantum mechanics dictates the geometries of atomic bonding interactions without being able to formally describe the intricacies of the molecule’s wave equation. Similarly, a physicist might think twice before ignoring the evolutionary history of a species, and claiming that quantum mechanics explains consciousness, or that consciousness is a universal property of matter. Such a level of disciplinary expertise can take extended experience to establish, but is critical to conveying what disciplinary mastery involves to students; it is the major justification for having disciplinary practitioners (professors) as instructors.

From a more prosaic educational perspective other key factors need to be acknowledged, namely a realistic appreciation of what people can learn in the time available to them, while also understanding at least some of their underlying motivations, which is to say that the relevance of a particular course to disciplinary goals or desired educational outcomes needs to be made explicit and as engaging as possible, or at least not overtly off putting, something that can happen when a poor unsuspecting molecular biology major takes a course in macroscopic physics, taught by an instructor who believes organisms are deducible from first principles based on the conditions of the big bang. Respecting the learner requires that we explicitly acknowledge that an unbridled thirst for an empirical, self-critical, mastery of a discipline is not a basic human trait, although it is something that can be cultivated, and may emerge given proper care. Understanding the real constraints that act on meaningful learning can help focus courses on what is foundational, and help eliminate the irrelevant or the excessively esoteric.

Unintended consequences arise from “pretending” to teach students, both majors and non-science majors, science. One is an erosion of humility in the face of the complexity of science and our own limited understanding, a point made in a recent National Academy report that linked superficial knowledge with more non-scientific attitudes. The end result is an enhancement of what is known as the Kruger-Dunning effect, the tendency of people to seriously over-estimate their own expertise: “the effect describes the way people who are the least competent at a task often rate their skills as exceptionally high because they are too ignorant to know what it would mean to have the skill”.

A person with a severe case of Kruger-Dunning-itis is likely to lose respect for people who actually know what they are talking about. The importance of true expertise is further eroded and trivialized by the current trend of having photogenic and well-speaking experts in one domain pretend to talk, or rather to pontificate, authoritatively on another (3). In a world of complex and arcane scientific disciplines, the role of a science guy or gal can promote rather than dispel scientific illiteracy.

We see the effects of the lack of scientific humility when people speak outside of their domain of established expertise to make claims of certainty, a common feature of the conspiracy theorist. An oft used example is the claim that vaccines cause autism (they don’t), when the actual causes of autism, whether genetic and/or environmental, are currently unknown and the subject of active scientific study. An honest expert can, in all humility, identify the limits of current knowledge as well as what is known for certain. Unfortunately, revealing and ameliorating the levels of someone’s Kruger-Dunning-itis involves a civil and constructive Socratic interrogation, something of an endangered species in this day and age, where unseemly certainty and unwarranted posturing have replaced circumspect and critical discourse. Any useful evaluation of what someone knows demands the time and effort inherent in a Socratic discourse, the willingness to explain how one knows what one thinks one knows, together with a reflective consideration of its implications, and what it is that other trained observers, people demonstrably proficient in the discipline, have concluded. It cannot be replaced by a multiple choice test.

Perhaps a new (old) model of encouraging in students, as well as politicians and pundits, an understanding of where science comes from, the habits of mind involved, the limits of, and constraints on, our current understanding is needed. At the college level, courses that replace superficial familiarity and unwarranted certainty with humble self-reflection and intellectual modesty might help treat the symptoms of Kruger-Dunning-itis, even though the underlying disease may be incurable, and perhaps genetically linked to other aspects of human neuronal processing.

some footnotes:

after all, why are rather distinct disciplines lumped together as STEM (science, technology, engineering and mathematics).
Given the long history of Homo sapiens before the appearance of science, it seems likely that such patterns of thinking are an unintended consequence of selection for some other trait, and the subsequent emergence of (perhaps excessively) complex and self-reflective nervous system.
Another example of Neil Postman’s premise that education is be replaced by edutainment (see “Amusing ourselves to Death”.