Every so often Al Frances says something that seems to surprise even him. Just now, for instance, in the predawn darkness of his comfortable, rambling home in Carmel, California, he has broken off his exercise routine to declare that “there is no definition of a mental disorder. It’s bullshit. I mean, you just can’t define it.” Then an odd, reflective look crosses his face, as if he’s taking in the strangeness of this scene: Allen Frances, lead editor of the fourth edition of the American Psychiatric Association’s Diagnostic and Statistical Manual of Mental Disorders (universally known as the DSM-IV), the guy who wrote the book on mental illness, confessing that “these concepts are virtually impossible to define precisely with bright lines at the boundaries.” For the first time in two days, the conversation comes to an awkward halt.

But he recovers quickly, and back in the living room he finishes explaining why he came out of a seemingly contented retirement to launch a bitter and protracted battle with the people, some of them friends, who are creating the next edition of the DSM. And to criticize them not just once, and not in professional mumbo jumbo that would keep the fight inside the professional family, but repeatedly and in plain English, in newspapers and magazines and blogs. And to accuse his colleagues not just of bad science but of bad faith, hubris, and blindness, of making diseases out of everyday suffering and, as a result, padding the bottom lines of drug companies. These aren’t new accusations to level at psychiatry, but Frances used to be their target, not their source. He’s hurling grenades into the bunker where he spent his entire career.

As a practicing psychotherapist myself, I can attest that this is a startling turn. But when Frances tries to explain it, he resists the kinds of reasons that mental health professionals usually give each other, the ones about character traits or personality quirks formed in childhood. He says he doesn’t want to give ammunition to his enemies, who have already shown their willingness to “shoot the messenger.” It’s not an unfounded concern. In its first official response to Frances, the APA diagnosed him with “pride of authorship” and pointed out that his royalty payments would end once the new edition was published—a fact that “should be considered when evaluating his critique and its timing.”

Frances, who claims he doesn’t care about the royalties (which amount, he says, to just 10 grand a year), also claims not to mind if the APA cites his faults. He just wishes they’d go after the right ones—the serious errors in the DSM-IV. “We made mistakes that had terrible consequences,” he says. Diagnoses of autism, attention-deficit hyperactivity disorder, and bipolar disorder skyrocketed, and Frances thinks his manual inadvertently facilitated these epidemics—and, in the bargain, fostered an increasing tendency to chalk up life’s difficulties to mental illness and then treat them with psychiatric drugs.

The insurgency against the DSM-5 (the APA has decided to shed the Roman numerals) has now spread far beyond just Allen Frances. Psychiatrists at the top of their specialties, clinicians at prominent hospitals, and even some contributors to the new edition have expressed deep reservations about it. Dissidents complain that the revision process is in disarray and that the preliminary results, made public for the first time in February 2010, are filled with potential clinical and public relations nightmares. Although most of the dissenters are squeamish about making their concerns public—especially because of a surprisingly restrictive nondisclosure agreement that all insiders were required to sign—they are becoming increasingly restive, and some are beginning to agree with Frances that public pressure may be the only way to derail a train that he fears will “take psychiatry off a cliff.”

At stake in the fight between Frances and the APA is more than professional turf, more than careers and reputations, more than the $6.5 million in sales that the DSM averages each year. The book is the basis of psychiatrists’ authority to pronounce upon our mental health, to command health care dollars from insurance companies for treatment and from government agencies for research. It is as important to psychiatrists as the Constitution is to the US government or the Bible is to Christians. Outside the profession, too, the DSM rules, serving as the authoritative text for psychologists, social workers, and other mental health workers; it is invoked by lawyers in arguing over the culpability of criminal defendants and by parents seeking school services for their children. If, as Frances warns, the new volume is an “absolute disaster,” it could cause a seismic shift in the way mental health care is practiced in this country. It could cause the APA to lose its franchise on our psychic suffering, the naming rights to our pain.

This is hardly the first time that defining mental illness has led to rancor within the profession. It happened in 1993, when feminists denounced Frances for considering the inclusion of “late luteal phase dysphoric disorder” (formerly known as premenstrual syndrome) as a possible diagnosis for DSM-IV. It happened in 1980, when psychoanalysts objected to the removal of the word neurosis—their bread and butter—from the DSM-III. It happened in 1973, when gay psychiatrists, after years of loud protest, finally forced a reluctant APA to acknowledge that homosexuality was not and never had been an illness. Indeed, it’s been happening since at least 1922, when two prominent psychiatrists warned that a planned change to the nomenclature would be tantamount to declaring that “the whole world is, or has been, insane.”

Some of this disputatiousness is the hazard of any professional specialty. But when psychiatrists say, as they have during each of these fights, that the success or failure of their efforts could sink the whole profession, they aren’t just scoring rhetorical points. The authority of any doctor depends on their ability to name a patient’s suffering. For patients to accept a diagnosis, they must believe that doctors know—in the same way that physicists know about gravity or biologists about mitosis—that their disease exists and that they have it. But this kind of certainty has eluded psychiatry, and every fight over nomenclature threatens to undermine the legitimacy of the profession by revealing its dirty secret: that for all their confident pronouncements, psychiatrists can’t rigorously differentiate illness from everyday suffering. This is why, as one psychiatrist wrote after the APA voted homosexuality out of the DSM, “there is a terrible sense of shame among psychiatrists, always wanting to show that our diagnoses are as good as the scientific ones used in real medicine.”

Since 1980, when the DSM-III was published, psychiatrists have tried to solve this problem by using what is called descriptive diagnosis: a checklist approach, whereby illnesses are defined wholly by the symptoms patients present. The main virtue of descriptive psychiatry is that it doesn’t rely on unprovable notions about the nature and causes of mental illness, as the Freudian theories behind all those “neuroses” had done. Two doctors who observe a patient carefully and consult the DSM’s criteria lists usually won’t disagree on the diagnosis—something that was embarrassingly common before 1980. But descriptive psychiatry also has a major problem: Its diagnoses are nothing more than groupings of symptoms. If, during a two-week period, you have five of the nine symptoms of depression listed in the DSM, then you have “major depression,” no matter your circumstances or your own perception of your troubles. “No one should be proud that we have a descriptive system,” Frances tells me. “The fact that we do only reveals our limitations.” Instead of curing the profession’s own malady, descriptive psychiatry has just covered it up.

The DSM-5 battle comes at a time when psychiatry’s authority seems more tenuous than ever. In terms of both research dollars and public attention, molecular biology—neuroscience and genetics—has come to dominate inquiries into what makes us tick. And indeed, a few tantalizing results from these disciplines have cast serious doubt on long-held psychiatric ideas. Take schizophrenia and bipolar disorder: For more than a century, those two illnesses have occupied separate branches of the psychiatric taxonomy. But research suggests that the same genetic factors predispose people to both illnesses, a discovery that casts doubt on whether this fundamental division exists in nature or only in the minds of psychiatrists. Other results suggest new diagnostic criteria for diseases: Depressed patients, for example, tend to have cell loss in the hippocampal regions, areas normally rich in serotonin. Certain mental illnesses are alleviated by brain therapies, such as transcranial magnetic stimulation, even as the reasons why are not entirely understood.

Some mental health researchers are convinced that the DSM might soon be completely revolutionized or even rendered obsolete. In recent years, the National Institute of Mental Health has launched an effort to transform psychiatry into what its director, Thomas Insel, calls clinical neuroscience. This project will focus on observable ways that brain circuitry affects the functional aspects of mental illness—symptoms, such as anger or anxiety or disordered thinking, that figure in our current diagnoses. The institute says it’s “agnostic” on the subject of whether, or how, this process would create new definitions of illnesses, but it seems poised to abandon the reigning DSM approach. “Our resources are more likely to be invested in a program to transform diagnosis by 2020,” Insel says, “rather than modifying the current paradigm.”

Although the APA doesn’t disagree that a revolution might be on the horizon, the organization doesn’t feel it can wait until 2020, or beyond, to revise the DSM-IV. Its categories line up poorly with the ways people actually suffer, leading to high rates of patients with multiple diagnoses. Neither does the manual help therapists draw on a body of knowledge, developed largely since DSM-IV, about how to match treatments to patients based on the specific features of their disorder. The profession cannot afford to wait for the science to catch up to its needs. Which means that the stakes are higher, the current crisis deeper, and the potential damage to psychiatry greater than ever before.

Allen Frances’ revolt against the DSM-5 was spurred by another unlikely revolutionary: Robert Spitzer, lead editor of the DSM-III and a man believed by many to have saved the profession by spearheading the shift to descriptive psychiatry. As the DSM-5 task force began its work, Spitzer was “dumbfounded” when Darrel Regier, the APA’s director of research and vice chair of the task force, refused his request to see the minutes of its meetings. Soon thereafter, he was appalled, he says, to discover that the APA had required psychiatrists involved with the revision to sign a paper promising they would never talk about what they were doing, except when necessary for their jobs. “The intent seemed to be not to let anyone know what the hell was going on,” Spitzer says.

In July 2008, Spitzer wrote a letter to Psychiatric News, an APA newsletter, complaining that the secrecy was at odds with scientific process, which “benefits from the very exchange of information that is prohibited by the confidentiality agreement.” He asked Frances to sign onto his letter, but Frances declined; a decade into his retirement from Duke University Medical School, he had mostly stayed on the sidelines since planning for the DSM-5 began in 1999, and he intended to keep it that way. “I told him I completely agreed that this was a disastrous way for DSM-5 to start, but I didn’t want to get involved at all. I wished him luck and went back to the beach.”

But that was before Frances found out about a new illness proposed for the DSM-5. In May 2009, during a party at the APA’s annual convention in San Francisco, he struck up a conversation with Will Carpenter, a psychiatrist at the University of Maryland. Carpenter is chair of the Psychotic Disorders work group, one of 13 DSM-5 panels that have been holding meetings since 2008 to consider revisions. These panels, each comprising 10 or so psychiatrists and other mental health professionals, report to the supervising task force, which consists of the work-group chairs and a dozen other experts. The task force will turn the work groups’ proposals into a rough draft to be field-tested, revised, and then ratified—first by the APA’s trustees and then by its 39,000 members.

At the party, Frances and Carpenter began to talk about “psychosis risk syndrome,” a diagnosis that Carpenter’s group was considering for the new edition. It would apply mostly to adolescents who occasionally have jumbled thoughts, hear voices, or experience delusions. Since these kids never fully lose contact with reality, they don’t qualify for any of the existing psychotic disorders. But “throughout medicine, there’s a presumption that early identification and intervention is better than late,” Carpenter says, citing the monitoring of cholesterol as an example. If adolescents on the brink of psychosis can be treated before a full-blown psychosis develops, he adds, “it could make a huge difference in their life story.”

This new disease reminded Frances of one of his keenest regrets about the DSM-IV: its role, as he perceives it, in the epidemic of bipolar diagnoses in children over the past decade. Shortly after the book came out, doctors began to declare children bipolar even if they had never had a manic episode and were too young to have shown the pattern of mood change associated with the disease. Within a dozen years, bipolar diagnoses among children had increased 40-fold. Many of these kids were put on antipsychotic drugs, whose effects on the developing brain are poorly understood but which are known to cause obesity and diabetes. In 2007, a series of investigative reports revealed that an influential advocate for diagnosing bipolar disorder in kids, the Harvard psychiatrist Joseph Biederman, failed to disclose money he’d received from Johnson & Johnson, makers of the bipolar drug Risperdal, or risperidone. (The New York Times reported that Biederman told the company his proposed trial of Risperdal in young children “will support the safety and effectiveness of risperidone in this age group.”) Frances believes this bipolar “fad” would not have occurred had the DSM-IV committee not rejected a move to limit the diagnosis to adults.

Frances found psychosis risk syndrome particularly troubling in light of research suggesting that only about a quarter of its sufferers would go on to develop full-blown psychoses. He worried that those numbers would not stop drug companies from seizing on the new diagnosis and sparking a new treatment fad—a danger that Frances thought Carpenter was grievously underestimating. He already regretted having remained silent when, in the 1980s, he watched the pharmaceutical industry insinuate itself into the APA’s training programs. (Annual drug company contributions to those programs reached as much as $3 million before the organization decided, in 2008, to phase out industry-supported education.) Frances didn’t want to be “a crusader for the world,” he says. But the idea of more “kids getting unneeded antipsychotics that would make them gain 12 pounds in 12 weeks hit me in the gut. It was uniquely my job and my duty to protect them. If not me to correct it, who? I was stuck without an excuse to convince myself.”

At the party, he found Bob Spitzer’s wife and asked her to tell her husband (who had been prevented from traveling due to illness) that he was going to join him in protesting the DSM-5.

Throughout 2009, Spitzer and Frances carried out their assault. That June, Frances published a broadside on the website of Psychiatric Times, an independent industry newsletter. Among the numerous alarms the piece sounded, Frances warned that the new DSM, with its emphasis on early intervention, would cause a “wholesale imperial medicalization of normality” and “a bonanza for the pharmaceutical industry,” for which patients would pay the “high price [of] adverse effects, dollars, and stigma.” Two weeks later, the two men wrote a letter to the APA’s trustees, urging them to consider forming an oversight committee and postponing publication, in order to avoid an “embarrassing DSM-5.” Such a committee was convened, and it did recommend a delay, because—as its chair, a former APA president, later put it—”the revision process hadn’t begun to coalesce as much as it should have.” In December 2009, the APA announced a one-year postponement, pushing publication back to 2013. (The organization insists that Frances “did not have an impact” on the rescheduling of the revision.)

James Scully, medical director of the APA, fills the big leather chair in his office overlooking the Potomac River and the government buildings beyond. He’s a large, ruddy-faced man with a shock of white hair, and when he leans forward, his monogrammed cuffs perched on his knees, to deliver his assessment of Frances, even though it’s only two words—”he’s wrong”—you can hear his rising gorge and the sense of betrayal that seems to be swelling behind it.

Of all the things that Frances is wrong about—and there are many, Scully says, including his position on psychosis risk syndrome—the confidentiality agreement seems to be the one that really galls. First of all, it’s simply an intellectual property agreement “about who owns the product.” Second, he insists, this is the most open and transparent DSM revision ever, certainly more open than the process that produced Spitzer’s and Frances’ manuals, which were written in the pre-Internet era, before it was possible to field, as the task force has, 8,000 online comments on the proposed changes.

The agreement may well be mere intellectual property boilerplate. But, as I explain to Scully and later to APA research chief Darrel Regier, that hasn’t reassured all the psychiatrists who’ve had to sign it. They fret privately that the DSM-5 will create “monumental screwups” that will turn the field into a “laughingstock.” They accuse the task force of “not knowing where they’re going” and of “not having managed this right from the very beginning.” They worry that the “slipshod nature of the whole process” will lead to a “crappy product” that alienates clinicians even as it makes psychiatry “look capricious and silly.” None of them, however, are willing to go on record, for fear—unfounded or not—of “retaliation” and “reprisal.”

Regier wants to know who said these things.

Not all the dissidents are insisting on anonymity. E. Jane Costello, codirector of the Center for Developmental Epidemiology at Duke Medical School, says she doesn’t mind going on record because she’s “too small a fish” for them to bother with. Costello was one of two psychiatrists who resigned from the Childhood Disorders work group in spring 2009. In her resignation letter, which she subsequently made public, Costello excoriated the DSM committee for refusing to wait for the results of longitudinal studies she was planning and for failing to underwrite adequate research of its own. The proposed revisions, she wrote, “seem to have little basis in new scientific findings or organized clinical or epidemiological studies.” (In a response, the APA cited “several billions of dollars” already spent over the past 40 years on research the revision is drawing upon.)

To critics, the greatest liability of the DSM-5 process is precisely this disconnect between its ambition on one hand and the current state of the science on the other. Of particular concern is a proposal to institute “dimensional assessment” as part of all diagnostic evaluations. In this approach, clinicians would use standardized, diagnostic-specific tests to assign a severity rating to each patient’s illness. Regier hopes that these ratings, tallied against data about the course and outcome of illnesses, will eventually lead to psychiatry’s holy grail: “statistically valid cutpoints between normal and pathological.” Able to reliably rate the clinical significance of a disorder, doctors would finally have a scientific way to separate the sick from the merely suffering.

No one, not even Frances, thinks it’s a bad idea to augment the current binary approach to diagnosis, in which you either have the requisite symptoms or you don’t, with a method for quantifying gradations in illness. Dimensional assessment could provide what Frances calls a “governor” on absurdly high rates of diagnosis—by DSM criteria, epidemiologists have noted, a staggering 30 percent of Americans are mentally ill in any given year—and thereby solve both a public health problem and a public relations problem.

But Michael First, a Columbia University psychiatrist who headed up the DSM-5’s Prelude Project to solicit feedback before the revision, believes that implementing dimensional assessment right now is a tremendous mistake. The tests, he says, are nowhere near ready for use; while some of them have a long track record, “it seems that many of them were made up by the work groups” without any real-world validation. Bad tests could be disastrous not just for the profession, which would erect its diagnostic regime on a shaky foundation, but also for patients: If the tests have been sanctioned in the DSM, insurance companies could use them to cut off coverage for patients deemed not sick enough. “If they really want to do dimensional assessment,” First says, “they should wait the five or 10 years it would take for the scales to be ready.”

Regier won’t say how many of the tests are usable yet. “I don’t think it will be useful to get into this level of detail,” he emails. He acknowledges that dimensional assessment is still evolving, and he says the DSM-5 field trials—studies in which doctors will test the rough draft of the manual with patients—will help refine the tests. But the field trials, too, are bumping up against formidable deadlines. Although trials were scheduled to begin in May 2010, as of October only a pilot study was actually under way—and protocols for the rest of the trials couldn’t be finalized until that study was completed. Meanwhile, Regier has pegged May 2013 as a drop-dead date for publication of the new manual, which means that two sets of field trials and revisions must be completed by September 2012.

The time crunch only gives critics more fuel. Frances, on hearing of the trials’ delay, BlackBerried out a communiqué about the task force’s “Keystone Kops” missteps—the “Rube Goldberg design,” the “numerous measures signifying nothing,” the “criteria sets that are unusable because so poorly written.” All of which, he wrote, will lead to “a mad dash to dreck at the end.”

When the rough draft of the DSM-5 was released, in February 2010, the diagnosis that had galvanized Frances—psychosis risk syndrome—wasn’t included. But another new proposed illness had taken its place: “attenuated psychotic symptoms syndrome,” which has essentially the same symptoms but with a name that no longer implies the patient will eventually develop a psychosis. In principle, Carpenter says, that change “eliminates the false-positive problem.” This is not as cynical as it might sound: Carpenter points out that a kid having even occasional hallucinations, especially one distressed enough to land in a psychiatrist’s office, is probably not entirely well, even if he doesn’t end up psychotic. Currently, a doctor confronted with such a patient has to resort to a diagnosis that doesn’t quite fit, often an anxiety or mood disorder.

But attenuated psychotic symptoms syndrome still creates a mental illness where there previously was none, giving drugmakers a new target for their hard sell and doctors, most of whom see it as part of their job to write prescriptions, more reason to medicate. Even Carpenter worries about this. “I wouldn’t bet a lot of money that clinicians will hold off on antipsychotics until there’s evidence of more severe symptoms,” he says. Nonetheless, he adds, “a diagnostic manual shouldn’t be organized to try to adjust to society’s problems.”

His implication is that the rest of medicine, in all its scientific rigor, doesn’t work that way. But in fact, medicine makes adjustments all the time. As obesity has become more of a social problem, for instance, doctors have created a new disease called metabolic syndrome, and they’re still arguing over the checklist of its definition: the blood pressure required for diagnosis, for example, and whether waist circumference should be a criterion. As Darrel Regier points out, diabetes is defined by a blood-glucose threshold, one that has changed over time. Whether physical or mental, a disease is really a statistical construct, a group of symptoms that afflicts a group of people similarly. We may think our doctors are like Gregory House, relentlessly stalking the biochemical culprits of our suffering, but in real medicine they are more like Darrel Regier, trying to discern the patterns in our distress and quantify them.

The fact that diseases can be invented (or, as with homosexuality, uninvented) and their criteria tweaked in response to social conditions is exactly what worries critics like Frances about some of the disorders proposed for the DSM-5—not only attenuated psychotic symptoms syndrome but also binge eating disorder, temper dysregulation disorder, and other “sub-threshold” diagnoses. To harness the power of medicine in service of kids with hallucinations, or compulsive overeaters, or 8-year-olds who throw frequent tantrums, is to command attention and resources for suffering that is undeniable. But it is also to increase psychiatry’s intrusion into everyday life, even as it gives us tidy names for our eternally messy problems.

I recently asked a former president of the APA how he used the DSM in his daily work. He told me his secretary had just asked him for a diagnosis on a patient he’d been seeing for a couple of months so that she could bill the insurance company. “I hadn’t really formulated it,” he told me. He consulted the DSM-IV and concluded that the patient had obsessive-compulsive disorder.

“Did it change the way you treated her?” I asked, noting that he’d worked with her for quite a while without naming what she had.

“No.”

“So what would you say was the value of the diagnosis?”

“I got paid.”

As scientific understanding of the brain advances, the APA has found itself caught between paradigms, forced to revise a manual that everyone agrees needs to be fixed but with no obvious way forward. Regier says he’s hopeful that “full understanding of the underlying pathophysiology of mental disorders” will someday establish an “absolute threshold between normality and psychopathology.” Realistically, though, a new manual based entirely on neuroscience—with biomarkers for every diagnosis, grave or mild—seems decades away, and perhaps impossible to achieve at all. To account for mental suffering entirely through neuroscience is probably tantamount to explaining the brain in toto, a task to which our scientific tools may never be matched. As Frances points out, a complete elucidation of the complexities of the brain has so far proven to be an “ever-receding target.”

What the battle over DSM-5 should make clear to all of us—professional and layman alike—is that psychiatric diagnosis will probably always be laden with uncertainty, that the labels doctors give us for our suffering will forever be at least as much the product of negotiations around a conference table as investigations at a lab bench. Regier and Scully are more than willing to acknowledge this. As Scully puts it, “The DSM will always be provisional; that’s the best we can do.” Regier, for his part, says, “The DSM is not biblical. It’s not on stone tablets.” The real problem is that insurers, juries, and (yes) patients aren’t ready to accept this fact. Nor are psychiatrists ready to lose the authority they derive from seeming to possess scientific certainty about the diseases they treat. After all, the DSM didn’t save the profession, and become a best seller in the bargain, by claiming to be only provisional.

It’s a problem that bothers Frances, and it even makes him wonder about the wisdom of his crusade against the DSM-5. Diagnosis, he says, is “part of the magic,” part of the power to heal patients—and to convince them to endure the difficulties of treatment. The sun is up now, and Frances is working on his first Diet Coke of the day. “You know those medieval maps?” he says. “In the places where they didn’t know what was going on, they wrote ‘Dragons live here.’”

He went on: “We have a dragon’s world here. But you wouldn’t want to be without that map.”

Gary Greenberg (garygreenbergonline.com) is the author of Manufacturing Depression: The Secret History of a Modern Disease.