Being Human

I've just a read a fantastic New York Times article from last year on the ongoing $1,000,000 Netflix challenge to create an algorithm that will predict what unseen films customers will liked based on their past preferences.

As well as an interesting insight into how companies are trying to guess our shopping preferences it is also a great guide to one of the central problems in scientific psychology: how we can reconcile numerical data with human thought and behaviour.

The Netflix prize teams have a bunch of data from customers who have rated films they've already seen and they have been challenged to write software that predicts future ratings.

Part of this process is hypothesis testing, essentially an experimental approach to find out what might be important in the decision process. For example, a team might guess that women will rate musicals higher than men. They can then test this prediction out on the data, making further predictions based on past conclusions, theories or even just hunches.

The other approach is to use mathematical techniques that look for patterns in the data. To use the jargon, these procedures look for 'higher order properties' - in other words, patterns in the patterns of data.

Think of it like looking at the relationship between different forests rather than thinking of everything as individual trees.

The trouble is, is that these mathematical procedures can sometimes find reliable high level patterns when it isn't obvious to us what they represent. For example, the article discusses the use of a technique called singular value decomposition (SVD) to categorise movies based on their ratings;

There’s a sort of unsettling, alien quality to their computers’ results. When the teams examine the ways that singular value decomposition is slotting movies into categories, sometimes it makes sense to them — as when the computer highlights what appears to be some essence of nerdiness in a bunch of sci-fi movies. But many categorizations are now so obscure that they cannot see the reasoning behind them. Possibly the algorithms are finding connections so deep and subconscious that customers themselves wouldn’t even recognize them.

At one point, Chabbert showed me a list of movies that his algorithm had discovered share some ineffable similarity; it includes a historical movie, “Joan of Arc,” a wrestling video, “W.W.E.: SummerSlam 2004,” the comedy “It Had to Be You” and a version of Charles Dickens’s “Bleak House.” For the life of me, I can’t figure out what possible connection they have, but Chabbert assures me that this singular value decomposition scored 4 percent higher than Cinematch — so it must be doing something right. As Volinsky surmised, “They’re able to tease out all of these things that we would never, ever think of ourselves.” The machine may be understanding something about us that we do not understand ourselves.

In these cases, it's tempting to think there's some deeply psychological property of the film that's been captured by the analysis. Maybe all trigger a wistful nostalgia, or perhaps each represents the same unconscious fantasy.

It could also be that each is under 90 minutes, or comes with free popcorn. It could even be that the grouping is entirely spurious and represents nothing significant. Importantly, the answer to these questions is not in the data to be discovered, we have to make the interpretation ourselves.

Experimental methods go from meaning to data, while exploratory methods go from data to meaning. Somewhere in the middle is our mind.

The Netflix challenge is this problem on steroids and the NYT piece brilliantly explores the practical problems in making sense of it all.

Link to NYT piece 'If You Liked This, You’re Sure to Love That'

Goshen College says goodbye to five faculty members
Goshen College News
He taught such courses as general psychology, developmental psychology, abnormal psychology, personality theory and contemporary viewpoints. ...

MyNews.in
Stages and learning approach that suits students
MyNews.in
Jaipur: If we go by the developmental psychology, which is the most recognized and respected knowledge area of science of mind and pedagogy; we can divide ...

I've just found a curious historical article discussing the early debates over whether anaesthesia could trigger sexual dreams in patients. As this was Britain in the 1800s, much of the fuss was centred on whether the Victorian lady was actually capable of such things:

In January, 1849, a discussion of “Chloroform in Midwifery” occurred during a meeting of the Westminster Medical Society in England. One of the physicians, Dr. G. T. Gream (Obstetrician, Queen Charlotte’s Lying-In Hospital, London, England) enumerated several reasons why he did not think that chloroform was appropriate for obstetric use, and in so doing, he “alluded to several cases in which women had, under the influence of chloroform, made use of obscene and disgusting language. This latter fact alone he considered sufficient to prevent the use of chloroform in English women”...

In a subsequent issue of The Lancet, notes from the Medico-Chirurgical Society of Edinburgh of February 7, 1849, were published. Sir James Young Simpson (Obstetrician, Edinburgh, Scotland, developer of chloroform anesthesia, and President of the Royal College of Physicians in 1849; 1811–1870) stated that after 15 months of use in thousands of cases, “he had never seen, nor had he ever heard of any other person having seen, any manifestation of sexual excitement result from the exhibition of chloroform…. The excitement, he was inclined to think, existed not in the individuals anesthetized, but was the result of impressions harbored in the minds of the practitioners, not in the minds of the chloroformed.”

Of course, there are some cases of criminal clinicians who have used sedation to attack their patients, but we now know that some modern anaesthetics, particularly midazolam and propofol, really do seem to be involved in causing sexual hallucinations and imagery in patients.

As far as I know, the reason why certain anaesthetics spark sexual imagery is still a mystery.

As we discussed earlier this year, the introduction of anaesthesia was controversial, partly because of the belief that pain was useful in keeping people alive and partly because experiencing pain was considered morally virtuous.

Link to PubMed entry for paper.

The next-to-latest xkcd:

There must have been many SF stories based on the premise that searchers for ET signaling Just Don't Get It, but I can't think of any.

Probably related: I can't think of any scientifically-plausible way to cash that premise in, other than dull things like really slow-moving sentients using nanohertz-range modulation of a spectrally prominent carrier.

No doubt some readers can remedy these deficiencies of memory and imagination.

[Update — I didn't mean stories about how ETs might actively prevent us from seeing their signals, or the signals of others; nor stories about how ETs might be so alien that communication would be impossible, so that even when we find their signals, we can't make sense of them, or perhaps completely misunderstand them. Those are both good themes, found in lots of stories that I can think of, and no doubt many more that I don't know.

What I had in mind was something more strictly analogous to the plight of the ants, who have looked carefully for chemical signals, but (presumably) have failed to consider the intrinsically implausible hypothesis that an intelligent and social species might make use of frequency-and-amplitude modulation of air-pressure variation at time scales of a hundred microseconds to 10 seconds or so.]

POLITICS-ITALY: Where Are the Women? - Part 1
Inter Press Service
Chiara Volpato, professor of social psychology at the Milano-Bicocca University, sees "historic factors" in the current impasse. ...
POLITICS-ITALY: Don't Even Speak of Equality! - Part 2Inter Press Service

all 3 news articles »

Historical and Contemporary Perspectives From Different Disciplines
Metapsychology
His main fields of interest are cross-cultural and social psychology, especially the development of social cognition. He is the author of A History of ...

Review by Lucas Keefer
Metapsychology
Social psychologists will be challenged by Greenwood's strong claim that "contemporary American social psychology has virtually abandoned the study of the ...

Rather mysteriously, no one can find anyone who has been blind from birth and has later been diagnosed with schizophrenia. I found this interesting snippet from a short article from Behavioral and Brain Sciences:

Five independent searches, varying considerably in scope, methods, and population, failed to identify even one well-defined co-occurrence of total blindness and schizophrenia (Abely & Carton 1967; Chevigny & Braverman 1950; Feierman 1982; Horrobin 1979; Riscalla 1980). We dedicated portions of 2000 and 2001 to e-mail and postal mail surveys of relevant professionals; e-mail and telephone discussions with officials of health, mental health, blindness, and schizophrenia organizations and research institutes; and extensive keyword probes of Medline, PsychINFO, and ScienceDirect databases. Some ambiguity was introduced by very low return rates for our surveys, but the consistent result of all these inquiries was that no instance of totally blind/schizophrenic co-occurrence was found.

The authors give a speculative hypothesis that this is because visual experience during development helps to shape brain pathways heavily reliant on the neurotransmitter glutamate and the NMDA receptor.

It is widely accepted that this system plays a role in the development of psychosis but the idea that it is shaped by visual experience to the point where schizophrenia is impossible is just an interesting idea at the present time.

That's not to say no-one with schizophrenia is blind (in fact, there are numerous tragic cases of self-blinding) but it is still the case that no-one has yet produced an example of someone who has been blind from birth who later has become psychotic.

If you do hear of anyone, get in touch, contact your nearest cognitive scientist, or if you are a researcher yourself, write up a case study, as it's an interesting anomaly in the medical literature.

Link to summary of paper on blindness and schizophrenia.

Good Question: Why Do Kids Ask Tough Questions?
WCCO
Dr. John Tauer studies social psychology, examining why we do the things we do. "The inherent needs are the need to feel good and the need to be right," he ...

and more »

Normal 0 0 1 572 3263 27 6 4007 11.1282 0 0 0

I got a nice email from Joshua Fruhlinger about my post on Harper’s denial about having any content in their magazine. It seems that I’m a bit in the dark about how this word is being used in the tech industry these days, which shouldn’t surprise anyone. Here’s what Joshua wrote to me:

The ad is a somewhat cheeky response to a particular way that the word “content” has come to be used in the publication industry in the last decade or so. As the internet has become the main (or at least the most novel and talked about) publishing platform, the tech folks who are designing the new infrastructure tend to label and lump together as “content” the stuff that isn’t in their department – the actual text, video, audio, or what have you that various exciting new publishing platforms are designed to present.

I see, so the tech folks think of “content” as the stuff that’s not in their infrastructure. So far, so good. Joshua continues:

Some people whose job it is to create that content sort of see this terminology as a symptom that the shoe is on the wrong foot.

I can certainly understand that reaction. Joshua explains more:

Web sites can become well known and get venture funding because they’re designed or structured in some exciting way, and are sold around the design and structure, because the people who designed them and structured them are the ones selling them. The people in the traditional media would normally be thought of as the whole purpose and selling point of the exercise – and the writers or filmmakers or artists are just called “content providers,” and are something of an afterthought.

It’s something of a specialized debate, and it was probably a more keenly felt issue back during the first dot-com boom, when people were really trying to sell web sites just because they were web sites, without regard for what information they actually conveyed. But Harper’s is basically saying that, for them, the actual writing is at the forefront of their enterprise; they’re not focusing on improving the packaging of some “content” they’re buying from the lowest bidder.

If I understand this properly, Harper’s wants to tell us that what really matters is the original content that is in the magazine, not the way the magazine is packaged and not the content that they could (but don’t) buy from others. So why then, does the ad tell us that Harper’s has no content? Why not tell us that it contains only original content written by their good writers? But the really interesting part comes next in Joshua’s email:

You have to be a little involved in the industry to get the nuance, which is always a mistake for an ad. People who write ads assume their readers are going to be just like them.

Ah-hah. Now we may be getting to the problem. Insiders use language the way other insiders use it. It’s efficient and appropriate for medical specialists to use shorthand terms, abbreviations, and terminology among themselves. There’s nothing wrong with this as long as they don’t expect outsiders to understand them. The same goes for many lawyers, accountants, mechanics, engineers, and, I hate to admit it, linguists.

The ad writing team may have used the meaning of “content” in the way those in the design and packaging business use it – to mean the opposite of content packaging. But, like Joshua, I wonder if Harper’s thought it was using “content” to mean original writing rather than stuff purchased from outside sources. The problem with both senses is that most of us (I think) don’t even know about the specialized meaning of “content” held by the packaging industry and we can be equally confused if Harper’s really meant that they use only original material. Either way, by advertising that Harper’s has no content, many readers are likely to be mystified.

Insider words can get you in trouble. Good ads see life from the perspective of their readers. Like Joshua, I still believe this ad didn’t do this.

In response to my (admittedly feeble) garden-path post a couple of days ago, Tim Leonard writes:

Ha!  That's not a garden-path sentence.  This is a garden path sentence:

"Police in Washington state captured a schizophrenic killer who had escaped during an outing from the mental hospital where he had been committed to a state fair."

Source: Dean Schabner, "Escaped Insane Killer Captured After Four-Day Manhunt", ABC News, 9/20/2009.

And for lagniappe, I don't think we've previously noted the "That's not a ___, this/that is a ___" phrasal template, which (I think) originated with this passage in the movie Crocodile Dundee:

I love this abstract of a scientific paper on 'Neurological Problems of Jazz Legends'. It's full of medical jargon but if you read it out loud it sounds like a beat poem. Try it with the same rhythm as Ginsberg's poem Howl.

Neurological problems of jazz legends

J Child Neurol. 2009 Aug;24(8):1037-42.

Pearl PL.

A variety of neurological problems have affected the lives of giants in the jazz genre.

Cole Porter courageously remained prolific after severe leg injuries secondary to an equestrian accident, until he succumbed to osteomyelitis, amputations, depression, and phantom limb pain.

George Gershwin resisted explanations for uncinate seizures and personality change and herniated from a right temporal lobe brain tumor, which was a benign cystic glioma.

Thelonious Monk had erratic moods, reflected in his pianism, and was ultimately mute and withdrawn, succumbing to cerebrovascular events.

Charlie Parker dealt with mood lability and drug dependence, the latter emanating from analgesics following an accident, and ultimately lived as hard as he played his famous bebop saxophone lines and arpeggios.

Charles Mingus hummed his last compositions into a tape recorder as he died with motor neuron disease.

Bud Powell had severe posttraumatic headaches after being struck by a police stick defending Thelonious Monk during a Harlem club raid.

If beat poetry aint your bag, try dropping it to the beat of Gang Starr's wonderful track Jazz Thing, which, among other things, taught me the recondite word 'recondite'.

Link to PubMed entry for 'Neurological Problems of Jazz Legends'.
Link to Gang Starr's Jazz Thing.

I've just discovered that the classic James Dean movie Rebel Without a Cause was inspired by a true life account of a psychiatrist's analysis of a young 'psychopath'.

According to this 1944 article from Time magazine, the book, called Rebel Without A Cause: The Hypnoanalysis of a Criminal Psychopath, was written by psychiatrist Mitchell Lindner and gave the public "one of the few play-by-play accounts of a psychoanalytic treatment ever published".

Lindner's subject is Harold, 21, serving a long term for a serious, unnamed crime. Harold, the son of a bull-tempered Polish laborer who speaks no English, has been in trouble with the police, mostly for pilfering, since the age of twelve. His most conspicuous psychopathic symptom was a constant blinking of his eyes.

Lindner began in orthodox analytic fashion by having the boy lie on a couch and encouraging him to talk freely. (Lindner got his transcript via a microphone concealed in the couch. Told about this at the end of the analysis, Harold himself urged the analyst to publish the record.) Without much hesitation, Harold gave the details of a hair-raising career of gun-toting, stealing, vandalism, fornication. Like all psychopaths, Harold was "a rebel without a cause, a revolutionary without a program," a grownup infant with no self-restraint and a craving for instant satisfactions.

If you're puzzled by the term 'hypnoanalysis' in the title, it was a form of Freudian psychoanalysis but where the patient was put into a hypnotic trance supposedly to encourage free association and facilitate access to the unconscious.

The idea was that it was a type of cranked up psychoanalysis that could give quicker results but, as the article notes, it was considered rather suspect by the forever orthodox Freudians.

An alternative juiced up version was 'narcoanalysis' that typically used barbiturate drugs for the same reason. This was the origin of the truth drug as it was wrongly thought that people hiding the truth might let it slip through if their unconscious was 'loosened' somewhat.

The connection between the book and the film seems to be fairly cursory though, as while the movie shares the title and is also about an antisocial young man, it's otherwise quite different.

Link to 1994 Time article on the book.

CMU The Tartan Online
Athlete Profile: Caroline Size
CMU The Tartan Online
Size is a business major who is now picking up an additional major in psychology, a decision she credits to social psychology, her favorite course so far. ...

and more »

The Boston Globe covers several recent studies that have been able to work out sensitive personal details from information made public on social networking sites, possibly including your sexual orientation.

As we discussed earlier this week, huge amounts of information can be gleaned about your life through social network analysis simply from the patterns in your interactions.

In computer security and counter intelligence this is part of a technique called traffic analysis which has a long history in law enforcement. For example, before the days of the internet the UK police would use the Harlequin system to work out social networks from phone call patterns as these were much easier to obtain than court orders allowing phone taps.

Now, we put much of this information online ourselves but are unaware of how much the explicit personal information that we deliberately keep private is still available implicitly in the public data trail.

Sociologists have known this for years but the rapid spread of electronic communication has spurred the development of analysis tools as well as providing the real world data on which it can be applied.

Discussions of privacy often focus on how to best keep things secret, whether it is making sure online financial transactions are secure from intruders, or telling people to think twice before opening their lives too widely on blogs or online profiles. But this work shows that people may reveal information about themselves in another way, and without knowing they are making it public.

Who we are can be revealed by, and even defined by, who our friends are: if all your friends are over 45, you’re probably not a teenager; if they all belong to a particular religion, it’s a decent bet that you do, too. The ability to connect with other people who have something in common is part of the power of social networks, but also a possible pitfall. If our friends reveal who we are, that challenges a conception of privacy built on the notion that there are things we tell, and things we don’t.

Link to Globe article on social networks and personal info (via MeFi).

What may be the most widely-discussed statistical over-interpretation in history is coming around for the third time. The first gust front of commentary blew in with David Leonhardt in the NYT Business Section in September of 2007, echoed a few days later by Steven Leavitt in the Freakonomics blog. In May of 2009, Ross Douthat's NYT column recycled the same research for another round of thumb-sucking. And the same material has just been promoted again by Arianna Huffington ("The Sad, Shocking Truth About How Women Are Feeling", "What's Happening to Women's Happiness?", etc.), with an assist by Maureen Dowd ("Blue is the New Black", 9/19/2009).

Ms. Huffington tells us that

According to study after study, women are becoming more and more unhappy. This drop in happiness is found in women across the social and economic landscape. It doesn't matter what their marital status is, how much money they make, whether or not they have children, their ethnic background, or the country they live in. Women around the world are in a funk.

And it's not because of the multitude of crises we are facing. Women's happiness has been on a downward trend since the early 1970s, when the General Social Survey, a landmark study, began examining the social attitudes of women and men — who, by the way, have gotten progressively happier over the years.

MoDo chimes in:

According to the General Social Survey, which has tracked Americans’ mood since 1972, and five other major studies around the world, women are getting gloomier and men are getting happier.

Before the ’70s, there was a gender gap in America in which women felt greater well-being. Now there’s a gender gap in which men feel better about their lives.

People love this story. They love to speculate about the reasons for the trend — the favorites are variants of "too much feminism" and "not enough feminism" — and to tell us about their own happiness or lack thereof. Tens of thousands of readers, across the repeated reprises of this story in the mass media, have commented on various newspaper and weblog sites.  In a certain sense, this tidal wave of response validates the story, which clearly resonates with something in the spirit of the times.  But in fact, the empirical basis for all this fuss is so thin as to be practically non-existent.

I'll focus on the General Social Survey results, since I've looked into them in detail, but the rest of the worldwide background is similar.

One way to see what's happening is to look at this graph of General Social Survey results, from the preprint that kicked it all off in 2007:

Those who prefer tables may like to see it this way (taken from an earlier post on the subject):

If we sum up all the GSS responses across years, we get these proportions of answers to the question "Taken all together, how would you say things are these days — would you say that you are very happy, pretty happy, or not too happy?"

Very happy Pretty happy Not too happy Male 31.2% 56.7% 12.1% Female 32.4% 55.1% 12.5%

In the responses for 1972, 1973, and 1974 (the earliest dates available), the overall proportions were:

Very happy Pretty happy Not too happy Male 31.9% 53.0% 15.1% Female 37.0% 49.4% 13.6%

In the responses for 2004, 2006, and 2008 (the most recent dates available), the proportions were:

Very happy Pretty happy Not too happy Male 29.8% 56.1% 14.0% Female 31.2% 54.9% 13.9%

The best way to describe this, I think, would be to say something like:

In the early 70s, women self-reported their happiness at levels somewhat higher than men did. Specifically, 5.1% more of the women reported themselves "Very happy", while 1.5% fewer reported themselves "Not too happy".

30-odd years later, in the mid 00s, women's self-reported happiness was closer to men's, though it was still slightly higher. 1.4% more of the women reported themselves "Very happy", while 0.1% fewer reported themselves "Not too happy".

To Arianna Huffington, this means that "women are becoming more and more unhappy", while "men … have gotten progressively happier over the years". To Maureen Dowd, this means that "Before the ’70s, there was a gender gap in America in which women felt greater well-being. Now there’s a gender gap in which men feel better about their lives."  Ross Douthat described these numbers with the generalization "In postfeminist America, men are happier than women."

All of these statements are either false or seriously misleading.  Maybe, if you look at the data through a sophisticated statistical model, you can support a conclusion about the relative signs of the long-term-trends for males and females.  But any way you slice and dice it, there's not much there there.

I've cited the earlier stages in this discussion as motivation for a moratorium on using generic plurals to describe small statistical differences.  The contributions of Arianna Huffington and Maureen Dowd are, if anything, even better arguments for this (hopeless) cause.

Past LL happiness-gap posts:

"The 'Happiness Gap' and the rhetoric of statistics" (9/26/2007)
"Gender-role resentment and Rorschach-blot news reports" (9/27/2007)
"Why are economists so misleading?" (10/1/2007)
"The gender happiness gap: statistical, practical and rhetorical significance", 10/4/2007
"The happiness gap returns" (7/26/2008)
"The happiness gap is back" (5/26/2009)
"Women's happiness and pundits' accuracy" (5/27/2009)

Also maybe relevant, if you're not completely sick of the whole topic:

"Myth is truth (p < .05)" (12/23/2007)

If you want to do your own modeling, a csv file of the GSS happiness answers is here (some background on the data is here).

[I also note a certain lack of journalistic courtesy — Douthat didn't mention that Leonhardt and Leavitt had covered the same material a year and a half earlier, and Huffington doesn't mention Douthat, Leavitt or Leonhardt.  I know that journalists don't need footnotes, but if a pundit reprises a story that's previously been featured by other pundits, doesn't journalistic etiquette suggest a tip of the hat to the earlier authors? Dowd does cite Huffington, which seems like the normal practice.]

Quality Of Early Child Care Plays Role In Later Reading, Math ...
Zikkir World (blog)
... in middle childhood,” according to Eric Dearing, associate professor of applied developmental psychology at Boston College and the study's lead author. ...

and more »

BBC News
Delinquents 'misinterpret anger'
BBC News
Professor Karen Pine, an expert in developmental psychology at the University of Hertfordshire, said the ability to "read" a person's emotions from their ...

and more »

That's not a-the-indefinite-article, it's a-the-immediate-future-marker, as in Kanye West's infamous "I'm a let you finish" interruption at the MTV awards. Steven Poole at Unspeak has a poll, where you can register your preference for how to spell it. (So far, "I'ma" has a plurality of 45%, with "I'm'a" next at 20%.)

Steven links to the discussion that Ella and I had about this back in 2005.