USN&WR Rankings-- How Does it Work?
It was interesting to see Monday’s AP story about a dozen university presidents leading a charge to boycott the US News & World Report ranking survey. We’ve already blogged about the focus placed on the rankings; today begins a three-part blog on how USN&WR rankings work, how to influence your placement and does anyone really make quantum leaps from year to year?
In recent months I tracked down information on the methodology of the often misunderstood peer assessment category within the USN&WR rankings and I obtained a copy of the survey instrument, which we are making available to you.
The comprehensive rankings are based upon the following categories and their assessment weight: peer assessment, 25%, retention, 20%; faculty resources, 20%; student selectivity, 15%; financial resources, 10%; graduation rate performance, 5%; and alumni giving, 5%. There is a paragraph within the methodology description that delineates the “peer assessment” rankings:
Peer assessment (weighting: 25 percent). The U.S. News ranking formula gives greatest weight to the opinions of those in a position to judge a school's undergraduate academic excellence. The peer assessment survey allows the top academics we consult—presidents, provosts, and deans of admissions—to account for intangibles such as faculty dedication to teaching. Each individual is asked to rate peer schools' academic programs on a scale from 1 (marginal) to 5 (distinguished). Those who don't know enough about a school to evaluate it fairly are asked to mark "don't know." Synovate, an opinion-research firm based near Chicago, collected the data; of the 4,089 people who were sent questionnaires, 58 percent responded.
So how does it work? I called and talked with Robert J. Morse, Director of Data Research for USN&WR (this means he is the rankings guru), and asked him for clarification on the peer assessment portion of the rankings. He was kind enough to walk me through the process used.
A school’s Carnegie classification is used as the basis for its overall categorization within the rankings. For explanation purposes, I’m going to use the liberal arts rankings to illustrate how the peer assessment calculation works. There are 215 schools within the liberal arts college category; USN&WR sends a survey to the president, provost and dean of admissions from each of the 215 institutions, for a total of 645 surveys distributed. The responses are used to calculate 25% of the overall ranking referenced above. The tiers used in the reporting of the rankings are based upon the overall score and reported annually as follows:
Top Schools: The top 50 percent of schools in this category, ranked numerically
Tier 3 Schools: The next 25 percent of schools in this category, listed alphabetically
Tier 4 Schools: The bottom 25 percent of schools in this category, listed alphabetically
So what’s on the survey? The three recipients from each school are asked to rank all 215 liberal arts colleges on a five-point scale ranging from distinguished to marginal, with “don’t know” serving as a sixth option, based upon “each program’s scholarship record, curriculum, and quality of faculty and graduates.” The scores are compiled, and the 25% of the overall ranking is calculated.
Here is the absolute coup of my telephone call: Mr. Morse was kind enough to send me a copy of the liberal arts survey used in 2006. Our blog software is unable to upload files, so please visit our resource page if you would like a copy of the survey. Enjoy!
Watch for the upcoming “how to influence the rankings” blog by Christopher Simpson, and Elizabeth Scarborough's entry detailing her analysis of schools' moves up and down the rankings ladder.