On the solicitation of external letters of evaluation.

 

Jan de Vries

Vice Provost for Academic Affairs

 

University policy requires external letters of evaluation for all appointment cases, promotions to tenure and to professor, for advancement to Professor, Step VI, and for advancement to above-scale salary status.  Thus, it is possible that a Berkeley faculty member in the course of his or her career will have external letters assembled for review purposes at five different occasions.  In any given year, our various personnel actions involve the solicitation of at least 3000 letters from colleagues around the country and the world.  This enterprise requires a very substantial effort on the part of campus staff and faculty and imposes a real burden on academic colleagues.  It is important for both the quality and the efficiency of our academic assessment exercises that the solicitation of external letters be properly focused and that our use of the letters be realistic.

 

General observations on the value of external letters.

 

External letters are often thought to be of value because they are written by specialists familiar with the candidate’s work (more familiar than most of the candidate’s colleagues) who are outside the university (thus having no particular personal stake in the outcome) and who, as fellow professors, are motivated by their occupational calling to profess the truth.  Thus, one expects from external letters, valuable information that is expert, disinterested (objective), and truthful (candid). 

 

In truth, many if not most external letters fall well short of these ideals.  There is a serious selection bias inherent to the process:  favorably inclined persons are much more likely than others to be asked, and are also more likely to respond.  The letter writer has much less incentive to write candidly than the same person would have in writing a referee report to a journal or publisher: the writer is not likely to hold the interests of the institution making the solicitation over other interests that can militate against a candid assessment.  Related to this, there is a serious lack of clarity about the primary audience of the external letter.  The letters are ordinarily solicited by the candidate’s department, but they are also read by deans, faculty committees, and provosts.  Are the letter writer’s efforts intended to protect his field from incompetence or to protect his field from the administration?  This ambiguity undermines candor quite apart from the serious challenge that the erosion of confidentiality has come to pose for letter writers:  unless one writes an anodyne letter of warm appreciation, one cannot assume that the letter will not come to the attention of the candidate, in one way or another.  Finally, and related to this confidentiality issue, many letter writers resort to a sort of “coded language” which provides criticisms in the form of superficially supportive statements.  Many faculty pride themselves in their abilities to write in code and to decode the letters of others, but there can be no doubt that these language games diminish the clarity, and, hence, the value of many letters.

 

Purposes of external letters.

 

A valuable external letter is a serious attempt to assess the work of a scholar, placing that work in the context of the discipline on a national or international level appropriate to the candidate’s career stage.  It adds information to what a department, as a serious academic community, should be capable of determining for itself.  Unless a department is deemed to be incapable of distinguishing excellent from mediocre scholarship (which can happen) the role of external letters should be supplemental to a department’s assessment rather than a substitute for it.

 

This raises the key question:  when we solicit external letters, what do we hope to gain that cannot be determined by those closest to the case and with the largest stake in the outcome? 

 

1. Detailed assessment of academic work (Candidate’s contribution to scholarship)

2. Comparison of candidate with others of his/her cohort (Candidate’s standing relative to others)

3. Reputational verification (Candidate is/is not “one of us”.)

 

The value of external letters will depend in large part on the appropriateness of those asked to write (are they in a position to fairly assess the candidate; are they in a position to advise UC Berkeley?) and on the clear identification in the letter of the desired sort of information or assessment.

 

Appointment, non-tenure

 

            Letters for recent Ph.D.s are normally supplied by the candidate and consist of the narrow circle of scholars who at that point are familiar with the work:  the dissertation advisor, committee members, and other graduate school teachers and post-doc directors.  These are not external letters in the usual sense, since they are not solicited directly, but are important precisely because the hiring department at this point has access to a limited range of information about the candidate.  Useful letters will provide a detailed assessment of the candidate’s work as well as a comparison of the candidate with others of his/her cohort.  Unfortunately, these comparisons often take the form of “best student in past 20 years; in top 5% of my graduate students,” and lack credibility.  This sort of inflated pseudo-quantitative rhetoric is encouraged by the evaluation forms used for graduate student recommendations by many institutions, including Berkeley.  Since we do not directly solicit letters for most junior appointments, our ability to shape the character of these letters is limited.  Still, we should ponder how best to solicit credible information about achievement and promise from the recommenders of entry-level candidates.  One step to consider is asking external reviewers to compare the candidate with his or her cohort, or with recently hired new assistant professors at the reviewer’s own institution.

 

Appointment, tenure level.

 

            Candidates for tenure level appointments ordinarily provide a list of persons prepared to provide letters of reference.  In this respect, such candidates are similar to junior-level applicants, except that we expect the names to go beyond persons close to the candidate.  Such letters should be supplemented by those solicited by us from leaders in the relevant discipline who are asked to assess the suitability of the candidate for appointment at Berkeley.[1]  The aim should be to receive 6-7 useful letters.  This may require working from a list of approximately 12 names (including up to half drawn from the names submitted by the candidate, to the extent they are thought suitable). 

 

Berkeley solicitation letters typically invite referees to assess the candidate’s work and offer a reputational endorsement (items 1 and 3, above).  It would be most helpful to ask for cohort assessments, in part to calibrate the value of the other information being offered in these letters.

 

Promotion to tenure.

 

            External letters figure prominently in promotion cases, and nowhere are the stakes higher than in promotions to tenure.  Universities divide into two basic types with respect to the types of letters they solicit.  Some request that external referees supply a detailed assessment of the candidate’s work and a comparison of the candidate with his/her cohort.  Some explicitly list members of the cohort in their request.  The intention is to situate the tenure candidate among others who might be hired in his or her place; the tenure standard depends in part on the competition. 

 

            Most universities request that external referees supply a detailed assessment of the candidate’s work and a sort of reputational assessment, by asking whether the candidate “meets the standard for tenure.”  Since the standard for tenure is not defined and the question is put to persons already holding tenure, much hinges on who is asked. At Berkeley there is no consistency in the solicitation letters.  Not all units explicitly ask the referees to address this question.  Thus we not only face the problem of evaluating the evaluator (does he/she understand our standard?) but we face the danger of drawing false inferences from letters that do not directly address the question.

 

            External letters that address the comparison question are likely to be more informative that those that are asked to address the more subjective “tenure standard” question.  The former will provide information about other scholars, which can then be compared for insightfulness with the similar responses of other referees.  Berkeley does not use this form since the tenure decision here is based on an evaluation of the candidate’s performance relative to a standard.  Someone who meets the standard should be tenured, even if there is someone, somewhere, who meets it even more abundantly.  This makes it especially important to select external referees who share and understand our standards, to communicate those standards in the solicitation letter.  How can these standards be expressed clearly in a standard solicitation letter? Could a request for cohort comparisons be useful to us in establishing benchmarks for assessing levels of accomplishment that help define our “standards”?[2]

 

            Current practice at Berkeley asks that a list of external letter writers be generated by suggestions from the candidate and from the department on roughly a 50-50 basis.[3] A total list of approximately 12 names is suggested in order to obtain 6–7 letters, but more often than not these numbers are exceeded.  All those solicited should be listed in a table, identifying who nominated the evaluator (the candidate or the department/school), the “standing” of each evaluator, and in the case of those who do not write letters, the reason for their non-cooperation (to the extent possible).  These guidelines assist reviewers in interpreting the external letters, but they also invite over-interpretation: speculation about the meaning of an evaluator’s declination to write; discounting of the statements of letters written by those known to be the candidate’s nominees. 

 

Summary:  Aim to submit approximately 6-7 external letters from persons who understand Berkeley’s tenure standards.  Solicit a letter that: 1. delves into the substance of the candidate’s academic work, highlighting its contribution to scholarship,  2. describes how this work satisfies Berkeley’s tenure standards, and 3. addresses the candidate’s prospects for future productivity and creativity.

 

Promotion to Professor.

 

            External letters are used in promotions to professor is much the same way as in promotions to tenure.  The same procedures are used as described above, and the same approximate number of letters is expected.  Since candidates for this promotion are already tenured, the stakes are less high.  But this does not mean that letters are more objective or more useful.  The standard for promotion to professor is, if anything, more difficult to articulate than that for tenure.  Moreover, the standard evidently changes the longer an associate professor remains in rank. 

 

            Requests to submit external letters are similar in form to those for promotions to tenure:  they should request an evaluation of the candidate’s work during their tenure as associate professor, and an assessment of whether the candidate meets the standard for promotion.  In the case of promotion to professor, this assessment tends to become a rather subjective reputational verification: “we full professors recognize the candidate as worthy of being one of us.”  For this reason, it is important to solicit information that substantiates such claims: graduate student mentorship, service to and standing within one’s profession, journal refereeing, etc. 

 

Summary:  Aim to submit approximately 6-7 external letters from persons who understand Berkeley’s standards for promotion to full professor.  Solicit a letter that: 1. delves into the substance of the candidate’s academic work during their tenure as associate professor, highlighting its contribution to scholarship, and 2. probes the claims of “ripeness” for promotion with supporting information.

 

 

Advancement to Professor, Step VI and Above-scale salary.

 

Thus far we have considered letters that are solicited by nearly all American universities.  We turn now to solicitations made only by the campuses of the UC system.  Because these solicitations are not common to American higher education, it is of special importance that the solicitation letter make clear what type of assessment is required, and that those solicited are actually in a position to make the required assessment.

 

Step VI.

 

The concept of the “Step VI barrier” was introduced in 1969, with the addition of a sixth step to the university’s rank and step system.  At that time, the special review required to determine who should be advanced to Step VI was akin to the current above-scale review, since it led to the system’s highest step.  Over time, as additional steps were added to the Professor rank, Step VI came to be positioned approximately at the mid point of most faculty members’ tenure as Professor, and has become, in practice, a barrier to advancement for faculty whose research productivity and/or quality has fallen below the high levels of expectation we hold for senior faculty at Berkeley.  Teaching and service can also play a role in this review, of course, but most faculty who remain at Step V for extended periods of time do so for issues related to their research record.

 

While I have cast the functional role of Step VI review in negative terms (a relatively weak research record prevents advancement), the APM definition of the standard for Step VI advancement is stated positively:  “highly distinguished scholarship, highly meritorious service, and evidence of excellent University teaching.”  It goes on to advise reviewers to “require evidence of excellence and high merit in original scholarship or creative achievement, teaching and service and, in addition, great distinction, recognized nationally or internationally, in scholarship or creative achievement or in teaching.” [APM 220]

 

The careful reader of the APM language will note that it restates the same point three times.  However, while the first two statements ask for excellent scholarship and service and teaching, the third statement asks for great distinction, recognized (inter)nationally in scholarship or teaching.  The APM language has been an ongoing source of confusion and discord within the UC System.  The language has been revised in subtle ways at least four times since 1969, and the very existence of a special Step VI review has been challenged, most recently in an Academic Council Task Force report of 2004.

 

In view of the difficulty UC faculty have with the Step VI concept, how should colleagues external to the University be asked to advise us on this advancement? Indeed, should external letters be solicited at all?  At present, the APM appears to mandate such letters, but perhaps we should work to revise university policy on this point.

 

There appears to be only one aspect of a Step VI advancement that would appear to call for external letters:  the validation of “great distinction, recognized nationally or internationally, in scholarship or teaching.”  So long as external letters are solicited, they should ask specifically for an assessment of the quality of the candidate’s work and its influence (rather than simply for an affirmation of the candidates [inter]national stature), and such requests should be made of persons well placed – by their own acknowledged professional standing – to make such an assessment. 

 

Summary:  Letters should be solicited from a small group of carefully selected scholars. What follows is a proposed text for the solicitation letter:

 

Prof. XXX is being considered for advancement to Prof., Step VI in the University of California’s salary scale.  Please note that Prof XXX already holds the rank of Professor, having been promoted to that rank in [YEAR].  Steps VI and above (there are nine in all) on the University’s salary scale are reserved for faculty who have attained “great distinction, recognized nationally or internationally, in scholarship or teaching.”  We hope you can provide us with a brief evaluation of Prof. XXX’s achievements, especially on the quality of his/her scholarship and its influence.  A comparison with other prominent scholars at his/her career stage would also be helpful.

 

Above-scale Salary.

 

The criteria for advancement beyond the university salary scale to an above-scale salary are stated in the APM:

 

APM 220-18-b-(04):  Advancement to an above-scale salary is reserved for scholars and teachers of the highest distinction whose work has been internationally recognized and acclaimed and whose teaching performance is excellent. …Mere length of service and continued good performance at Step IX is not justification for further salary advancement.  There must be demonstration of additional merit and distinction beyond the performance on which advancement to Step IX was based. (Emphasis added.)

 

Clearly, external evaluators are in no position to address the second underlined criterion.  They must be asked to focus on the first underlined passage, yet how is this to be distinguished from advancement to Step VI?  Taken literally, step VI requires “recognized” scholarship and great distinction, while above-scale asks for highest distinction and “recognized and acclaimed” scholarship plus excellent teaching.

 

Summary:  Once again, letters should be solicited from a small group of carefully selected scholars: in this case, from persons who would themselves be above-scale if at Berkeley.  It is not obvious why the candidate for advancement should contribute names for this review.  A proposed text for the solicitation letter:

 

Prof. XXX is being considered for a special advancement, to a salary that exceeds the highest step of the University of California’s salary scale. Advancement to this “above-scale” status is reserved for scholars and teachers of the highest distinction whose work has been internationally recognized and acclaimed and whose teaching performance is excellent. We hope you can provide us with a brief evaluation of Prof. XXX’s achievements and his/her scholarly influence.  A comparison with other leading scholars in his/her discipline would also be helpful.

 

In general, the value of letters for Step VI and Above-scale advancement will depend heavily on the careful selection of evaluators.  A small number (three or four) from leading scholars will have greater value than a large number (twelve or more, as is now common) from the candidate’s close colleagues and from institutions that may have notions very different from Berkeley as to what constitutes “great distinction.”

 

Conclusions.

 

1.  All of us engaged in academic personnel assessment need to understand the very limited reliability of external letters.  Letter writers do not necessarily know enough to perform the evaluation we desire, do not necessarily know whom they are addressing, and do not necessarily have our institutional interests in mind.

 

2.  The place of external letters in Berkeley’s academic evaluation processes should be limited and strategic.  They should supplement the considered judgment of our own academic units and review bodies rather than define the outcome.  More importantly, we should not hide behind these letters, abdicating the use of our own judgment because of the communis sensus of some external community of correspondents.

 

3.  Letters should be solicited from persons who are in a position to respond knowledgably to the specific evaluative task we put before them and our solicitation letters should be as clear as possible about the sort of information/evaluation we require.  Quality is more important that quantity.

 

4.  We routinely solicit letters requesting assessments of work and of reputation/standing.  More use should be made of cohort assessment in our external letter requests.

 

Question

 

5.  We solicit letters for five types of academic personnel action.  Of these, the Step VI barrier is certainly the most confusing to external informants and the least necessary to the decision at hand.  Should we consider ceasing this practice?