balance, they worked well notwithstanding the distractions caused by the errors.
Several Chiefs indicated awareness that supervisory style questionnaires had been seen in the firehouses before the examination, thus confirming the problem of security identified above. They indicated that a combination of the multiple choice part and the fire scene scenario part of the examination ought to produce a valid examination. This conclusion carries great weight with the Special Master.
B. The Administration of the Tests
There were a number of problems with the administration of the tests. Some of these can best, be described as inconsistencies in the ways monitors in various rooms handled the tests. With the elimination of the supervisory style questionnaire, the Special Master has concluded that none of the administrative problems was sufficiently serious to undermine the overall fairness of the tests. Essentially, the multiple choice questions were not adversely affected by any administrative actions, and the manner in which the grading key was arrived at minimizes significantly any problems in the administration of the supervisory style questionnaires.
Although it appears that the monitoring could have been improved significantly, no serious allegations of actual cheating have been made. The absence of such allegations is significant, particularly in view of the opportunity provided all firefighters to call any and all problems to the Special Master's attention.
The Battalion Chiefs anticipated some of the possible confusion that might have arisen with respect to the fire scene scenarios and the manner in which answers were to be recorded. Their foresight was not only commendable, but also of importance in eliminating the possible problems in the administration as well as the substance of the scenarios.
Were it not for the work performed by the Battalion Fire Chiefs, the tests prepared by the TDC might well have proved to be inadequate. The deficiencies that have been noted are deplorable and particularly difficult to understand in view of the fact that the TDC was appointed in 1986 and has had years to inform itself of the District's procedures and to refine tests.
The goal of security did deprive the TDC of the input of the Fire Department once the tests were drafted. And the time deadlines fixed by the settlement and the Court's approval may have required somewhat faster work than is typical. But, the Special Master joins the various firefighters who have stated that many of the errors could and should have been avoided.
Fortunately, the Battalion Fire Chiefs worked carefully on the multiple choice questions and saw that all objectionable questions were deleted from the scoring. This left the remaining questions as a fair, objective test of job knowledge. And, perhaps more fortunately, the Chiefs discovered all of the problems with the fire scene scenarios and worked around them in preparing grading keys that were sensitive the problems and sufficient to assure that test takers would be rewarded for answering questions as their superiors in the District would expect them to be answered pursuant to standard operating procedures.
Because the Battalion Chiefs saved the day, the Special Master has concluded that the multiple choice and fire scene scenarios constitute a valid examination. That it is a valid examination is the judgment of the TDC, and that judgment is supported by the Chiefs who were so integral a part of the grading process. A rank order list will be provided to the District by the TDC. Points will be added to the scores provided by the TDC, tiebreakers will be employed, and the District will announce promotional lists for sergeant, lieutenant and captain on or about February 1, 1993, as provided for in the settlement and in the order and Decree of the District Court.3a
No one need be satisfied that the substance of the tests or the administration was perfect; it was not. No one need be persuaded that the errors in substance or administration were unavoidable; many could have been avoided.
But, the Special Master has considered every alleged defect in the tests and administration, has determined that the supervisory style questionnaire may not be used because of a security breach (a judgment with which the TDC concurs), and that the remaining portions of the test are substantial and sufficient, particularly so in the eyes of the leaders of the Fire Department who carefully examined each aspect of the tests, and constitute a valid promotional examination.
The TDC has determined that the tests are job-related, the tests were constructed in a nondiscriminatory manner by experts agreed to by the parties, and the results should now be implemented so that promotions can be made. The lessons of these examinations should not be forgotten, however, and every effort should be made to assure that the substance and procedures are improved next time around. Indeed, the Special Master strongly believes that precautions must be taken prior to the administration of another set of examinations to assure that the quality of the tests and the manner in which they are administered are beyond reproach.
DATED: January 29, 1991
Stephen A. Saltzburg