This paper is not intended to be a commentary on the substance of the Guidelines. It is simply an attempt to make the UGESP requirements related to content validity easier to understand than they would be by reading the Guidelines as published.
Not only are the references to content validity scattered throughout the Guidelines, but there are actually two varieties of content validity discussed in the Guidelines. I shall refer to them as "classic" and "extended" content validity. In addition the standards that are applied to them are, in some instances, substantially different. This has, not surprisingly, led to much confusion. After pointing out (in the Supplementary Information Section on page 38292) that:
There are three concepts which can be used to validate a selection procedure. These concepts reflect different approaches to investigating the job relatedness of selection procedures...a short straightforward definition of "classic" content validity is given:
In content validity, a selection procedure is justified by showing that it representatively samples significant parts of the job, such as a typing test for a typist.
(All verbatim quotations from the Guidelines are indented and shown in bold type.)
The key idea is that the test is constructed by taking a "representative sample." Thus in classic content validity the content of the test is the same as the content of the job. This is, of course, the source of the name. Sameness is about as strong as a relationship can get!
Later in the section, in an "Analysis of comments" subsection the adopting agencies responded to the comments that were received in response to the publication of a draft of the guidelines. The genesis of what I am calling "extended" content validity is explained (on page 38295) as follows:
8. Content validity. The Division of Industrial and Organizational Psychology of A.P.A. correctly perceived that the provisions of the draft guidelines concerning content validity, with their emphasis on observable work behaviors and work products, were "greatly concerned with minimizing the inferential leap between test and performance." That division expressed the view that the draft guidelines neglected situations where a knowledge, skill or ability is necessary to an outcome, but where the work behavior cannot be replicated in a test. They recommended that the section be revised.
We believe that the emphasis on observable work behaviors or observable work products is appropriate and that in order to show content validity, the gap between the test and performance on the job should be a small one. We recognize, however that content validity may be appropriate to support a test which measures a knowledge, skill or ability which is a necessary prerequisite to the performance of the job, even though the test might not be close enough to the work behavior to be considered a work sample, and the guidelines have been revised appropriately. On the other hand, tests of mental processes which are not directly observable and which may be difficult to determine on the basis of observable work behaviors or work products should not be supported by content validity.
Because I had chaired the committee that wrote the California Guidelines on Employee Selection Procedures, I was consulted by several of the people in the adopting agencies who were responsible for writing the federal guidelines. I can thus state based on direct knowledge that the concern about extending the definition of content validity to cover knowledges, skills and abilities (KSA's) was very great. They felt that they had to do it, but they were fearful that it would be badly abused. A substantial amount of the effort required to produce the final draft was devoted to the attempt to define a line between acceptable and unacceptable uses of content validity.
By bringing together the UGESP provisions that apply to each type of content validity I believe that it is easier to understand what the requirements are in each case.
should determine whether it is appropriate to conduct such a study in the particular employment context. A selection procedure can be supported by a content validity strategy to the extent that it is a representative sample of the content of the job.Thus using the classic approach is appropriate only if a test can be constructed by taking a representative sample of job content. The standards for demonstrating classic content validity are given in 14C(4):
To demonstrate the content validity of a selection procedure, a user should show that the behaviors demonstrated in the selection procedure are a representative sample of the behavior(s) of the job in question or that the selection procedure provides a representative sample of the work product of the job.
An example of a work product would be a properly welded angle joint as one item in a sample drawn from a welder's job which is then used in a test for selecting welders. Work behavior is defined in the definitions section (16Y) as follows (page 38308):
An activity performed to achieve the objectives of the job. Work behaviors involve observable (physical) and unobservable (mental) components. A work behavior consists of one or more tasks. Knowledges, skills, and abilities are not behaviors, although they may be applied in work behaviors.
The key word is activity. A work behavior is something that the worker does. Another part of the standards for demonstrating classic content validity require that all aspects of the test closely resemble the job (page 38302, column 3):
If a test purports to sample a work behavior or to provide a sample of a work product, the manner and setting of the selection procedure and its level and complexity should closely approximate the work situation. The closer the content and the context of the selection procedure are to work samples or work behaviors, the stronger is the basis for showing content validity. As the content of the selection procedure less resembles a work behavior, or the setting and manner of the administration of the selection procedure less resembles the work situation, or the result less resembles a work product, the less likely the selection procedure is to be content valid, and the greater the need for other evidence of validity.
The documentation section (15C) defines the elements of the showing that is required (page 38305). A job analysis (15C(3))must be provided:
The work behaviors, the associated tasks, and, if the behavior results in a work product, the work products should be completely described (essential). Measures of criticality and/or importance of the work behavior(s) and the method of determining these measures should be provided (essential).
The selection procedure and its content (15C(4)):
should be completely and explicitly described or attached (essential).
The documentation of the relationship between the selection procedure and the job (15C(5)) requires, among other things, that (page 38305):
The evidence demonstrating that the selection procedure is a representative work sample [or] a representative sample of the work behavior(s) . . . should be provided (essential). The user should identify the work behavior(s) which each item or part of the selection procedure is intended to sample or measure.The theme that unifies all of the content validity documentation requirements is that the user is expected to provide the detail and specificity needed to clearly relate the content of the test to the content of the job so that the "inferential leap" is very small.
Selection procedures which purport to measure knowledges, skills, or abilities may in certain circumstances be justified by content validity, although they may not be representative samples, if the knowledge, skill, or ability measured by the selection procedure can be operationally defined as provided in section 14C(4) below, and if that knowledge, skill, or ability is a necessary prerequisite to successful job performance.
Two conditions must be satisfied if KSA's are used. The first is that the KSA can be "operationally defined" using the restrictive standards in 14C(4) and the second is that the KSA is a necessary prerequisite to success on the job. Another method used "to put a fence around KSA's" as it was referred to at the time was to define the three terms in a much more restricted way than their standard dictionary definitions (Section 16 on page 38307):
M. Knowledge. A body of information applied directly to the performance of a function.
T. Skill. A present, observable competence to perform a learned psychomotor act
A. Ability. A present competence to perform an observable behavior or a behavior which results in an observable product.
O. Observable. Able to be seen, heard, or otherwise perceived by a person other than the person performing the action.
Notice that skill is restricted to psychomotor acts. A dictionary definition of psychomotor is: "Of or relating to movement or muscular activity associated with mental processes." The intent is to clearly exclude purely mental or social skills which must be validated by criterion-related strategies. Also note that skills and abilities are limited to observable events which must be perceived rather than inferred.
Many of the abuses of content validity are attributable to the use of broad dictionary definitions of KSA's which, if accepted by the adopting agencies, would allow claims of content validity in almost any situation. In many of these situations where content validity is inappropriately used, a criterion-related study would show that the test is, in fact, not job-related. This, of course, is why criterion-related strategies must be required for KSA's that involve inferences. The inferences may be incorrect.
The appropriateness standard (14C(1)) concludes with a paragraph describing some of the instances where extended content validity is not appropriate and therefore not acceptable:
A selection procedure based upon inferences about mental processes cannot be supported solely or primarily on the basis of content validity. Thus, a content strategy is not appropriate for demonstrating the validity of selection procedures which purport to measure traits or constructs, such as intelligence, aptitude, personality, commonsense, judgment, leadership, and spatial ability. Content validity is also not an appropriate strategy when the selection procedure involves knowledges, skills, or abilities which an employee will be expected to learn on the job.
In the section on standards for demonstrating content validity (14C(4)), the same principles are continued in the attempt to restrict the "inferential leap."
In the case of a selection procedure measuring a knowledge, skill, or ability, the knowledge, skill, or ability being measured should be operationally defined.
"Operationally defined" has a clear professional meaning. "Operationalism" is one of the major approaches to the issues that are involved in properly defining what is meant by a "scientific method." A dictionary definition of operationalism is, "The view that all theoretical terms in science must be defined only by their procedures or operations." First the operational definition of knowledge is given:
In the case of a selection procedure measuring a knowledge, the knowledge being measured should be operationally defined as that body of learned information which is used in and is a necessary prerequisite for observable aspects of work behavior of the job.
Skills and abilities are treated together:
In the case of skills or abilities, the skill or ability being measured should be operationally defined in terms of observable aspects of work behavior of the job.
In addition, to be content valid, a selection procedure measuring a skill or ability should either closely approximate an observable work behavior, or its product should closely approximate an observable work product.
In addition to providing operational definitions, there are two other required showings:
For any selection procedure measuring a knowledge, skill, or ability the user should show that (a) the selection procedure measures and is a representative sample of that knowledge skill, or ability; and (b) that knowledge, skill or ability is used in and is a necessary prerequisite to performance of critical or important work behavior(s).
Given the way that KSA's are defined and given the standards that are applied to them, most of the requirements in the documentation section (15C) can apply to both classic and extended content validity. There are, however, three important specific mentions of KSA's that impose additional requirements beyond the requirements for classic content validity. The first involves the job analysis (15C(3)) if KSA's are used:
Where the job analysis also identified the knowledges, skills, and abilities used in work behavior(s), an operational definition for each knowledge in terms of a body of learned information and for each skill and ability in terms of observable behaviors and outcomes, and the relationship between each knowledge, skill, or ability and each work behavior, as well as the method used to determine this relationship, should be provided (essential). The work situation should be described, including the setting in which the work behavior(s) are performed, and where appropriate, the manner in which knowledges, skills, or abilities are used, and the complexity and difficulty of the knowledge, skill, or ability as used in the work behavior(s).
The second adds to the requirements for information regarding the selection procedure and its content (15C(4)):
Where the selection procedure purports to measure a knowledge, skill, or ability, evidence that the selection procedure measures and is a representative sample of the knowledge, skill, or ability should be provided (essential).
The third addition involves the relationship between the selection procedure and the job (15C(5)):
The evidence demonstrating that the selection procedure is . . . a representative sample of a knowledge, skill, or ability as used as a part of a work behavior and necessary for that behavior should be provided (essential).