/
Psychometric Validity and Reliability
Save to my account
Sign up
Psychometric Validity and Reliability
Chapter 12 Flashcards
Study
1
Question
How does construct underrepresentation differ from construct irrelevance in counseling assessment?
Page 1
Answer
Construct underrepresentation means a measure is too narrow and omits vital facets of a construct. Construct irrelevance means scores are influenced by extraneous variables unrelated to the target construct. Contrast hook: Underrepresentation leaves the target basket half-empty; irrelevance dumps foreign trash into the basket.
2
Question
What primary problem arises when a measure exhibits construct underrepresentation?
Page 1
Answer
The instrument lacks comprehensiveness because it omits items that accurately and completely assess the construct of interest. Memory hook: Imagine a trauma clinician measuring only sweating palms while completely ignoring weeping eyes.
3
Question
How does reading comprehension operate as construct irrelevance on a mathematics exam?
Page 1
Answer
Poor reading comprehension impedes students from deciphering word problems even when their mathematical understanding is sound. The test inadvertently scores language ability instead of pure math ability. Memory hook: A math compass trapped inside a heavy dictionary.
4
Question
Why can counselor ratings introduce construct irrelevance on the Crisis Stabilization Scale?
Page 1
Answer
Counselors completing the instrument may exhibit subjective response bias by overreporting or underreporting therapeutic progress. This introduces systematic rater error that is extraneous to the adolescent client's true stabilization.
5
Question
Which vivid contrast hook distinguishes construct underrepresentation from construct irrelevance?
Page 1
Answer
Underrepresentation leaves the target basket half-empty, whereas irrelevance dumps foreign trash into the basket.
6
Question
What five essential sources of validity evidence were established by AERA et al. (2014)?
Page 1
Answer
The five sources are evidence based on test content, response processes, internal structure, relationships to other variables, and consequences of testing. Acronym hook: CRISC (Content, Response, Internal structure, Similar/other variables, Consequences).
7
Question
How do researchers empirically demonstrate evidence of test content when constructing a measure?
Page 2
Answer
Researchers conduct exhaustive literature reviews and solicit expert reviewers to inspect item themes, wording, and format. They refine or drop ambiguous items to verify alignment with the theoretical model. Memory hook: Content Connoisseurs wielding blue editorial pens.
8
Question
Why is readability analysis considered an element of evidence based on test content?
Page 2
Answer
Instruments exceeding an eighth-grade reading level can confound responses from the general public. High reading difficulty prevents respondents from comprehending items, undermining content validity.
9
Question
How is evidence of response processes gathered in psychological measurement?
Page 2
Answer
It is examined by assessing the cognitive mechanisms and subjective interpretations respondents use to answer prompts. Theoretical evidence is established by detailing how items, such as semantic differential word pairs, are processed and selected.
10
Question
Which acronym represents the five essential sources of validity evidence identified by AERA et al. (2014)?
Page 1
Answer
CRISC, which anchors Content, Response processes, Internal structure, Similar/other variables, and Consequences of testing.
11
Question
How does the Forgiveness Reconciliation Inventory format its response scale to capture cognitive process?
Page 2
Answer
It uses semantic differential rating scales with five checkboxes situated between opposing adjective pairs. The two outer boxes represent strong alignment with each pole, and the center box represents neutrality.
12
Question
What is the primary role of statistical analyses when establishing evidence of internal structure?
Page 2
Answer
Statistical analyses evaluate relationships among individual items and subscales to verify how the measure is organized and scored. Typical procedures include exploratory factor analysis, confirmatory factor analysis, and reliability estimates.
13
Question
In psychometrics, what is the definition of a factor or latent variable?
Page 2
Answer
A factor is an unobserved underlying construct inferred mathematically from clusters of items that correlate together. Memory hook: A puppeteer hidden behind a black curtain whose presence is revealed only when multiple strings twitch in unison.
14
Question
What is the primary objective of conducting an exploratory factor analysis (EFA)?
Page 2
Answer
EFA serves as a preliminary evaluation to determine whether an instrument is unidimensional or multidimensional. It groups items based on correlations without imposing a rigid a priori factor structure.
15
Question
Which memory image anchors the psychometric definition of an unobserved latent variable?
Page 2
Answer
A hidden puppeteer behind a curtain whose presence is deduced only because multiple marionette strings move simultaneously.
16
Question
What numerical threshold defines an ideal factor loading for an item in exploratory factor analysis?
Page 2
Answer
An ideal factor loading is \(0.40\) or greater on the designated factor, accompanied by lower loadings across non-target factors. Number hook: \(0.40\) is the speed limit of forty miles per hour for boarding the factor bus.
17
Question
What participant-to-item ratio is recommended to conduct exploratory factor analysis?
Page 2
Answer
Researchers generally require \(10\) to \(20\) participants per individual survey item. Ratio hook: Ten to twenty hungry testers seated around one single test item plate.
18
Question
How does confirmatory factor analysis differ from exploratory factor analysis?
Page 2
Answer
EFA discovers underlying dimensions when researchers are uncertain about dimensionality. CFA tests an a priori, theory-driven factor model against empirical data to determine model fit. Contrast hook: EFA explores an uncharted forest, while CFA checks a pre-drawn architectural blueprint.
19
Question
Why must researchers double their sample size when conducting both EFA and CFA in one study?
Page 2
Answer
EFA and CFA must be executed on completely separate normative samples to prevent circular validation. Each analysis requires \(10\) to \(20\) participants per item, requiring two independent cohorts.
20
Question
Which contrast hook illustrates the core distinction between exploratory factor analysis and confirmatory factor analysis?
Page 2
Answer
EFA explores an uncharted wild forest without a map, whereas CFA inspects a pre-drawn architectural blueprint to verify construction.
21
Question
What does a statistically significant chi-square test indicate in confirmatory factor analysis?
Page 2
Answer
A statistically significant chi-square indicates that the distribution of the empirical sample data differs significantly from the hypothesized model, suggesting poor fit.
22
Question
Why can a large sample size produce a statistically significant chi-square in CFA despite good practical fit?
Page 2
Answer
Large sample sizes dramatically inflate statistical power, causing negligible deviations from the hypothesized model to reach statistical significance. Researchers therefore examine alternative fit indices.
23
Question
What two explanations account for a statistically significant chi-square in confirmatory factor analysis?
Page 2
Answer
The discrepancy occurs either because the researcher misspecified the theoretical model or because a very large sample size artificially inflated statistical significance.
24
Question
Which alternative fit indices did Balkin et al. (2014) report to substantiate FRI model fit?
Page 3
Answer
They reported the Comparative Fit Index, the Tucker-Lewis Index, and the Standardized Root-Mean-Square Residual. Acronym hook: CTS (Comparative, Tucker-Lewis, Standardized residual).
25
Question
Which acronym anchors the three alternative fit indices reported by Balkin et al. (2014) for the FRI?
Page 3
Answer
CTS, standing for Comparative Fit Index, Tucker-Lewis Index, and Standardized Root-Mean-Square Residual.
26
Question
What specific values did Balkin et al. (2014) report for the CFI, TLI, and SRMR on the FRI?
Page 3
Answer
They reported \(\text{CFI} = 0.91\), \(\text{TLI} = 0.90\), and \(\text{SRMR} = 0.069\). Value hook: Ninety-one and ninety score high passes, while the point zero six nine remainder stays safely small.
27
Question
Why did Balkin et al. (2014) conduct a CFA instead of an EFA during initial FRI validation?
Page 3
Answer
The FRI was constructed from a pre-existing published theoretical framework with an established four-factor structure. Because the structure was hypothesized a priori, a confirmatory approach was indicated.
28
Question
Why is reliability correctly attributed to test scores rather than to the test instrument itself?
Page 3
Answer
Reliability varies across diverse populations and sample demographics. A scale that generates consistent scores with one population may yield unreliable scores in another. Memory hook: A pair of boots is not warm by itself; warmth depends on the foot wearing it.
29
Question
How is test-retest reliability evaluated across longitudinal administrations?
Page 3
Answer
The identical measure is administered to the same group at two different time points, typically within two weeks without intervention. Scores should show high consistency, ideally correlating at \(0.70\) or higher.
30
Question
Which memory image illustrates why reliability belongs to sample scores rather than the instrument?
Page 3
Answer
A winter boot is not warm on its own; warmth depends entirely on the specific foot wearing the boot.