Appeals of Material Supervisory Determinations: Guidelines and Determinations
SARC- 2005-03 (November 10, 2005)
This appeal arises from a dispute over the development and validation of a
credit scoring model used in underwriting consumer loan applications by
individuals with little or no credit history. On December 21, 2004, the
Regional Director of the *** Regional Office (the “Regional Director”) of
FDIC’s Division of Supervision and Consumer Protection (“DSC”) issued a
material supervisory determination reached following a Compliance
Examination on January 21, 2003 of *** (“the Bank”). The Regional Director determined
that the Bank’s policies and practices discriminated against applicants on
the basis of age, violating the Equal Credit Opportunity Act (“ECOA”) 15
U.S.C. §§ 1691 et seq. and Sections 202.4(a) and 202.6(b)(2) of ECOA’s
implementing provision, Regulation B. On February 18, 2005, the Bank timely
sought review, and on March 18, 2005, DSC’s Director in Washington (the
“Director”) affirmed the Regional Director’s decision. The Bank timely filed
an appeal with the Supervision Appeals Review Committee (the “Committee”) by
letter dated June 30, 2005.
The Bank argues that (1) FDIC has interpreted the provisions of ECOA and
Regulation B addressing the use of credit scoring models in an overly
restrictive manner not required by the law; (2) the Bank ceased using the
model when it had sufficient evidence to find its predictive qualities
deficient; (3) no credit applicant was harmed by use of the model; and (4)
as no credit applicant was “discouraged” by any pattern or practice of the
Bank, the matter should not be referred to the U.S. Department of Justice
(the “Department of Justice,” “DOJ”).
DSC responds that (1) its interpretation of Regulation B is in line with
that of the Federal Reserve Board, which promulgated the Regulation, and
that the Bank’s model was not statistically validated as required; (2) the
Bank continued to use the model for at least three years after actual
results of its use should have demonstrated its insufficient predictive
ability to meet the validation requirements of the Regulation; (3) if
applicants for credit were treated differently on the basis of age without a
valid statistical basis justifying the difference in treatment, the Bank has
violated the Regulation, regardless of the outcome of the underwriting
decision; and (4) as the model overtly and impermissibly treated credit
applicants differently on the basis of age without appropriate validation
and adjustment to maintain predictive ability, FDIC must refer the matter to
the Department of Justice.
In accordance with the
Guidelines for Appeals of Material Supervisory Determinations1,
the Committee reviews the appeal for consistency with the policies,
practices and mission of FDIC, as well as the reasonableness of, support
for, and respective positions of the parties. The Committee has considered
the written submissions of the Bank. The scope of the Committee’s review is
limited to the facts and circumstances extant at the time of the
examination. The Committee found, as asserted by the Bank, that the evidence
fails to demonstrate that the NORT credit scoring model was statistically
unsound as developed. However, the Committee also found that the Bank did
not properly revalidate and adjust the model on a continuing basis as
required by Regulation B. This finding compels a referral to the Department
of Justice as required by the Equal Credit Opportunity Act, 15 U.S.C. §
A. Development of the Model.
In 1999, the Bank contracted with Company X (“X”), an acknowledged expert in
predictive modeling, scoring and analysis, to develop a set of “scorecards”
for use in underwriting consumer loan applications. For applicants with
established credit histories, five scorecards were fashioned, a separate one
for each of five products – (1) home equity loans, (2) home equity lines of
credit, (3) automobile loans, (4) other secured personal loans, and (5)
unsecured personal loans.
The Bank also sought to create a scoring system that would enable it to
extend credit to applicants who otherwise would be denied credit due to a
lack of credit history. A single scorecard was formulated to underwrite
consumer loan applications by individuals who either had no credit bureau
record or had no trade lines on their credit bureau reports. This “no
record/no trade line” (“NORT”) scorecard was the only model of the six
developed that considered age as a variable directly scored. For the vast
majority of NORT applicants, only two variables other than age were scored –
length of time at present residence, and length of time at present
employment. The fact that a single scorecard was created for applicants with
no credit history meant that the NORT model was developed using repayment
data aggregated from five different loan products. Loan type was not a
variable in the NORT model, and the Bank did not track loan product data for
According to X, the NORT model was developed using widely accepted
statistical principles and modeling techniques and was constructed in a
manner similar to other NORT models that it has developed, some of which
explicitly take age into account. X asserts that it commonly develops a
single scorecard that applies to multiple loan products. The Bank adopted
the model only after X evaluated its statistical validity. At the outset,
the results reported a Kolmogorov-Smirnoff statistic – a commonly used
measure of scorecard performance – of 29.9, a score that is, though within
acceptable bounds, considered marginal.
B. Continuing Validation of the Model
The Bank conducted semiannual validations of the NORT credit scoring model
until October 2003. The Bank’s continuing validation reports indicate that
the NORT model was statistically weak in two key respects: (1) the actual
applicant pool (customers who “walked in the door”) varied from the
development applicant pool (the profile of customers used to develop the
model), and (2) the model did not adequately separate good from bad loans.
The Bank never adjusted the model, but finally abandoned its use in February
A. ECOA and Regulation B.
The Equal Credit Opportunity Act was enacted in 1974 to prevent
discrimination in the extension of consumer credit. The Act makes it
“unlawful for any creditor to discriminate against any applicant, with
respect to any aspect of a credit transaction on the basis of . . . age.” 15
U.S.C. § 1691(a). Regulation B proscribes discrimination against an
applicant on a prohibited basis regarding any aspect of a credit
transaction. 12 C.F.R. § 202.4(a). Although age is a prohibited basis2,
Regulation B permits age to be used as a predictive variable
[i]n an empirically derived, demonstrably and statistically sound,
credit scoring system . . .
12 C.F.R. § 202.6(b)(2)(ii).
To qualify as “empirically derived, demonstrably and statistically
sound,” Regulation B mandates, among other requirements, that the system be
(iii) Developed and validated using accepted statistical principles and
(iv) Periodically revalidated by the use of appropriate statistical
principles and methodology and adjusted as necessary to maintain
12 C.F.R. § 202.2(p)(1)(iii), (iv).
B. The Development Phase of the Model.
DSC contends that the population characteristics of applicants who apply for
one credit product tend to vary considerably from the characteristics of
those who apply for a different product. Similarly, performance tends to
vary appreciably from one product to another. For example, applicants
perform significantly better on home equity loans than on unsecured consumer
loans. DSC reasons that because the NORT model combines applicants for five
diverse credit products and does not track performance by product, it is
impossible to tell, using accepted statistical methodology, whether the
scorecard is valid for any of the five products considered individually.
The Bank, citing analysis performed by X, counters that there is not
always a predictive benefit to having a sub-model developed for each
The segmentation analysis for the NORT model showed that superior
predictive lift was obtained from having a sub-model specifically for the
NORT population regardless of product.
Letter from X to the Bank, dated August 13, 2004.
The evidence presented fails to demonstrate to the Committee that it was
known that the model was not statistically sound as developed. Thus,
the Committee declines to find a violation with respect to the development
phase of the NORT credit scoring model.
C. The Validation Phase of the Model.
The Bank had ample information prior to October 2002 to conclude that the
NORT scorecard had insufficient predictive ability to meet Regulation B
requirements. The Bank’s first validation report in June 2000 revealed a
significant population shift from the “development” population to the actual
loan applicants. This shift was due largely to a difference in age
distribution. Further, the Bank staff viewed the model as commanding low
predictive power, as pointed out in its October 2002 validation review:
In only the [NORT] model is the PSI3
higher than the acceptable threshold of (0.150). Unlike the [NORT] model
in Dealer Finance, the [NORT] model in the Direct line of business scores
slightly lower than expected. Since each line of business considers the [NORT]
model to be an important tool in the auto-decline process, the statistical
validation of the model is not of primary importance. Again, the
predictive power of this model is extremely low considering the lack of
available credit history on the customers (sic) credit bureau file.
Later validation testing demonstrated that the statistics remained
extremely poor and that the model was not performing well. The Bank admitted
in its own review that the validation statistics could not separate good
from bad loans at a statistically significant level, nor did the model meet
the Bank’s validation benchmark. In addition to demonstrating poor
statistical performance, the validation reports contained statements by the
Bank staff that the Bank was aware of the model’s poor performance, but
nonetheless continued to use it. That is, beginning in October 2000 and
continuing through May 2003, the validation reports note that the
“predictive power of [the NORT] model is intuitively low,” but state that
“business reasons” still justify using the model for the Bank’s
In sum, the Bank’s validation reports confirm that, despite conducting
regular examinations of the NORT credit scoring model, the Bank did not
“revalidate using appropriate statistical principles and methodology and
adjust as necessary to maintain predictive ability.” 12 C.F.R. §
202.2(p)(1)(iv). Despite poor results from 2000 forward, the Bank never
adjusted the NORT model and, in fact, continued its use until February 2004.4
For these reasons, we conclude that the Bank has failed to show that the
NORT model was periodically revalidated using demonstrably and statistically
sound principles, as required by Regulation B.
D. Referral to the Department of Justice.
Since 1991, FDIC and the federal agencies with responsibility for
administrative enforcement of ECOA, have been required to refer violations
of that statute to DOJ, where there is reason to believe that one or more
creditors has engaged in a pattern or practice of denial or discouragement
on a prohibited basis.
Although the Bank appears to have been well-intended in attempting to
construct an automated scoring system that would enable it to extend credit
to an underserved population, under NORT, younger applicants were treated
less favorably than older applicants. The validation statistic could not
separate good from bad loans, and the model had low predictive power. The
Bank did not adjust the model based upon its consistently weak statistical
performance. Accordingly, in our view, the Bank’s model violated ECOA and
Regulation B: it directly considered age in a credit scoring model that was
not revalidated on a demonstrably and statistically sound basis. Such a
finding compels a referral to the Department of Justice.
For the reasons set forth above, the Committee finds, as asserted by the
Bank, that the evidence fails to demonstrate that the NORT credit scoring
model was statistically unsound as developed, but also finds that the Bank
failed to properly revalidate and adjust the model on a continuing basis as
required by Regulation B. This decision is considered a final supervisory
decision by FDIC.
By direction of the Supervision Appeals Review Committee of FDIC, dated
November 10, 2005.
Valerie J. Best
Assistant Executive Secretary
The Guidelines are set out at 69 Fed. Reg. 41479 (July 9, 2004) and
in FDIC Financial Letter (“FIL”) 113-2004 (Oct. 13, 2004).
2 12 C.F.R. § 202.2(z).
3 PSI is the conventional
statistical method of comparing the actual “in-the-door” applicants to the
data used to develop