1) Behavioral Demand Letter Experiment
Findings from a Randomized Controlled Trial
Tammy Chang
August 12, 2014
2) Background
• Federal agencies refer delinquent debts to
Treasury at 180 days past delinquency
(batch/daily/etc referrals)
• Treasury issues initial “demand letters” to
debtors
– Notice of Referral
– Debt information
– Repayment options
• Debts serviced internally for 30 days
Page 2
LEAD ∙ TRANSFORM ∙ DELIVER
3) Analytics Objectives
• Design and run a rigorous and statistically
sound experiment
– Implement a Randomized Controlled Trial (RCT)
• Develop and test an alternative letter
– Can a letter incorporating insights from
behavioral science change debtor behavior?
Page 3
LEAD ∙ TRANSFORM ∙ DELIVER
4) Behavioral Science Concepts
•
•
•
•
•
•
Social norm
Limited attention
Information overload
Personalization
Hassle factors
Primacy
Page 4
LEAD ∙ TRANSFORM ∙ DELIVER
5) Collaboration and Context
• US Social and Behavioral Sciences Team
– Identify opportunities for behavioral insights to
increase efficiency and efficacy of
federal agency programs
• Evidence and Innovation Agenda (OMB M-13-17)
– Promote evidence-based policy
– Using data to inform and improve agency results
– Rigorous, low-cost experiments
• 2014 Economic Report of the President
Page 5
LEAD ∙ TRANSFORM ∙ DELIVER
6) Outcome Measures of Interest
• Collections
– Likelihood of collection
• Debtor contact
– Likelihood of inbound calls
• Payment method
– Likelihood of self-initiated Pay.gov payment
Page 6
LEAD ∙ TRANSFORM ∙ DELIVER
7) Demand Letter Examples
Control
Page 7
Treatment
LEAD ∙ TRANSFORM ∙ DELIVER
8) Demand Letter Examples
Limited Attention
Treatment
Social Norm
Personalization
Primacy
Hassle Factor
Previous Pay.gov link:
https://www.pay.gov/paygov/forms/formIn
stance.html?agencyFormId=16531440
New Pay.gov link:
www.pay.gov/paygov/paymydebt
Messenger
Page 8
LEAD ∙ TRANSFORM ∙ DELIVER
9) Measuring Impact: Non-RCT Example
• Before vs. After
5
4.5
4
Start of Pilot
Percent
3.5
3
Impact?
2.5
2
1.5
1
0.5
0
Time
0
1
• What would have happened without the pilot?
Page 9
LEAD ∙ TRANSFORM ∙ DELIVER
10) Measuring Impact: Example 1
• Control vs. Treatment
5
4.5
4
Treatment
Start of Pilot
3.5
Percent
3
Impact
2.5
Control/Counterfactual
2
1.5
1
0.5
0
Page 10
Time
0
1
LEAD ∙ TRANSFORM ∙ DELIVER
11) Measuring Impact: Example 2
• Control vs. Treatment
5
Control/Counterfactual
4.5
4
Start of Pilot
Treatment
Impact
3.5
Percent
3
2.5
2
1.5
1
0.5
0
Page 11
Time
0
1
LEAD ∙ TRANSFORM ∙ DELIVER
12) Measuring Impact: Example 3
• Control vs. Treatment
5
4.5
4
Start of Pilot
3.5
Percent
3
2.5
2
1.5
Impact
1
0.5
0
Page 12
Treatment
Control/Counterfactual
Time
0
1
LEAD ∙ TRANSFORM ∙ DELIVER
13) RCT Methodology
• Use of control/treatment groups
• Random assignment
– Last digit of Social Security Number (SSN)
– SSN Structure XXX-XX-XXXX
(prior to June 2011)
Area number Group Number Serial Number
(random)
Even Digit: Control Group
Current Letter
XXX-XX-XXX0
XXX-XX-XXX2
XXX-XX-XXX4
XXX-XX-XXX6
XXX-XX-XXX8
Page 13
Odd Digit: Treatment Group
New Letter
XXX-XX-XXX1
XXX-XX-XXX3
XXX-XX-XXX5
XXX-XX-XXX7
XXX-XX-XXX9
LEAD ∙ TRANSFORM ∙ DELIVER
14) Sample
• Six agency programs
– Consistent referrals
– Smaller balances
– Administrative debts
Agency Programs
• VA: Compensation and Pension
• Railroad Retirement Board: Unemployment
Insurance Act Annuitant
DOD: Defense Financial Accounting Services
•
• DOD: Brooke Army Medical Center
• USDA: Administrative Consumer
• Treasury: Securities and Savings Bonds
• Referrals from October 28, 2013 – March 31, 2014
Page 14
LEAD ∙ TRANSFORM ∙ DELIVER
15) Outcome Measures Data
• Existing data from FedDebt
–
–
–
–
–
Referral Dates
Collection Dates and Type
Call Dates and Types
Payment Type
Debt Characteristics
• Timeframe
– 30 days after referral
– 60 days after referral
Page 15
LEAD ∙ TRANSFORM ∙ DELIVER
16) Descriptive Statistics
• Equal assignment across control/treatment
Control
Treatment
Total
% Control
1,270
1,264
2,534
50.12%
359
361
720
49.86%
DFAS
8,246
8,282
16,528
49.89%
BAMC
205
186
391
52.43%
USDA
445
374
819
54.33%
RRB
167
146
313
53.35%
10,692
10,613
21,305
50.19%
VA
BPD
Total
• Percent of debts with any collection
(within 30 days of referral, by agency)
Collection
167
No
Collection
2,367
Total
2,534
% With Coll
6.59%
BPD
92
628
720
12.78%
DFAS
900
15,628
16,528
5.45%
BAMC
7
384
391
1.79%
USDA
65
754
819
7.94%
RRB
23
290
313
7.35%
Total
1,254
20,051
21,305
5.89%
VA
Page 16
LEAD ∙ TRANSFORM ∙ DELIVER
17) Results: Collections
• Likelihood of Collection
– Lower likelihood of collection on treatment group
Percent of Debts with Collection
6%
Collections within 30 days of referral
Control
No collection
Total
% With Coll
Treatment
Total
637
617
1,254
5.89%
10,055
9,996
20,051
94.11%
10,692
10,613
21,305
100%
50.19%
Collection
49.81%
5.96%
5.81%
Difference: -0.15%
5%
4%
3%
2%
1%
0%
Control
– Difference statistically insignificant
Page 17
LEAD ∙ TRANSFORM ∙ DELIVER
Treatment
18) Results: Calls
• Likelihood of Inbound Call
– Lower likelihood of calls from treatment group
14%
Inbound calls within 30 days of referral
Control
Treatment
Total
Inbound call
1,473
1,396
2,869
13.47%
No inbound call
9,219
9,217
18,436
86.53%
10,692
10,613
21,305
100%
13.78%
13.15%
Difference: -0.62%
Total
% With Call
Percent of Debts with Inbound
Call Record
12%
10%
8%
6%
4%
2%
0%
Control
– Difference statistically insignificant
Page 18
LEAD ∙ TRANSFORM ∙ DELIVER
Treatment
19) Results: Pay.gov
• Likelihood of Pay.gov payment
– Higher likelihood of self-initiated Pay.gov
Percent of Debts with
payments from treatment group
Pay.gov Payment
5%
Payments within 30 days of referral
Control
Treatment
Total
529
593
1,122
5.27%
10,163
10,020
20,183
94.73%
Total
10,692
10,613
21,305
100%
% Pay.gov
4.95%
5.59%
Difference: 0.64%
Pay.gov
No or other
payment
– Odds ratio of 1.5: treatment group odds 1.5
times higher than control group odds of having
Pay.gov payment
4%
3%
2%
1%
0%
– Difference statistically significant
Page 19
LEAD ∙ TRANSFORM ∙ DELIVER
Control
Treatment
20) Discussion/Recommendations
• Findings on likelihood of collection depart
from previous research
• No effect on payment or contact likelihood
– What does this tell us?
• Ability to influence payment method decision
– Change other Pay.gov references
• Successful RCT!
Page 20
LEAD ∙ TRANSFORM ∙ DELIVER
21) Contact Information
Questions?
Tammy Chang
Sr Economist
Debt Management Services
Bureau of the Fiscal Service
202-874-5205
tammy.chang@fiscal.treasury.gov
Page 21
LEAD ∙ TRANSFORM ∙ DELIVER
