<p>How financial service providers use data and what must be done to protect already-underserved populations against algorithmic bias in automated decision-making.</p>


When Goldman Sachs came under fire for offering women lower lines of credit than men on the now <a href="https://www.nytimes.com/2019/11/10/business/Apple-credit-card-investigation.html" target="_blank" rel="noopener noreferrer">infamous Apple Card</a>, they defended the product against charges of sexism by explaining that gender had not been an input to the credit scoring model. If an applicant’s gender was never fed into the algorithm, how did credit decisions end up discriminating against women who had similar credit histories as men?

Gender-blind credit scoring is an example of what many AI bias researchers call “fairness through unawareness,” an approach that intentionally excludes data on sensitive attributes like race, ethnicity, religion, or sex with the goal of reducing discriminatory outcomes. Behind this approach is the idea that eliminating markers of different group identities will prevent basing decisions on those characteristics. This has led to failures in the past, notably <a href="https://cpb-us-e1.wpmucdn.com/sites.suffolk.edu/dist/3/1172/files/2014/01/Rice-Swesnik_Lead.pdf" target="_blank" rel="noopener noreferrer">redlining</a>. Recent research has shown it can also make <a href="https://arxiv.org/abs/2009.06251" target="_blank" rel="noopener noreferrer">discrimination harder to detect and address</a> in the era of algorithmic decisions.
<blockquote class="pullquote--right">“Fairness through unawareness” intentionally excludes data on sensitive attributes like race, ethnicity, religion, or sex with the goal of reducing discriminatory outcomes.</blockquote>
As algorithms <a href="https://www.centerforfinancialinclusion.org/the-stories-algorithms-tell-bias-and-financial-inclusion-at-the-data-margins" target="_blank" rel="noopener noreferrer">transform the financial sector</a>, we need a better understanding of how to mitigate against discriminatory outcomes to ensure that those left out of the digital ecosystem today do not face further financial exclusion. Otherwise, we may not know how many Apple Card equivalents exist in inclusive finance.

This post explores the challenges and tradeoffs of using sensitive attribute data for detecting discrimination in the context of evolving data protection norms and regulations.
<h2>The Perils of Fairness through Unawareness</h2>
The idea may seem counterintuitive: if you remove gender from the data inputs of a credit model, how does the algorithm consistently assign a lower limit to women? The reason is that credit scoring algorithms are crunching larger amounts and new sources of data, which means that the model can inadvertently use proxies for sensitive attribute data in decisions, even if that was not the intention of the model developers. For instance, the <a href="https://journalofbigdata.springeropen.com/articles/10.1186/s40537-019-0180-9" target="_blank" rel="noopener noreferrer">type of phone</a> someone uses or the <a href="https://dl.acm.org/doi/10.1145/2721896.2721908" target="_blank" rel="noopener noreferrer">apps a person installs on their phone</a> can act as proxies for gender, age, or other characteristics.
<blockquote class="pullquote--left">Models can inadvertently use proxies for sensitive attribute data in decisions, even if that was not the intention.</blockquote>
A very simple method for testing algorithmic bias within a credit scoring model requires only a few variables like the actual default rate, the actual score of the individual when approved, and the variable you want to test with your algorithm, such as gender or race.
<h2>Data Tensions: Privacy vs. Anti-Discrimination</h2>
So, why isn’t this done? The answer lies in data limitations, sometimes based on very legitimate concerns about histories of injustice. It illustrates one of the challenges of detecting and monitoring for bias in algorithmic systems without infringing on privacy rights.

Many countries around the world prohibit collecting sensitive data or basing credit decisions on protected categories either under their banking regulations or under new data privacy legislation. In India, lenders are prohibited from discriminating on the <a href="https://www.rbi.org.in/Scripts/PublicationReportDetails.aspx?ID=764" target="_blank" rel="noopener noreferrer">grounds of sex, caste, or religion</a>. In the U.S., the Equal Credit Opportunity Act (ECOA) made it illegal for credit providers to discriminate based on sex, race, national origin, and other protected characteristics in response to widespread denials of credit to women and people of color. ECOA prohibits lenders from collecting this data in the first place, even for the purpose of detecting discrimination, because of fears it will be used improperly to further perpetuate bias in consumer lending. Research in the US shows that discriminatory trends have been <a href="https://arxiv.org/pdf/1912.06171.pdf" target="_blank" rel="noopener noreferrer">hard to detect in consumer lending</a> and that a lack of sensitive attribute data has also created enforcement challenges.

While not all lending regulations prohibit the collection of sensitive data, data privacy legislation introduces new limits on this data. The General Data Protection Regulation (GDPR), which has become the model for legislation in many developing markets, bars certain sensitive data from being collected or processed -- including data on race and ethnic origin, political opinions, philosophical beliefs, sexual orientation, and biometrics -- without individual consent.
<blockquote class="pullquote--right">Concerns about safely and properly using sensitive attribute data are valid given histories of discrimination and surveillance of minority groups in both wealthy and developing countries.</blockquote>
On top of GDPR, the European Commission has recently put forward <a href="https://digital-strategy.ec.europa.eu/en/library/proposal-regulation-laying-down-harmonised-rules-artificial-intelligence" target="_blank" rel="noopener noreferrer">draft legislation to regulate artificial intelligence</a>, with specific provisions for AI systems classified as high risk, which includes credit scoring. The draft legislation suggests that AI providers should process “special categories of personal data” to monitor and correct for bias in the public interest, but does not clarify collection practices or consent processes for such data. Concerns about safely and properly using sensitive attribute data are valid given histories of discrimination and surveillance of minority groups in both wealthy and developing countries. Changes in regulatory frameworks to allow for use of this data to mitigate against further discrimination require public trust bolstered by technical solutions that protect against misuse. Depending on a country’s history or in places experiencing active or simmering conflict, technical solutions may not be enough. For example, in a country like Rwanda where <a href="https://www.nytimes.com/2004/04/09/world/a-decade-after-massacres-rwanda-outlaws-ethnicity.html" target="_blank" rel="noopener noreferrer">ethnic distinctions are banned</a>, the tensions around sensitive identities make it difficult to fathom collecting and using such information, especially by the government.
<h2>Data Ecosystems in Emerging Markets: It’s Complicated</h2>
This tension between privacy and anti-discrimination may be exacerbated in developing markets by the <a href="https://centerforfinancialinclusion.org/rethinking-financial-inclusion-at-the-data-margins/" target="_blank" rel="noopener noreferrer">limited digital footprints</a> of customers at the data margins, as well as paltry knowledge on how data trails and protected groups intersect.

Sex-disaggregated data is still <a href="https://data2x.org/wp-content/uploads/2020/03/MappingGenderDataGaps_Economic.pdf" target="_blank" rel="noopener noreferrer">largely unavailable</a> in the financial sector and for many economic indicators in developing markets. Beyond availability, there is less understanding of how women’s lives and the barriers that shape their economic behaviors are reflected in data used for credit decisions. The reality is that gender, ethnic, or religious identities interact with other indicators (like income, asset accumulation, or technology use) and will impact individuals’ data trails in ways that may misrepresent individuals’ potential and result in disparate outcomes (i.e. access to a financial product). Fairness through unawareness approaches are blind to this intersectionality.
<h2>Where Do We Go From Here?</h2>
As an industry, we can do more to understand the data inputs for digital financial services and how socioeconomic realities present in these datasets, especially for traditionally excluded groups. There will still be tradeoffs between privacy and anti-discrimination, but managing those tradeoffs requires a conversation between providers, regulators, donors, and the public to identify what is possible and explore the right balance.

These conversations can start with the many open questions that exist from a technical perspective. For instance, can <a href="https://ieeexplore.ieee.org/document/8970908" target="_blank" rel="noopener noreferrer">third parties</a> play a role in securely storing sensitive data, or are there options to use <a href="http://proceedings.mlr.press/v80/kilbertus18a/kilbertus18a.pdf" target="_blank" rel="noopener noreferrer">encryption</a> or <a href="https://spectrum.ieee.org/artificial-intelligence/machine-learning/engineering-bias-out-of-ai" target="_blank" rel="noopener noreferrer">synthetic data</a> techniques? Can regulators develop tools that allow them to estimate the demographic characteristics of a dataset, like the <a href="https://www.consumerfinance.gov/data-research/research-reports/using-publicly-available-information-to-proxy-for-unidentified-race-and-ethnicity/" target="_blank" rel="noopener noreferrer">Consumer Financial Protection Bureau has done</a> in the U.S.? Can investors encourage investees to design monitoring and evaluation indicators to help answer questions about how their algorithms perform on inclusion?

While the road ahead may be complex, starting these conversations and bringing the public into them, especially those whose lives will be affected by algorithmic decisions, will be crucial for building trust. We need to engage with emerging research on possible solutions to see what might be appropriate in developing markets. At the very least, financial inclusion stakeholders should know enough to respond to any gender-blind or fairness through unawareness approaches in digital finance with healthy skepticism.
<blockquote>Starting these conversations and bringing the public into them, especially those whose lives will be affected by algorithmic decisions, will be crucial for building trust.</blockquote>

Reducing Bias in Algorithmic Decisions Cannot Rely on “Blind” Approaches

In 2020, CFI and Women’s World Banking began workstreams on algorithmic bias. This conversation between Alex Rizzi (CFI) and Sonja Kelly (Women’s World Banking) builds on recent reports by <a href="https://centerforfinancialinclusion.org/the-stories-algorithms-tell-bias-and-financial-inclusion-at-the-data-margins/" target="_blank" rel="noopener noreferrer">CFI</a> and <a href="https://www.womensworldbanking.org/insights-and-impact/report-algorithmic-bias-financial-inclusion-and-gender-primer-opening-up-new-credit-to-women-in-emerging-economies/" target="_blank" rel="noopener noreferrer">Women’s World Banking</a>. Below are their answers to some important questions about the future of AI and machine learning as they apply to financial inclusion.
<h3>Where are we now in the conversation on algorithmic bias? Is it too late to address this?</h3>
<strong>Alex</strong>: It’s just the right time! While it may feel like global conversations around responsible tech have been going on for years, they haven’t been grounded squarely in our field. For instance, there hasn’t been widespread testing of de-biasing tools in inclusive finance (though Sonja, we’re excited to hear about the results of your upcoming work on that front!) or mechanisms akin to credit guarantees to incentivize digital lenders to expand the pool of applicants their algorithms deem creditworthy. At the same time, there are a bunch of data protection frameworks being passed in emerging markets that are modeled on the European GDPR and give consumers data rights related to automated decisions, for example. These frameworks are very new, and it’s still unclear whether and how they might bring more algorithmic accountability. So it’s absolutely not too late to address this issue.

<strong>Sonja</strong>: I completely agree that now is the time, Alex. Just a few weeks ago we saw a <a href="https://www.federalreserve.gov/newsevents/pressreleases/bcreg20210329a.htm" target="_blank" rel="noopener noreferrer">request for information</a> here in the U.S. for how financial service providers use artificial intelligence and machine learning. It’s clear there is an interest on the policymaking and regulatory side to better understand and address the challenges posed by these technologies, which makes it an ideal time for financial service providers to be proactive about guardrails to keep bias from algorithms. I also think that technology enables us to do much more about the issue of bias -- we can actually turn algorithms around to audit and mitigate bias with very low effort. We now have both the motivation and the tools to be able to address this issue in a big way.
<h3>What are some trends leading us down the wrong path when it comes to algorithmic bias?</h3>
<strong>Sonja</strong>: At the risk of being too broad, I think the biggest trend is lack of awareness. Like I said before, fixing algorithmic bias doesn’t have to be hard, but it does require everyone -- at all levels and within all responsibilities -- to understand and track progress on mitigating bias. The biggest red flag I saw in our interviews contributing to our report was when an executive said that bias isn’t an issue in their organization. My co-author Mehrdad Mirpourian and I found that bias is always an issue. It emerges from biased or unbalanced data, the code of the algorithm itself, or the final decision on who gets credit and who does not. No company can meet all definitions of fairness for all groups simultaneously. Admitting the possibility of bias costs nothing, and fixing it is not that difficult. Somehow it slips off the agenda, and that is very discouraging to me.
<blockquote class="pullquote--left">Bias is always an issue</blockquote>
<strong>Alex</strong>: One of the concepts we’ve been thinking a lot about is the idea of how digital data trails may reflect or further encode existing societal inequities. For instance, we know that women are <a href="https://www.gsma.com/r/gender-gap/" target="_blank" rel="noopener noreferrer">less likely to own phones than men</a> and less likely to use mobile internet or certain apps; these differences create disparate data trails, and might not tell a provider the full story about a woman’s economic potential. And what about the <a href="https://centerforfinancialinclusion.org/rethinking-financial-inclusion-at-the-data-margins/" target="_blank" rel="noopener noreferrer">myriad of other marginalized groups</a>, whose disparate data trails are not clearly articulated?
<h3>Who else needs to be here in this conversation as we move forward?</h3>
<strong>Alex</strong>: For my colleague Alex Kessler and me, a huge takeaway from the exploratory work was that there are plenty of entry points to these conversations for non-data scientists, and it’s crucial for a range of voices to be at the table. We originally had this notion that we needed to be fluent in the code-creation and machine learning models to contribute, but the conversations should be interdisciplinary and should reflect a strong understanding of the contexts in which these algorithms are deployed.
<blockquote class="pullquote--right">There are plenty of entry points to these conversations for non-data scientists.</blockquote>
<strong>Sonja</strong>: I love that. It’s exactly right. I would also like to see more media attention on this issue. We know from other industries that we can increase innovation by peer learning. If sharing both the promise and pitfalls of AI and machine learning becomes normal, we can learn from it. Media attention would help us get there.
<h3>What are the immediate next steps here? What are you focused on changing tomorrow?</h3>
<strong>Sonja</strong>: When I share our report with external audiences, I first hear shock and concern about the very idea of using machines to make predictions about people’s repayment behavior. But our technology-enabled future doesn’t have to look like a dystopian sci-fi novel. Technology can increase financial inclusion when deployed well. Our next step should be to start piloting and proof-testing approaches to mitigating algorithmic bias. Women’s World Banking is doing this over the next couple of years in <a href="https://www.womensworldbanking.org/insights-and-impact/awardees-announced-as-part-of-data-org-inclusive-growth-and-recovery-challenge/" target="_blank" rel="noopener noreferrer">partnership with the University of Zurich and data.org</a> with a number of our network members, and we’ll share our insights as we go along. Assembling some basic resources and proving what works will get us closer to fairness.
<blockquote class="pullquote--left">Our technology-enabled future doesn’t have to look like a dystopian sci-fi novel.</blockquote>
<strong>Alex</strong>: These are early days. We don’t expect there to be universal alignment on de-biasing tools anytime soon, or best practices available on how to enforce data protection frameworks in emerging markets. Right now, it’s important to simply get this issue on the radar of those who are in a position to influence and engage with providers, regulators, and investors. Only with that awareness can we start to advance good practice, peer exchange, and capacity building.

<em>Follow CFI (<a href="https://centerforfinancialinclusion.us5.list-manage.com/subscribe/post?u=394352ae963270aef6c51680d&amp;id=9df382106a">newsletter</a>, <a href="https://www.linkedin.com/showcase/center-for-financial-inclusion-at-accion" target="_blank" rel="noopener noreferrer">LinkedIn</a>, and <a href="https://twitter.com/CFI_Accion" target="_blank" rel="noopener noreferrer">Twitter</a>) and Women’s World Banking over the coming months to stay up-to-date on algorithm bias and financial inclusion.</em>

Help Wanted: A Global Push Toward Algorithmic Fairness

Across all industries, predictive, data-driven algorithms -- mathematical recipes ranging from the simple to the complex -- are being used to tell stories about individuals and can drive high-stakes decisions: who receives a loan, what sentencing a judge will recommend, what therapeutics a doctor will provide.

In the financial services industry, the exploding data ecosystem has created billions of new stories. At the Center for Financial Inclusion (CFI), we are most interested in the ones they try to tell (or don’t tell) about low-income consumers.

This publication has two main objectives:
<ul>
 	<li>to ground some universal challenges on the use of algorithms, automated decisions, alternative data, and bias in the context of inclusive financial services</li>
 	<li>to present the current state of play among inclusive finance actors from desk research and interviews with a sample of fintechs, regulators, and other experts.</li>
</ul>
Our insights are aimed primarily at financial inclusion stakeholders who can influence the trajectory of the inclusive finance industry, with specific recommendations for regulators, investors, and donors. But anyone with an interest in algorithmic decision making or financial inclusion will find this publication thought provoking and useful.

The full publication is divided into the following sections:
<ul>
 	<li>The <strong>introduction</strong> explores algorithms and bias in inclusive finance, presents a framework for understanding algorithmic bias, and delves into what we want to know about algorithms, bias, and financial inclusion.</li>
 	<li>Section 1 is about <strong>data trails in the digital economy</strong>.</li>
 	<li>Section 2 digs into the <strong>risks of bias and emerging tools</strong> through the three categories: inputs, code, and context.</li>
 	<li>Section 3 lays out <strong>recommendations for inclusive finance stakeholders</strong> to advance evidence, solutions, and incentives for responsible algorithms.</li>
</ul>

The Stories Algorithms Tell: Bias and Financial Inclusion at the Data Margins

In emerging economies, the growth of mobile phone use has contributed to an explosion of data that can be fed into algorithms to make predictions about individuals, including who is creditworthy, and offer financial products and services. As of 2020, <a href="https://www.gsma.com/r/wp-content/uploads/2020/09/GSMA-State-of-Mobile-Internet-Connectivity-Report-2020.pdf" target="_blank" rel="noopener noreferrer">half of the world’s population used mobile internet</a> and digital finance offerings have grown in tandem, often framed as engines of financial inclusion. The massive growth in available data and companies to serve customers that had previously been invisible to the financial system might suggest that the problem of thin-file customers has been solved and long-standing financial inclusion gaps are closing.

But the truth is, we don’t know what the impact is yet.

So amidst the rush to generate and leverage new data trails, CFI is embarking on a learning agenda around financial inclusion at the data margins. In pursuit of that, we are rethinking assumptions, and learning from broader tech-oriented conversations about the intersection of alternative data, algorithms and digital profiling, and how these more technical issues interact with socio-economic and gender-oriented issues surrounding digital literacy, financial capabilities, and social norms.
<h2>What Does it Mean to Lend to a Phone Line?</h2>
In the context of financial inclusion, making people visible to the financial market nowadays increasingly means having a digital financial identity based on volumes of (mobile) data that’s both related and unrelated to the actual financial behaviors of an individual. The expansion of digital lending, for example, utilizes non-traditional mobile data such as SMS, device usage and model information, social network activity, contact lists, call logs, app use, location data, and more to construct a picture of an individual’s cash flow or income.

Some of these sources are intuitively linked to financial activity, such as SMS receipts for mobile wallet transactions. Other data -- referred to by some as <a href="https://www.upturn.org/static/files/Knowing_the_Score_Oct_2014_v1_1.pdf" target="_blank" rel="noopener noreferrer">“fringe” alternative data</a> -- like social media activity, is further removed and its value as a fair and accurate predictor of creditworthiness is less established.

Given that data-driven models leverage phone-specific variables, the limits on women’s phone use or migrants’ access to a SIM card will change the data trail they leave and, in turn, the decision that an algorithm may make about their access to a loan.
<blockquote class="pullquote--right">The limits on women’s phone use or migrants’ access to a SIM card may change the decision an algorithm makes about their access to a loan.</blockquote>
In developing economies, women are less likely to own a phone and <a href="https://www.gsma.com/mobilefordevelopment/wp-content/uploads/2020/05/GSMA-The-Mobile-Gender-Gap-Report-2020.pdf" target="_blank" rel="noopener noreferrer">access mobile internet</a>. Phone-sharing commonly acts as a barrier to phone ownership for women as well as other vulnerable groups like migrants or refugees. It’s prevalent due to cost constraints or gaps in identity documents required for registering a SIM card. What does it mean when an algorithm is facilitating lending to a phone line, rather than an individual? What are the implications of this and other extant barriers in determining how inclusive these new tools are?
<h2>Risks of Data Marginalization: Where to Begin?</h2>
In the U.S. and Europe, risks and evidence of data marginalization have prompted calls for greater oversight, but little research has documented what these risks look like in the data environments of developing markets. There is much that we need to learn:
<ul>
 	<li>How large is the risk of cementing exclusion in a data-driven financial system for consumers at the data margins?</li>
 	<li>What strategies and tools are available to companies, investors, and regulators to address these risks, and what resources and skills do they require to be effective?</li>
 	<li>What are the incentives for private sector companies to act on this risk?</li>
 	<li>Globally, we see strides made on data privacy and emerging digital rights, but how will people with limited digital capability be able to meet these new rights and responsibilities?</li>
</ul>
CFI and others, like our friends at <a href="https://www.womensworldbanking.org/insights-and-impact/report-algorithmic-bias-financial-inclusion-and-gender-primer-opening-up-new-credit-to-women-in-emerging-economies/" target="_blank" rel="noopener noreferrer">Women’s World Banking</a>, are beginning to address these questions and establish a learning agenda for the sector.

It is critical we set out on a path for answers now in this period of global data ecosystem transformation. As a fintech from Southeast Asia explained, “You can’t make those [anti-bias] changes when you’re doing a million loans a day… a little bit of a loss now and a gain later is better than taking a loss at that scale.”

Let this remark serve as a wake-up call for all development stakeholders that are building data solutions today not to lose sight of the people behind the mobile phones and to examine how technology and data can help us build an inclusive tomorrow.

This topic is one of CFI’s strategic pillars, so consider this the first mile of a learning journey that will take all of us to a better understanding of data risks and opportunities as they relate to financial inclusion. Stay tuned to this space, sign up for <a href="https://centerforfinancialinclusion.us5.list-manage.com/subscribe/post?u=394352ae963270aef6c51680d&amp;id=9df382106a" target="_blank" rel="noopener noreferrer">CFI email updates</a>, and follow us on social media for coming explorations of data risks and opportunities and how they affect low-income consumers.

Rethinking Financial Inclusion at the Data Margins

How financial service providers use data and what must be done to protect already-underserved populations against algorithmic bias in automated decision-making.

Financial Inclusion at the Data Margins

Filter

Filter

Reducing Bias in Algorithmic Decisions Cannot Rely on “Blind” Approaches

Help Wanted: A Global Push Toward Algorithmic Fairness

The Stories Algorithms Tell: Bias and Financial Inclusion at the Data Margins

Rethinking Financial Inclusion at the Data Margins