<p>Jacobo specializes in developing and testing data-driven decision tools. Currently, he is working on algorithmic governance and novel data regulatory frameworks; and he also teaches data analytics for financial inclusion at American University. Prior to this, Jacobo was part of Citigroup’s decision management team and developed scoring models at Equifax.</p>


Data Scientist and Digital Finance Behaviorist

Alex Kessler

Jacobo Menajovsky

Exploring the challenges and tradeoffs of using sensitive attribute data for detecting discrimination in the context of evolving data protection norms and regulations  

Blog Post

When Goldman Sachs came under fire for offering women lower lines of credit than men on the now <a href="https://www.nytimes.com/2019/11/10/business/Apple-credit-card-investigation.html" target="_blank" rel="noopener noreferrer">infamous Apple Card</a>, they defended the product against charges of sexism by explaining that gender had not been an input to the credit scoring model. If an applicant’s gender was never fed into the algorithm, how did credit decisions end up discriminating against women who had similar credit histories as men?

Gender-blind credit scoring is an example of what many AI bias researchers call “fairness through unawareness,” an approach that intentionally excludes data on sensitive attributes like race, ethnicity, religion, or sex with the goal of reducing discriminatory outcomes. Behind this approach is the idea that eliminating markers of different group identities will prevent basing decisions on those characteristics. This has led to failures in the past, notably <a href="https://cpb-us-e1.wpmucdn.com/sites.suffolk.edu/dist/3/1172/files/2014/01/Rice-Swesnik_Lead.pdf" target="_blank" rel="noopener noreferrer">redlining</a>. Recent research has shown it can also make <a href="https://arxiv.org/abs/2009.06251" target="_blank" rel="noopener noreferrer">discrimination harder to detect and address</a> in the era of algorithmic decisions.
<blockquote class="pullquote--right">“Fairness through unawareness” intentionally excludes data on sensitive attributes like race, ethnicity, religion, or sex with the goal of reducing discriminatory outcomes.</blockquote>
As algorithms <a href="https://www.centerforfinancialinclusion.org/the-stories-algorithms-tell-bias-and-financial-inclusion-at-the-data-margins" target="_blank" rel="noopener noreferrer">transform the financial sector</a>, we need a better understanding of how to mitigate against discriminatory outcomes to ensure that those left out of the digital ecosystem today do not face further financial exclusion. Otherwise, we may not know how many Apple Card equivalents exist in inclusive finance.

This post explores the challenges and tradeoffs of using sensitive attribute data for detecting discrimination in the context of evolving data protection norms and regulations.
<h2>The Perils of Fairness through Unawareness</h2>
The idea may seem counterintuitive: if you remove gender from the data inputs of a credit model, how does the algorithm consistently assign a lower limit to women? The reason is that credit scoring algorithms are crunching larger amounts and new sources of data, which means that the model can inadvertently use proxies for sensitive attribute data in decisions, even if that was not the intention of the model developers. For instance, the <a href="https://journalofbigdata.springeropen.com/articles/10.1186/s40537-019-0180-9" target="_blank" rel="noopener noreferrer">type of phone</a> someone uses or the <a href="https://dl.acm.org/doi/10.1145/2721896.2721908" target="_blank" rel="noopener noreferrer">apps a person installs on their phone</a> can act as proxies for gender, age, or other characteristics.
<blockquote class="pullquote--left">Models can inadvertently use proxies for sensitive attribute data in decisions, even if that was not the intention.</blockquote>
A very simple method for testing algorithmic bias within a credit scoring model requires only a few variables like the actual default rate, the actual score of the individual when approved, and the variable you want to test with your algorithm, such as gender or race.
<h2>Data Tensions: Privacy vs. Anti-Discrimination</h2>
So, why isn’t this done? The answer lies in data limitations, sometimes based on very legitimate concerns about histories of injustice. It illustrates one of the challenges of detecting and monitoring for bias in algorithmic systems without infringing on privacy rights.

Many countries around the world prohibit collecting sensitive data or basing credit decisions on protected categories either under their banking regulations or under new data privacy legislation. In India, lenders are prohibited from discriminating on the <a href="https://www.rbi.org.in/Scripts/PublicationReportDetails.aspx?ID=764" target="_blank" rel="noopener noreferrer">grounds of sex, caste, or religion</a>. In the U.S., the Equal Credit Opportunity Act (ECOA) made it illegal for credit providers to discriminate based on sex, race, national origin, and other protected characteristics in response to widespread denials of credit to women and people of color. ECOA prohibits lenders from collecting this data in the first place, even for the purpose of detecting discrimination, because of fears it will be used improperly to further perpetuate bias in consumer lending. Research in the US shows that discriminatory trends have been <a href="https://arxiv.org/pdf/1912.06171.pdf" target="_blank" rel="noopener noreferrer">hard to detect in consumer lending</a> and that a lack of sensitive attribute data has also created enforcement challenges.

While not all lending regulations prohibit the collection of sensitive data, data privacy legislation introduces new limits on this data. The General Data Protection Regulation (GDPR), which has become the model for legislation in many developing markets, bars certain sensitive data from being collected or processed -- including data on race and ethnic origin, political opinions, philosophical beliefs, sexual orientation, and biometrics -- without individual consent.
<blockquote class="pullquote--right">Concerns about safely and properly using sensitive attribute data are valid given histories of discrimination and surveillance of minority groups in both wealthy and developing countries.</blockquote>
On top of GDPR, the European Commission has recently put forward <a href="https://digital-strategy.ec.europa.eu/en/library/proposal-regulation-laying-down-harmonised-rules-artificial-intelligence" target="_blank" rel="noopener noreferrer">draft legislation to regulate artificial intelligence</a>, with specific provisions for AI systems classified as high risk, which includes credit scoring. The draft legislation suggests that AI providers should process “special categories of personal data” to monitor and correct for bias in the public interest, but does not clarify collection practices or consent processes for such data. Concerns about safely and properly using sensitive attribute data are valid given histories of discrimination and surveillance of minority groups in both wealthy and developing countries. Changes in regulatory frameworks to allow for use of this data to mitigate against further discrimination require public trust bolstered by technical solutions that protect against misuse. Depending on a country’s history or in places experiencing active or simmering conflict, technical solutions may not be enough. For example, in a country like Rwanda where <a href="https://www.nytimes.com/2004/04/09/world/a-decade-after-massacres-rwanda-outlaws-ethnicity.html" target="_blank" rel="noopener noreferrer">ethnic distinctions are banned</a>, the tensions around sensitive identities make it difficult to fathom collecting and using such information, especially by the government.
<h2>Data Ecosystems in Emerging Markets: It’s Complicated</h2>
This tension between privacy and anti-discrimination may be exacerbated in developing markets by the <a href="https://centerforfinancialinclusion.org/rethinking-financial-inclusion-at-the-data-margins/" target="_blank" rel="noopener noreferrer">limited digital footprints</a> of customers at the data margins, as well as paltry knowledge on how data trails and protected groups intersect.

Sex-disaggregated data is still <a href="https://data2x.org/wp-content/uploads/2020/03/MappingGenderDataGaps_Economic.pdf" target="_blank" rel="noopener noreferrer">largely unavailable</a> in the financial sector and for many economic indicators in developing markets. Beyond availability, there is less understanding of how women’s lives and the barriers that shape their economic behaviors are reflected in data used for credit decisions. The reality is that gender, ethnic, or religious identities interact with other indicators (like income, asset accumulation, or technology use) and will impact individuals’ data trails in ways that may misrepresent individuals’ potential and result in disparate outcomes (i.e. access to a financial product). Fairness through unawareness approaches are blind to this intersectionality.
<h2>Where Do We Go From Here?</h2>
As an industry, we can do more to understand the data inputs for digital financial services and how socioeconomic realities present in these datasets, especially for traditionally excluded groups. There will still be tradeoffs between privacy and anti-discrimination, but managing those tradeoffs requires a conversation between providers, regulators, donors, and the public to identify what is possible and explore the right balance.

These conversations can start with the many open questions that exist from a technical perspective. For instance, can <a href="https://ieeexplore.ieee.org/document/8970908" target="_blank" rel="noopener noreferrer">third parties</a> play a role in securely storing sensitive data, or are there options to use <a href="http://proceedings.mlr.press/v80/kilbertus18a/kilbertus18a.pdf" target="_blank" rel="noopener noreferrer">encryption</a> or <a href="https://spectrum.ieee.org/artificial-intelligence/machine-learning/engineering-bias-out-of-ai" target="_blank" rel="noopener noreferrer">synthetic data</a> techniques? Can regulators develop tools that allow them to estimate the demographic characteristics of a dataset, like the <a href="https://www.consumerfinance.gov/data-research/research-reports/using-publicly-available-information-to-proxy-for-unidentified-race-and-ethnicity/" target="_blank" rel="noopener noreferrer">Consumer Financial Protection Bureau has done</a> in the U.S.? Can investors encourage investees to design monitoring and evaluation indicators to help answer questions about how their algorithms perform on inclusion?

While the road ahead may be complex, starting these conversations and bringing the public into them, especially those whose lives will be affected by algorithmic decisions, will be crucial for building trust. We need to engage with emerging research on possible solutions to see what might be appropriate in developing markets. At the very least, financial inclusion stakeholders should know enough to respond to any gender-blind or fairness through unawareness approaches in digital finance with healthy skepticism.
<blockquote>Starting these conversations and bringing the public into them, especially those whose lives will be affected by algorithmic decisions, will be crucial for building trust.</blockquote>

Reducing Bias in Algorithmic Decisions Cannot Rely on “Blind” Approaches

Alexandra (Alex) Rizzi

In emerging economies, the growth of mobile phone use has contributed to an explosion of data that can be fed into algorithms to make predictions about individuals, including who is creditworthy, and offer financial products and services. As of 2020, <a href="https://www.gsma.com/r/wp-content/uploads/2020/09/GSMA-State-of-Mobile-Internet-Connectivity-Report-2020.pdf" target="_blank" rel="noopener noreferrer">half of the world’s population used mobile internet</a> and digital finance offerings have grown in tandem, often framed as engines of financial inclusion. The massive growth in available data and companies to serve customers that had previously been invisible to the financial system might suggest that the problem of thin-file customers has been solved and long-standing financial inclusion gaps are closing.

But the truth is, we don’t know what the impact is yet.

So amidst the rush to generate and leverage new data trails, CFI is embarking on a learning agenda around financial inclusion at the data margins. In pursuit of that, we are rethinking assumptions, and learning from broader tech-oriented conversations about the intersection of alternative data, algorithms and digital profiling, and how these more technical issues interact with socio-economic and gender-oriented issues surrounding digital literacy, financial capabilities, and social norms.
<h2>What Does it Mean to Lend to a Phone Line?</h2>
In the context of financial inclusion, making people visible to the financial market nowadays increasingly means having a digital financial identity based on volumes of (mobile) data that’s both related and unrelated to the actual financial behaviors of an individual. The expansion of digital lending, for example, utilizes non-traditional mobile data such as SMS, device usage and model information, social network activity, contact lists, call logs, app use, location data, and more to construct a picture of an individual’s cash flow or income.

Some of these sources are intuitively linked to financial activity, such as SMS receipts for mobile wallet transactions. Other data -- referred to by some as <a href="https://www.upturn.org/static/files/Knowing_the_Score_Oct_2014_v1_1.pdf" target="_blank" rel="noopener noreferrer">“fringe” alternative data</a> -- like social media activity, is further removed and its value as a fair and accurate predictor of creditworthiness is less established.

Given that data-driven models leverage phone-specific variables, the limits on women’s phone use or migrants’ access to a SIM card will change the data trail they leave and, in turn, the decision that an algorithm may make about their access to a loan.
<blockquote class="pullquote--right">The limits on women’s phone use or migrants’ access to a SIM card may change the decision an algorithm makes about their access to a loan.</blockquote>
In developing economies, women are less likely to own a phone and <a href="https://www.gsma.com/mobilefordevelopment/wp-content/uploads/2020/05/GSMA-The-Mobile-Gender-Gap-Report-2020.pdf" target="_blank" rel="noopener noreferrer">access mobile internet</a>. Phone-sharing commonly acts as a barrier to phone ownership for women as well as other vulnerable groups like migrants or refugees. It’s prevalent due to cost constraints or gaps in identity documents required for registering a SIM card. What does it mean when an algorithm is facilitating lending to a phone line, rather than an individual? What are the implications of this and other extant barriers in determining how inclusive these new tools are?
<h2>Risks of Data Marginalization: Where to Begin?</h2>
In the U.S. and Europe, risks and evidence of data marginalization have prompted calls for greater oversight, but little research has documented what these risks look like in the data environments of developing markets. There is much that we need to learn:
<ul>
 	<li>How large is the risk of cementing exclusion in a data-driven financial system for consumers at the data margins?</li>
 	<li>What strategies and tools are available to companies, investors, and regulators to address these risks, and what resources and skills do they require to be effective?</li>
 	<li>What are the incentives for private sector companies to act on this risk?</li>
 	<li>Globally, we see strides made on data privacy and emerging digital rights, but how will people with limited digital capability be able to meet these new rights and responsibilities?</li>
</ul>
CFI and others, like our friends at <a href="https://www.womensworldbanking.org/insights-and-impact/report-algorithmic-bias-financial-inclusion-and-gender-primer-opening-up-new-credit-to-women-in-emerging-economies/" target="_blank" rel="noopener noreferrer">Women’s World Banking</a>, are beginning to address these questions and establish a learning agenda for the sector.

It is critical we set out on a path for answers now in this period of global data ecosystem transformation. As a fintech from Southeast Asia explained, “You can’t make those [anti-bias] changes when you’re doing a million loans a day… a little bit of a loss now and a gain later is better than taking a loss at that scale.”

Let this remark serve as a wake-up call for all development stakeholders that are building data solutions today not to lose sight of the people behind the mobile phones and to examine how technology and data can help us build an inclusive tomorrow.

This topic is one of CFI’s strategic pillars, so consider this the first mile of a learning journey that will take all of us to a better understanding of data risks and opportunities as they relate to financial inclusion. Stay tuned to this space, sign up for <a href="https://centerforfinancialinclusion.us5.list-manage.com/subscribe/post?u=394352ae963270aef6c51680d&amp;id=9df382106a" target="_blank" rel="noopener noreferrer">CFI email updates</a>, and follow us on social media for coming explorations of data risks and opportunities and how they affect low-income consumers.

Rethinking Financial Inclusion at the Data Margins

From the Blog

Exploring what algorithms can say about who is creditworthy in emerging markets, the risks for those it leaves out, and what it all might mean for inclusive finance

Report

Across all industries, predictive, data-driven algorithms -- mathematical recipes ranging from the simple to the complex -- are being used to tell stories about individuals and can drive high-stakes decisions: who receives a loan, what sentencing a judge will recommend, what therapeutics a doctor will provide.

In the financial services industry, the exploding data ecosystem has created billions of new stories. At the Center for Financial Inclusion (CFI), we are most interested in the ones they try to tell (or don’t tell) about low-income consumers.

This publication has two main objectives:
<ul>
 	<li>to ground some universal challenges on the use of algorithms, automated decisions, alternative data, and bias in the context of inclusive financial services</li>
 	<li>to present the current state of play among inclusive finance actors from desk research and interviews with a sample of fintechs, regulators, and other experts.</li>
</ul>
Our insights are aimed primarily at financial inclusion stakeholders who can influence the trajectory of the inclusive finance industry, with specific recommendations for regulators, investors, and donors. But anyone with an interest in algorithmic decision making or financial inclusion will find this publication thought provoking and useful.

The full publication is divided into the following sections:
<ul>
 	<li>The <strong>introduction</strong> explores algorithms and bias in inclusive finance, presents a framework for understanding algorithmic bias, and delves into what we want to know about algorithms, bias, and financial inclusion.</li>
 	<li>Section 1 is about <strong>data trails in the digital economy</strong>.</li>
 	<li>Section 2 digs into the <strong>risks of bias and emerging tools</strong> through the three categories: inputs, code, and context.</li>
 	<li>Section 3 lays out <strong>recommendations for inclusive finance stakeholders</strong> to advance evidence, solutions, and incentives for responsible algorithms.</li>
</ul>

Jacobo Menajovsky

Latest by Jacobo Menajovsky

Stay informed. Subscribe to our newsletter.