Summary How We Investigated France’s Mass Profiling Machine - Lighthouse Reports www.lighthousereports.com
2,977 words - html page - View html page
One Line
Lighthouse Reports exposed discriminatory practices and privacy breaches in France's CNAF predictive risk assessments, where low-income and disability benefits led to higher risk scores.
Slides
Slide Presentation (10 slides)
Key Points
- Lighthouse Reports and its media partners have investigated predictive risk assessments in welfare systems across Europe.
- Lighthouse Reports partnered with Le Monde to investigate an algorithm deployed by Frances Caisse Nationale des Allocations Familiales (CNAF) to predict benefit fraud.
- The investigation found evidence of discrimination, privacy invasions, and design flaws in the algorithm.
- The algorithm assigns risk scores to over 13 million households in France, targeting people in precarious situations.
- Variables related to financial resources and demographic characteristics increase the risk scores, while high income reduces the score.
- The CNAF did not provide evidence to support the statistical substantiation of the variables used in the algorithm.
- The CNAF has never audited its models for bias or discrimination, but has convened a team to examine its use of risk-scoring.
Summaries
21 word summary
Lighthouse Reports uncovered discrimination, privacy breaches, and flaws in France's CNAF predictive risk assessments. Low-income and disability benefits raised risk scores.
73 word summary
Lighthouse Reports and media partners exposed discrimination, privacy breaches, and flaws in France's Caisse Nationale des Allocations Familiales (CNAF) predictive risk assessments. The investigation obtained CNAF's source code, revealing that low income and disability benefits increased risk scores. 35% of flagged households had to repay benefits, while 17% of investigations found the CNAF owed beneficiaries money. Access to code and deeper analysis were limited, and the CNAF provided no comprehensive statistics or evidence.
130 word summary
Lighthouse Reports and its media partners investigated the use of predictive risk assessments in welfare systems, specifically focusing on France's Caisse Nationale des Allocations Familiales (CNAF). The investigation obtained the source code for three models used by the CNAF and revealed discrimination, privacy invasions, and design flaws in the system. The CNAF algorithm utilizes logistic regression to construct risk-scoring models based on variables such as income and family status. The investigation found that low income and disability benefits increased the risk scores. Furthermore, 35% of flagged households had to repay benefits, while 17% of investigations revealed that the CNAF owed money to beneficiaries. The investigation also highlighted limitations in accessing the code and conducting deeper analysis. The CNAF defended its use of variables but provided no comprehensive statistics or evidence.
426 word summary
Lighthouse Reports and its media partners have investigated the use of predictive risk assessments in welfare systems across Europe. These assessments produce scores that can have significant consequences for vulnerable populations. The investigation focused on an algorithm used by France's Caisse Nationale des Allocations Familiales (CNAF) to predict welfare fraud. The algorithm assigns risk scores to all 13 million households receiving benefits, and Lighthouse Reports obtained the source code for three models deployed by the CNAF between 2010 and 2023. The investigation revealed evidence of discrimination, privacy invasions, and design flaws in the system.
The CNAF algorithm uses logistic regression, a simple machine learning algorithm, to construct its risk-scoring models. The models are trained on a representative dataset of households randomly investigated for fraud. The code assigns coefficients to different variables, such as income and family status, and calculates a raw score. This raw score is then transformed into a risk score using a sigmoid function. The investigation analyzed the impact of different variables on the risk scores and found that certain characteristics, such as low income and disability benefits, increased the risk scores.
The investigation also examined the performance of the CNAF model and found that 35% of flagged households had to repay part of their benefits, amounting to approximately 604 million euros since 2019. However, 17% of investigations concluded that the CNAF actually owed money to the beneficiaries. The investigation identified disparities in which profiles were more likely to pass the high-risk threshold for investigation, with stable income families having a lower average risk score compared to individuals who work and receive disability benefits.
The investigation highlighted limitations in obtaining access to runnable code and deeper analysis of the system. The relationship between different variables was not known, and granular outcome data was not available to test for bias or discrimination. The CNAF did not provide comprehensive statistics or evidence to support its use of variables in the model but stated that it is reflecting on its use of algorithms and will publish conclusions in 2024.
In conclusion, the investigation into France's mass profiling machine revealed evidence of discrimination, privacy invasions, and design flaws in the CNAF algorithm used to predict welfare fraud. The investigation analyzed the impact of different variables on risk scores, identified disparities in which profiles were more likely to be flagged for investigation, and highlighted limitations in obtaining access to code and deeper analysis. The CNAF responded to the investigation with a general statement defending its use of variables but did not provide comprehensive statistics or evidence of statistical substantiation.