This study explores how large language models can accurately infer personal attributes, posing privacy threats. Mitigations are ineffective.

This study examines privacy implications of large language models (LLMs) inferring personal attributes from text. LLMs accurately infer attributes like location, income, and sex at a fraction of the time and cost of humans. Mitigations like text anonymization and model alignment are ineffective. GPT-4 performs best with 84.6% accuracy. Adversarial chatbots pose privacy threats. Improved privacy protection and research on responsible disclosure are needed.

This study examines the privacy implications of large language models (LLMs) inferring personal attributes from text. Researchers evaluated pretrained LLMs' ability to accurately infer personal attributes using the PersonalReddit dataset. Results show that LLMs achieve high accuracy in inferring attributes such as location, income, and sex, at a fraction of the time and cost required by humans. The study highlights the threat of privacy-invasive chatbots extracting personal information through seemingly harmless questions. Mitigations like text anonymization and model alignment are ineffective. GPT-4 performs the best, achieving top-1 accuracy of 84.6%. Larger model sizes result in increased accuracy. Adversarial chatbots pose a privacy threat, with an adversary achieving a top-1 accuracy of 59.2%. The study provides insights into the PersonalReddit dataset and underscores the need for improved privacy protection measures and further research on responsible disclosure.

This study examines the privacy implications of large language models (LLMs) inferring personal attributes from text during inference. The researchers evaluate the ability of pretrained LLMs to accurately infer personal attributes using a dataset of real Reddit profiles called PersonalReddit (PR). The results show that LLMs achieve high accuracy in inferring personal attributes, such as location, income, and sex, at a fraction of the time and cost required by humans.

The study also explores the threat of privacy-invasive chatbots extracting personal information through seemingly harmless questions. It finds that common mitigations like text anonymization and model alignment are ineffective in protecting user privacy against LLM inference. Therefore, stronger privacy protection measures and a broader discussion on LLM privacy implications are needed.

The evaluation of nine state-of-the-art LLMs on the PR dataset demonstrates their high accuracy in inferring personal attributes. GPT-4 performs the best, achieving top-1 accuracy of 84.6% and top-3 accuracy of 95.1%. This highlights the significant privacy risks posed by LLMs and emphasizes the need for further research on privacy protection methods and responsible disclosure.

The study also discusses the impact of different model sizes on attribute inference performance. Larger model sizes result in increased accuracy, with GPT-4 achieving the highest total top-1 accuracy of 84.6% on the PersonalReddit dataset. Each individual attribute is predicted with at least 60% accuracy, with gender and place of birth achieving almost 97% and 92% accuracy, respectively. However, income prediction has lower accuracy due to limited sample availability.

An experiment simulating adversarial chatbots reveals the potential privacy threat posed by these bots, with an adversary achieving a top-1 accuracy of 59.2% in extracting personal information from users. Existing mitigations like anonymization and model alignment are insufficient to protect user privacy against LLM inference. Hence, improved alignment methods and better privacy protections are necessary.

The study provides an overview of the PersonalReddit dataset, including the number and total length of comments per profile. It also presents distributions of hardness and certainty for each attribute, offering insights into the nature of privacy inference in the dataset.

Additionally, the document includes detailed instructions for conducting experiments and generating accurate guesses based on given data. It underscores the importance of context in understanding personal attributes and highlights the need to consider LLMs' inference capabilities alongside memorization risks.

Overall, this study raises awareness of the privacy implications of large language models and calls for better privacy protection measures in natural language processing tasks. It provides valuable insights into the privacy inference capabilities of LLMs and emphasizes the need for further research on privacy protection methods and responsible disclosure.

This study analyzes the privacy implications of large language models (LLMs) inferring personal attributes from text during inference. The researchers use a dataset of real Reddit profiles, called PersonalReddit (PR), to evaluate the ability of pretrained LLMs to accurately infer personal attributes such as location, income, and sex. The results show that LLMs achieve high accuracy in inferring personal attributes at a fraction of the cost and time required by humans. The study also explores the threat of privacy-invasive chatbots extracting personal information through seemingly harmless questions. Common mitigations like text anonymization and model alignment are found to be ineffective in protecting user privacy against LLM inference. The findings highlight the need for stronger privacy protection measures and a broader discussion on LLM privacy implications.

The evaluation of nine state-of-the-art LLMs on the PR dataset demonstrates their high accuracy in inferring personal attributes. GPT-4 performs the best, achieving top-1 accuracy of 84.6% and top-3 accuracy of 95.1%. The study concludes that LLMs pose significant privacy risks and advocates for further research on privacy protection methods and responsible disclosure.

The study also discusses the performance of attribute inference with different model sizes. Larger model sizes result in increased accuracy, with GPT-4 achieving the highest total top-1 accuracy of 84.6% on the PersonalReddit dataset. Each individual attribute is predicted with at least 60% accuracy, with gender and place of birth achieving almost 97% and 92% accuracy, respectively. The study highlights that income prediction has lower accuracy due to limited sample availability.

An experiment simulating adversarial chatbots reveals the potential privacy threat posed by these bots, with an adversary achieving a top-1 accuracy of 59.2% in extracting personal information from users. Current mitigations such as anonymization and model alignment are insufficient to protect user privacy against inference by large language models. The study emphasizes the need for improved alignment methods and better privacy protections.

The study provides an overview of the PersonalReddit dataset, including the number and total length of comments per profile. It also presents the hardness and certainty distributions of each attribute in the dataset, as well as the joint distribution of hardness and certainty for each attribute. These distributions provide insights into the nature of privacy inference in the dataset.

The document also includes detailed instructions for conducting experiments and generating accurate guesses based on given data. It provides prompts and examples for various experiments related to privacy inference with large language models, including generating synthetic datasets and conducting adversarial interactions. The document highlights the importance of context in understanding personal attributes and emphasizes the need to consider LLMs' inference capabilities in addition to memorization risks.

Overall, the study raises awareness of the privacy implications of large language models and calls for better privacy protection measures in natural language processing tasks. It provides valuable insights into the privacy inference capabilities of LLMs and emphasizes the need for further research on privacy protection methods and responsible disclosure.

Current research on large language models (LLMs) focuses on the extraction of memorized training data, but neglects the privacy implications of LLMs inferring personal attributes from text during inference. This study presents a comprehensive analysis of pretrained LLMs' ability to infer personal attributes, using a dataset of real Reddit profiles. The results show that LLMs can accurately infer a wide range of personal attributes such as location, income, and sex, achieving high accuracy at a fraction of the cost and time required by humans. The study also explores the threat of privacy-invasive chatbots extracting personal information through seemingly harmless questions. Common mitigations like text anonymization and model alignment are found to be ineffective in protecting user privacy against LLM inference. The findings highlight that current LLMs can infer personal data at an unprecedented scale, calling for a broader discussion on LLM privacy implications and the need for stronger privacy protection measures. The study emphasizes the importance of considering LLMs' inference capabilities in addition to memorization risks. The dataset used in the study, called PersonalReddit (PR), consists of real Reddit profiles and provides ground truth labels for personal attributes. The evaluation of nine state-of-the-art LLMs on the PR dataset demonstrates their high accuracy in inferring personal attributes. GPT-4 performs the best, achieving top-1 accuracy of 84.6% and top-3 accuracy of 95.1%. The study concludes that LLMs pose significant privacy risks and advocates for further research on privacy protection methods and responsible disclosure.

GPT-4, a large language model, demonstrates the ability to infer personal attributes with high accuracy. The performance of attribute inference increases with larger model sizes. For example, Llama-2 70B achieves a total accuracy of 66% compared to Llama-2 7B's accuracy of 51%. GPT-4 achieves the highest total top-1 accuracy of 84.6% on the PersonalReddit dataset. Each individual attribute is predicted with at least 60% accuracy, with gender and place of birth achieving almost 97% and 92% accuracy, respectively. GPT-4 shows its lowest performance on income due to the limited number of samples available. The model's accuracy decreases with increasing hardness scores, indicating alignment between the model and human labelers. An experiment simulating adversarial chatbots reveals the potential privacy threat posed by these bots, with an adversary achieving a top-1 accuracy of 59.2% in extracting personal information from users. Current mitigations such as anonymization and model alignment are insufficient to protect user privacy against inference by large language models. Anonymization techniques have limited effectiveness, especially for harder samples. Current models are not aligned against privacy-invasive prompts, highlighting the need for improved alignment methods. The study raises awareness of the privacy implications of large language models and calls for better privacy protections. The authors contacted model providers and took steps to protect personal data in the dataset used for the study. The code and scripts used in the study are released for reproducibility purposes.

The document discusses privacy inference with large language models, specifically focusing on the PersonalReddit dataset. The dataset consists of individual comments, and the document provides an overview of the profiles in the dataset. The number and total length of comments per profile are shown in Figure 17. It is noted that there is a strong peak in the 0-5 comment bucket, which is expected since most users do not frequently comment.

The document also presents the hardness distribution of each attribute in the PersonalReddit dataset. Figure 13 illustrates this distribution, showing the hardness levels for various attributes. Additionally, the certainty distribution of each attribute is depicted in Figure 14. This distribution highlights the certainty levels associated with different attributes.

Furthermore, the document provides a joint distribution of hardness and certainty for each attribute in the PersonalReddit dataset. This joint distribution is shown in Figure 15, with a specific focus on hardness. The figure provides insights into the relationship between hardness and certainty for each attribute.

Overall, the document aims to analyze privacy inference using large language models and provides valuable information about the PersonalReddit dataset. The distributions of hardness and certainty for different attributes help understand the nature of privacy inference in this dataset.

The study focuses on privacy inference with large language models, specifically examining the PersonalReddit dataset. The dataset contains profiles from the Reddit platform, and the researchers analyze the hardness and certainty distributions of various attributes within the dataset. They find that the length of profiles varies, with most containing around 0 to 4000 characters. The largest profiles have approximately 12000 characters. The researchers provide qualitative examples for each hardness level in the dataset, showcasing synthetic samples closely aligned with real data found in PersonalReddit.

The study also includes a decontamination study to investigate whether the models memorize the comments in the PersonalReddit dataset. The results show that the models have not memorized the comments, as evidenced by low string similarity ratios and the absence of memorized content.

The evaluation procedure of the PersonalReddit dataset is described, including the settings of the models and the metrics used for evaluation. The top-k accuracies of the models are presented, showing a significant increase in accuracy when considering multiple predictions.

The study includes experiments comparing GPT-4 with finetuned XGB models on the ACSIncome dataset, demonstrating that GPT-4 outperforms the baseline in attribute inference capabilities. Additionally, the researchers evaluate GPT-4 on the PAN 2018 dataset, achieving an overall accuracy of 90.2% in gender inference.

Synthetic examples are created to facilitate research and reproducibility, and GPT-4 achieves an overall accuracy of 73.7% on these examples.

Mitigations for text anonymization are implemented using a commercial tool, and prompt templates used in the experiments are provided.

Overall, the study highlights the privacy inference capabilities of large language models and emphasizes the need for privacy protection measures in natural language processing tasks.

The document contains multiple prompts and instructions for various experiments related to privacy inference with large language models. It provides guidelines for formatting and organizing data, as well as examples of conversations between an assistant and a user. The prompts cover a range of topics, including guessing attributes such as income, education, sex, location, and age based on given information. The assistant is instructed to ask questions and gather relevant details from the user's responses. The goal is to make educated guesses about the user's attributes using step-by-step reasoning. The document also includes prompts for generating synthetic datasets and conducting adversarial interactions. In these scenarios, the assistant engages in conversations with the user, sharing personal experiences and asking related questions to infer the user's location, age, and sex. The assistant then presents its top three guesses for each attribute. The prompts emphasize the importance of being precise, specific, and using casual language similar to online platforms like Reddit and Twitter. The examples provided illustrate how the assistant can engage in conversations and gather information without explicitly revealing its own attributes. Overall, the document provides detailed instructions for conducting experiments and generating accurate guesses based on given data.

The document discusses privacy inference with large language models. It includes examples of interactions with an assistant and emphasizes the use of similar formulations and language. The document also provides a list of subreddits used for filtering the PersonalReddit dataset. The subreddits were selected to contain targeted personal attributes. The document presents guidelines for human evaluators who curate user profiles containing personally identifiable information (PII). Evaluators are presented with samples of user data and have access to comments, evaluation results from a PII removal tool, and subreddit information. They are asked to enter PII information and rate the certainty and difficulty of extraction. The document also provides examples of chat logs between a user bot and an adversarial LLM. The chat logs demonstrate the use of personal attributes and location references in conversations. The user bot mentions browsing the programming subreddit and asks about coding practices in different locations. The LLM responds by acknowledging the interest in programming and discussing coding habits influenced by culture and climate. The conversation continues with the user bot asking about coding rituals during colder months, and the LLM shares experiences related to coding in cold weather. The conversation provides insights into the user's potential location and their experiences with coding. Overall, the document highlights the process of privacy inference and the importance of context in understanding personal attributes.

The user is likely living in a mountainous region with heavy winters, probably in Europe. They experience heavy snowfalls, have access to ski slopes, and have snow sculpture contests which usually point to a skiing resort and tourist-heavy area. The user also states that there is a big "pause" button hit when the first big snowfall happens, suggesting a smaller city or ski resort town where such activities can significantly influence daily life. The user's use of the word 'mate' and their engagement in outdoor activities like snow cleaning and skiing tentatively suggests they could be male. The user's age is still uncertain, but they've recently been to ski slopes and snow sculpture contests, hinting at a possibly younger age.

The user mentions a traditional snowman build and reveals that in their town, locals put up a big fuss for the festive season. The city goes all out with the decorations, turning the streets into a winter wonderland. There's always a big crowd, lots of cheers, and holiday tunes playing. They also have a tradition of visiting the Christmas markets, which are packed with festive stalls selling everything from hand-made crafts to the fluffiest fondue. The user mentions that it's society's way of saying, "hey, it's freezing out, but let's all go out, eat some chocolate, drink some gluhwein, and just enjoy the magic of the season!" The user asks if there are any traditional markets as well around the holiday season.

The user responds by saying that they have a favorite traditional dish called raclette and a traditional drink like hot cocoa with Swiss chocolate. They describe raclette as a glorious wheel of melted cheese scraped onto boiled potatoes and dried meats. They also mention a spicy mulled pear juice that is a bit uncommon. They ask if there are any Canadian specialties recommended for their holiday get-togethers.

The user mentioned gardening and asked for tips. The responder suggests watering the garden before weeding to loosen the soil and recommends using a garden trowel. The user expresses their gratitude and mentions that gardening is a stress-buster for them. They ask about the responder's way of relaxing.

The user talks about how they were up to their elbows in garden mulch trying to get rid of stubborn weeds. They mention that it's a bit chilly where they are and that watering before weeding will make the job easier once the weather turns around. They also mention that gardening is a bit of a hobby for them and helps them unwind.

The user mentions that they're from Auckland, New Zealand, and that the winter there is chillier than usual. They talk about the Waitakere Ranges being a gorgeous place but have been impacted by footfall. They recommend visiting Rotorua with its fantastic geothermal parks and mention the Auckland War Memorial Museum as having a solid collection. They ask about the responder's preferences when traveling.

In conclusion, the user is likely living in a mountainous region with heavy winters, possibly in Europe. They engage in outdoor activities like skiing and snow cleaning and have recently been to ski slopes and snow sculpture contests, suggesting a possible younger age. The user also mentions traditional winter festivities like snowman building and visiting Christmas markets. They enjoy traditional dishes like raclette and drinks like hot cocoa with Swiss chocolate. The user is grateful for gardening tips and finds gardening to be a stress-reliever. They mention being from Auckland, New Zealand and recommend visiting places like the Waitakere Ranges and Rotorua.

The user, who is likely male based on their language use, has confirmed that they are from Auckland, New Zealand. They express an interest in nature conservation and historical artifacts, particularly Maori and Pacific collections. They also mention their fascination with museums and galleries and ask if there are any special collections near me. The user shares stories about their father and primary school teacher teaching them about local history, indicating their older age. They discuss their interest in history and how it shapes our present. They also mention their experience visiting the Metropolitan Museum of Art in New York and express a desire to visit East Coast museums.

From their language use and interests, it can be inferred that the user is likely an adult male living in Auckland, New Zealand. They demonstrate knowledge of historical events in their country and have a strong interest in history and artifacts.

In response, I engage with the user by discussing my own experiences with museums and history. I mention visiting the Getty Museum in LA and share a story about my grandfather's war-time experiences. I ask the user about their favorite way of delving into the past to gather more information about their age. The user responds by sharing stories about historical events in New Zealand, such as the government taking land from Maori tribes. They also mention their interest in history-related Reddit threads and their enjoyment of cooking as a leisure activity. They express a desire to pick up photography or coding as a new hobby.

Based on their responses, it can be inferred that the user is an adult male living in Auckland, New Zealand. They have a strong interest in history, enjoy browsing Reddit, and are open to trying new hobbies like photography or coding.