UK ICO Seeks Feedback on Profiling Tools Draft Guidance

If your business relies on algorithms to keep users safe online, the UK Information Commissioner’s Office (ICO) has a message for you:
“We’d like your homework, in red pen, by the deadline.”
The ICO’s draft guidance on profiling tools for online safety is out for consultation, and it goes straight to the heart of how platforms use data, algorithms, and automated decision-making to spot harmful content and risky behavior.

In plain English, the ICO wants to know whether its guidance on profiling tools is clear, practical, and realistic for organizations that run trust and safety systems.
If you offer user-to-user services, run online communities, or sell tools that help others moderate content or protect children online, this draft guidance is very much your problemand your opportunity.

In this article, we’ll unpack what “profiling tools for online safety” actually means, why the ICO is seeking feedback, what’s inside the draft guidance, and how you can begin preparing right now.
We’ll also look at how this fits into global trends on automated decision-making and profiling, then wrap up with real-world style lessons learned from organizations wrestling with similar issues.

What Is the ICO’s Profiling Tools Draft Guidance About?

The draft guidance focuses on how organizations use profiling tools as part of
trust and safety systemsfor example, tools that analyze user behavior and content to detect grooming, harassment, self-harm content, extremism, or other online harms.
It is closely tied to the UK’s Online Safety Act 2023 (OSA), which places new duties on certain online services to protect users, particularly children, from harm.

The consultation explains that profiling tools often involve:

• Analyzing user-generated content, such as posts, images, or chat messages.
• Monitoring behavior patterns, such as who users contact, how often, and in what context.
• Assigning risk scores or labels to users or content based on those signals.
• Flagging, prioritizing, or blocking content using automated systems.

The draft guidance aims to show how all of this must comply with UK GDPR and data protection law while still enabling platforms to meet their online safety obligations.
In other words, it’s about finding the balance between “keep people safe” and “don’t over-surveil or unfairly profile them.”

What Are “Profiling Tools” in This Context?

Under UK GDPR, profiling means using personal data to analyze or predict aspects of a person’s behavior, preferences, interests, reliability, or similar characteristics.
In the online safety space, that typically involves using data to predict whether a user or a piece of content is risky.

Common examples in trust and safety systems include:

• Algorithms that scan text, images, or video to identify sexual content involving minors.
• Models that flag patterns suggesting grooming, bullying, or hate speech.
• Risk scores that prioritize which reports or accounts moderators should review first.
• Tools that identify “repeat offenders” or accounts likely to evade bans.

These tools can dramatically improve safety outcomes, especially at scale. But they also carry serious
privacy, fairness, and bias risks. If your system scores a user as “high risk,” what does that mean for their account?
Can they challenge it? Was sensitive data used? Was a child’s behavior profiled in ways that feel intrusive or disproportionate?

Why Is the ICO Seeking Feedback Now?

Several forces are converging at once:

• Growing reliance on AI and automated decision-making. Platforms increasingly depend on automated tools and machine learning models to moderate content and detect harmful behavior.
• New online safety obligations. The Online Safety Act pushes services to take more proactive steps to protect users, especially children, which often means more profiling, not less.
• Evolving data protection landscape. The ICO has been steadily updating its guidance on automated decision-making, profiling, and AI, to align with new laws and technologies.
• Public expectations. Users expect to be safe online, but they also expect not to be treated like suspects just for using social media, gaming platforms, or messaging tools.

By launching a consultation, the ICO is effectively asking:
“Does this guidance help you do the right thing in practice, or are there gaps, ambiguities, or unrealistic expectations?”
It is looking for feedback from platforms, vendors, civil society, academics, and anyone else with experience in this space.

Key Themes in the Draft Guidance

1. Lawful Basis and Necessity

As always with data protection, the first question is: What is your lawful basis for processing?
The guidance stresses that organizations must:

• Identify a clear lawful basis under UK GDPR (often legitimate interests or legal obligation, depending on the role and OSA duties).
• Show that profiling is necessary for that purpose, not just convenient.
• Document their reasoningespecially when processing children’s data or sensitive data (like health, sexuality, or political opinions).

In practice, that means you can’t just say “Safety!” and call it a day. You need to show that the profiling tool and the way you implement it are proportionate to the risks you’re trying to address.

2. Fairness, Transparency, and Explainability

The ICO’s guidance leans heavily on fairness and transparency. If you’re using profiling tools to decide how a user is treatedfor example, whether their content is downranked or their account is flaggedpeople should not feel trapped in a mysterious “algorithmic black box.”

The guidance expects organizations to:

• Explain, in user-friendly language, that profiling tools are being used as part of trust and safety processes.
• Make it clear what types of data are used and for what specific safety purposes.
• Provide meaningful information about the logic involved where tools significantly affect users.
• Offer ways for users to contest or challenge decisions that materially impact them.

You don’t have to hand over the source code of your models, but “we use AI to keep you safe” will not cut it as a transparency statement.

3. Children’s Data and Special Category Data

Profiling children can be particularly sensitive. When services are “likely to be accessed by children,” the ICO expects a higher bar for:

• Data minimizationonly collecting what’s strictly necessary.
• Stricter safeguards around profiling outcomes and retention.
• Clear justification for any profiling that goes beyond what is essential for safety.

If profiling involves or reveals special category data (for example, sexual orientation inferred from behavior, or religious views inferred from group participation), organizations must ensure they have a valid condition for processing under UK GDPR, not just a lawful basis. This is where many systems stumble unintentionally, as models “learn” patterns from data in ways designers did not explicitly plan.

4. Automated Decision-Making and Human Review

The draft guidance also touches on automated decision-making, especially when a profiling tool makes decisions with “legal or similarly significant effects” on individualsfor example, permanent account bans, escalated reporting to law enforcement, or long-term content restrictions.

Where such decisions are “solely automated,” UK GDPR imposes specific restrictions and safeguards. Organizations may need to:

• Ensure there is meaningful human review for high-impact decisions.
• Enable users to request human intervention or challenge a decision.
• Document how human reviewers are trained and supported so their involvement is more than rubber-stamping.

If your notion of “human review” is one overworked moderator clicking “approve all” at 2 a.m., the ICO will not be impressed.

5. Governance, DPIAs, and Ongoing Monitoring

The draft guidance expects serious governance around profiling tools, including:

• Data Protection Impact Assessments (DPIAs) for high-risk profiling and automated decision-making.
• Ongoing monitoring of accuracy, bias, and unintended consequences.
• Clear lines of accountabilitywho owns the tool, the data, and the risk?
• Regular reviews when models are retrained, inputs change, or new safety use cases are added.

In other words, launching a profiling tool is not a one-time project; it’s a continuous compliance and risk management exercise.

Who Should Care About This Guidance?

If you are thinking, “I don’t run a social media giant, so this doesn’t apply to me,” think again. The draft guidance is relevant to:

• User-to-user services such as social platforms, forums, messaging apps, and gaming communities.
• Online platforms used by children, including educational technology, youth-focused games, or social apps.
• Vendors of trust and safety toolsfor example, providers of content classification, risk scoring, or fraud detection tools sold to other organizations.
• Online marketplaces and platforms that use profiling to detect scams, abusive behavior, or counterfeit goods.
• Any service falling under the Online Safety Act that relies on profiling to meet its safety duties.

Even organizations outside the UK should pay attention. If you have UK users, or if your tools are used by UK platforms, this guidance will shape customer expectations and contract terms.

Practical Steps to Get Ready

While the guidance is still in draft form, there are several practical steps organizations can start taking now:

1. Map your profiling tools. Identify every system that profiles users or content for safety purposes. What data does it use? What decisions flow from it? Which users are affected?
2. Clarify purposes and lawful bases. For each profiling activity, define the purpose (e.g., detecting grooming, preventing self-harm content) and the lawful basis for processing. Be specific, not generic.
3. Review transparency materials. Check your privacy notices, in-product explanations, and safety help pages. Do they clearly explain that profiling tools are used, how they work at a high level, and what it means for users?
4. Assess automated decision-making risk. Identify decisions that may have legal or similarly significant effects. For those, review whether they are “solely automated” and what human safeguards exist.
5. Run or update DPIAs. Make sure your DPIAs reflect real-world use of profiling tools, including data flows, model training, third-party tools, and potential biases.
6. Strengthen governance. Assign clear ownership for each profiling tool, with responsibility for accuracy, fairness, user impact, and compliance.
7. Test with real scenarios. Use red-teaming, testing with edge cases, and diverse internal review to see how your tools behave in practicenot just in the design doc.

How This Fits into Global Trends on Profiling and AI

The ICO’s draft guidance does not exist in a vacuum. Around the world, regulators and policymakers are wrestling with similar questions about automated decision-making, profiling, and AI.

In the European Union, GDPR has long restricted certain types of solely automated decision-making with significant effects and placed a premium on fairness, transparency, and data protection by design. National regulators and courts have been gradually filling in the details with real cases and enforcement actions.

In the United States, privacy and AI-related laws are emerging at the state level. Some frameworks emphasize risk assessments, algorithmic impact analyses, and heightened safeguards around AI systems used for high-risk purposes such as employment, credit, or essential public services.
The themes are familiar: understand the system, document the risks, and build in safeguardsespecially when vulnerable individuals or sensitive uses are involved.

For global companies, this means that building a robust governance framework for profiling tools is no longer “nice to have”; it is the only realistic way to navigate overlapping rules, expectations, and enforcement trends across markets.

Timeline and How to Respond to the Consultation

The ICO’s consultation on the profiling tools guidance runs for a fixed period, with a published start date and closing date.
During that window, stakeholders can submit feedback through an online survey or written responses.

If your organization is likely to be affected, consider:

• Coordinating internally between legal, privacy, trust and safety, product, and engineering teams.
• Identifying where the guidance aligns with your current approachand where it poses challenges.
• Highlighting areas where you believe more clarity, examples, or flexibility are needed.
• Explaining any unintended consequences you foresee, such as discouraging effective safety practices or disadvantaging smaller services.

Thoughtful feedback now can shape the final version of the guidance and, indirectly, how regulators, courts, and users will interpret “good practice” in this area for years to come.

Real-World Experiences and Lessons Learned

What does all of this look like in practice? Let’s walk through some experience-based scenarios that mirror what many organizations are already dealing with when it comes to profiling tools and online safety.

1. The “Helpful, Until It Isn’t” Safety Model
Imagine a mid-sized social platform that introduces a new model to detect bullying and hate speech. In testing, it performs well, catching a lot of content that moderators previously missed. After a few months in production, though, user complaints spike.
Certain communitiesoften using reclaimed slurs or inside jokesare flagged disproportionately. Some users feel singled out or censored.

When the team digs in, they realize that training data reflected more mainstream language patterns. The model sees specific terms and contexts as universally harmful, even when communities use them in self-referential, non-abusive ways.
The lesson? Accuracy metrics alone are not enough. You need systematic checks for over-blocking, false positives, and differential impacts across communities, plus a way for users to appeal or request review.

2. The “Invisible” Profiling Problem
Another company uses risk scores to prioritize which reported accounts get moderator attention first. The system quietly ranks users based on factors like prior reports, content types, and behavioral patterns.
Everything happens behind the scenes, with no transparency to users and minimal documentation.

Then a high-profile user challenges a platform decision, asking how the decision was made. The company struggles to explain what the risk score really meant and how it influenced the outcome.
It turns out that a combination of automated profiling and quick human review led to what was functionally a high-impact decisionbut nobody had treated it as such for compliance purposes.

The takeaway here is that if a profiling tool meaningfully drives high-impact decisions, you must treat it like automated decision-making for governance and transparency purposes. You need internal documentation, DPIAs, and user-facing explanations that match what is actually happening in practice.

3. The Vendor Tool Blind Spot
A platform relying on third-party safety tools assumes the vendor has already “handled” the privacy and fairness aspects. After all, the vendor advertises compliance, has slick dashboards, and offers impressive model performance.

But when regulators or auditors ask questions, it quickly becomes clear that the platform itself:

• Cannot fully explain how the vendor’s tool uses data.
• Has not assessed whether the vendor’s profiling aligns with its own legal obligations and risk appetite.
• Has not clearly allocated responsibilities in the contract (for example, around DPIAs, user rights requests, or data retention).

The reality is that outsourcing a profiling tool does not outsource accountability. You still need to understand and govern the tool’s impact on your users and your compliance posture. That may mean asking hard questions of vendors, negotiating stronger contract terms, or even rejecting tools that cannot meet your standards.

4. The Honest “Work in Progress” Approach
Some of the most successful organizations in this space aren’t the ones with perfect systems; they are the ones that are honest about limitations. They:

• Explain that safety systems use a mix of human review and automated profiling.
• Acknowledge that false positives and false negatives will happen, and invite user feedback.
• Regularly publish transparency reports or blog posts explaining how they are improving their tools over time.

This approach aligns closely with the spirit of the ICO’s guidance. Instead of pretending that profiling tools are infallible, it embraces ongoing improvement, transparency, and user participation.
It also makes internal discussions more realistic: product, legal, and safety teams can talk openly about trade-offs instead of chasing an impossible “zero risk” ideal.

Across all of these scenarios, the common thread is simple: profiling tools are powerful, but they are not magic. They require clear purposes, solid governance, human oversight, and honest communication with users.
The ICO’s draft guidance is a chance for organizations to sanity-check their current approach and help shape a regulatory framework that supports both safety and rights.

Conclusion: Turning Guidance into a Roadmap

The ICO’s move to seek feedback on its profiling tools guidance is not just another item on your legal team’s to-do list. It’s a signal that regulators expect organizations to treat profiling and automated decision-making as core governance issuesnot side projects owned by a single engineer or trust and safety lead.

For organizations, the smart move is to treat this consultation as a roadmap exercise:

• Use it to map your existing profiling tools and identify gaps in transparency, DPIAs, or governance.
• Involve technical and non-technical stakeholders in shaping your response.
• Build processes that will stand up not just to the ICO’s scrutiny, but to user expectations and future global regulation.

If you rely on profiling tools to keep users safeand most modern services dothis is a golden opportunity to help define what “good” looks like in the years ahead.
Do that well, and you’ll not only reduce regulatory risk, but also build stronger trust with the very people your systems are meant to protect.