Everyone feels organized for about… a month.
Then the sprawl starts.
A folder called “Final” shows up. Then “Final FINAL”. Then someone shares a Google Drive link in Slack, someone else uploads a copy to OneDrive, and a third person emails the same file as an attachment because they cannot find either of those links. Six months later, nobody knows where the real version lives.
And that’s before we even talk about the stuff you do not know you have.
That stuff has a name. Dark data.
Not spooky, just… invisible.
It’s the files sitting in your cloud that are not indexed properly, not tagged, not owned clearly, not referenced in any workflows. Old exports, duplicates, orphaned project folders, meeting recordings nobody watched, PDFs in personal drives that should have been in a shared space. Data you pay to store, back up, secure, and sometimes retain for legal reasons. But you cannot easily find it. Or explain why it exists.
This article is about fixing that. In a practical way. Not a “buy an expensive platform and hope” way.
Let’s get into it.
What dark data actually is (and why it keeps multiplying)
Dark data is basically any file in your cloud that has low visibility and low usefulness. Not because it’s worthless, but because it’s not connected to anything. No clean metadata. No consistent location. No clear owner. No retention rule. No one searching for it because they do not know it exists.
Some common examples:
- A contractor uploads deliverables into their personal Google Drive, shares it with one person, then leaves. The files live on forever.
- An employee syncs their desktop to OneDrive, which includes Downloads, which includes every PDF they ever opened.
- Teams move from Dropbox to Google Drive to SharePoint, but nobody cleans up after the migration. So now you have three copies, and two are missing permissions, and one is locked.
- Old Slack exports, Zoom recordings, call transcripts, CSV dumps from tools you no longer use.
- “Temporary” folders that became permanent because deleting feels risky.
The cloud makes storing cheap and easy, so nobody feels pain immediately. Until search stops working, audits happen, security gets nervous, or storage costs quietly creep up.
Dark data multiplies because the incentives are backwards.
It’s always easier to create a new file than to find the existing one.
Why this is not just “clutter” (real risks, real money)
If this was only a cleanliness problem, fine. Annoying, but not urgent.
But dark data has three real consequences.
1. Security exposure you cannot see
You cannot protect what you cannot locate.
A single forgotten folder with public link sharing enabled. One spreadsheet with customer data sitting in a personal drive. One export from a CRM. That’s how breaches happen in normal companies, not just the dramatic hacker movie ones.
The worst part is not even that it exists. It’s that you have no inventory.
2. Compliance and legal risk
Retention rules are hard when you do not know what you’re retaining.
If your organization has to comply with GDPR, HIPAA, SOC 2, ISO 27001, or just standard contractual obligations, you need to be able to answer questions like:
- Where is customer data stored?
- Who has access to it?
- How long do we keep it?
- Can we delete it when requested?
Dark data makes those questions painful. Sometimes impossible.
3. Productivity tax (the slow bleed)
This is the sneaky one.
People waste time searching, recreating, and re-sharing files. They ask in chat. They wait. They make new versions. They lose context.
Even if you never get audited and never get breached, you still pay for the chaos daily.
The goal: build a “file inventory” before you try to clean anything
Here’s the mistake many teams make.
They start with cleanup rules. Delete anything older than X. Archive anything not accessed in Y months.
Sounds efficient, until someone yells because you archived the one spreadsheet Finance uses once per quarter. Or you deleted an old contract that Legal needed for renewal terms.
So step one is not deletion. It’s visibility.
You want an inventory. A map. A way to answer, for any file:
- Where is it?
- What is it?
- Who owns it?
- Who can access it?
- How sensitive is it?
- Is it active, stale, duplicate, or orphaned?
Once you have that, cleanup becomes a series of decisions, not a guessing game.
Step 1: Pick your scope (because “entire cloud” is how projects die)
If you try to scan every system, every drive, every shared folder, across Google Drive, SharePoint, OneDrive, Dropbox, Box, Slack, Jira attachments, Notion exports, and random S3 buckets…
You will not finish.
Start with one of these scopes:
- Your most sensitive department (HR, Finance, Legal)
- Your biggest storage area (often Google Drive or SharePoint)
- Your highest risk content type (spreadsheets and PDFs, usually)
- A single business unit that will cooperate
A good starting point is: shared drives and team sites first, personal drives second.
Personal drives are where the nastiest dark data lives, but they are also where politics lives. Start where you can win.
Step 2: Crawl and index your cloud content (the unsexy but necessary part)
To find hidden files, you need to list them. Not just “search”. Search only finds what you already suspect exists.
Indexing means pulling metadata for each file:
- Path / location
- File name
- Type (doc, pdf, csv, ppt, etc)
- Owner and last editor
- Created date and last modified date
- Last accessed date (if available)
- Sharing settings (internal, external, public link)
- Permissions and groups
- Size
- Hash or fingerprint (for duplicate detection, if possible)
How you do this depends on your stack:
If you’re on Microsoft 365 (SharePoint and OneDrive)
You can get a lot done with:
- Microsoft Purview (for data discovery, sensitivity labels, DLP)
- SharePoint admin reports
- Graph API for deeper inventory and automation
- Third party governance tools if you need cross-tenant or advanced reporting
If you’re on Google Workspace (Drive)
Useful options:
- Google Drive audit logs (in Admin console)
- Google Vault (more legal hold and retention, but still helpful)
- Drive API for a real inventory
- Security center investigations (depending on edition)
If you have multiple clouds
This is where teams often bring in a dedicated data discovery or DSPM tool. Not because it’s trendy, but because stitching together three APIs and normalizing permissions is a pain.
But even then, be clear about what you need: discovery, classification, and remediation workflows. Not a pretty dashboard that shows you a number and calls it a day.
Step 3: Find the dark patterns (what to look for once you have the list)
Once you have an inventory export, you start hunting for patterns. Dark data is not random. It clusters.
Here are the biggest clusters I see.
Orphaned content (no clear owner)
Files owned by:
- ex employees
- deleted accounts
- service accounts
- contractors
These are high risk because nobody is responsible for them, but they can still be shared externally.
Quick win: identify anything owned by inactive accounts and assign ownership or move it to a controlled shared space.
Over-shared content (permissions drift)
Look for:
- “Anyone with the link”
- Public links
- External sharing to personal email domains
- Files shared to huge groups that do not need them
- Nested folders where permissions get weird
This category alone can justify the entire project to leadership, because it is measurable and scary in a useful way.
Stale content that is still expensive
Big files nobody accessed in 12 to 24 months:
- old video recordings
- raw design exports
- zip archives
- VM images (if you have them in cloud storage)
- dataset dumps
Stale does not mean deletable. It means “decide”. Often the right move is archive to cheaper storage or lock it down.
Duplicates and near-duplicates
Duplicates are everywhere:
- Copy of a copy in a different folder
- Same file with different names
- Same presentation exported as PDF ten times
- “v2”, “v3”, “final2”, “final_final_really”
Detecting true duplicates is easiest with file hashes, but you can also get decent results with size plus name similarity plus timestamps.
Even if you do not delete duplicates, you can at least identify a system of record and reduce usage confusion.
Dark corners created by integrations
Some tools create attachments and never clean them up:
- CRM exports
- ticketing system attachments
- form upload folders
- automation tools that dump logs and CSVs
- “backup” folders created by migration utilities
If you find a folder that grows forever and has no owner, it’s usually an integration doing it.
Step 4: Classify what you find (lightweight beats perfect)
This is where teams get stuck. They think classification requires a massive taxonomy and a months-long committee.
You do not.
Start with a simple classification model:
- Public
- Internal
- Confidential
- Restricted (or Highly Confidential)
Then decide what triggers Restricted, for your company. Usually:
- customer PII
- employee PII
- financial account numbers
- credentials and API keys
- health data
- legal contracts and NDAs
- security architecture docs
How do you classify at scale?
- Pattern matching (regex) for obvious identifiers
- Keyword and phrase matching for common document types
- Built-in sensitivity labeling tools (Purview, Google DLP)
- ML classifiers if you have them, but do not wait for perfect accuracy
Important: classification is not only about content. It is also about context.
A harmless file in a locked shared drive is different from the same file publicly shared from a personal drive.
So combine:
- content sensitivity
- sharing exposure
- ownership clarity
- activity recency
That combo tells you what is truly risky.
Step 5: Remediate in waves, not all at once
Remediation is where humans get emotional. Because files feel personal, and deleting feels permanent.
So do it in waves with clear rules.
Wave 1: Fix obviously dangerous sharing
- Remove public links
- Restrict external sharing where it’s not allowed
- Replace broad groups with smaller ones
- Enforce link expiration if your platform supports it
This wave reduces risk fast without deleting anything.
Wave 2: Assign ownership and move orphaned content
- Transfer files from ex employees
- Move critical shared folders into managed shared drives or team sites
- Remove contractors from access lists where appropriate
Ownership is governance. Once a file has an owner, it stops being dark.
Wave 3: Archive stale content
- Choose a threshold, like 18 months no access
- Archive, do not delete, on first pass
- Put archives behind tighter permissions
- Move large archives to cheaper storage tiers if possible
You can set review windows. “Archived for 90 days, then eligible for deletion” works better than “delete today”.
Wave 4: Deduplicate with a system of record
For each duplicate cluster:
- pick the canonical file
- update links if needed
- keep one copy
- archive the rest
This is annoying work, yes. But you can focus on high impact areas, like templates and frequently used reference docs.
Step 6: Put guardrails in place so dark data does not come back next month
If you only clean once, you are basically doing spring cleaning in a house with no closets. It will get messy again.
You need a few boring guardrails.
Naming and location rules that are realistic
Do not write a 20 page policy nobody reads.
Write a 1 pager:
- Where final documents live (shared drives, team sites)
- What belongs in personal drives (drafts, personal notes)
- What cannot be stored in personal drives (contracts, HR files, customer exports)
- Simple naming guidance for projects
Make it easy to follow. If it feels like bureaucracy, people route around it.
Default sharing settings
Set safer defaults:
- Internal only by default
- External sharing requires explicit approval or justification
- Public link sharing disabled unless you truly need it
- Link expiration for external guests
Defaults do most of the work. People rarely change defaults.
Lifecycle and retention automation
Use retention labels or policies so files do not live forever by accident.
Examples:
- Sales call recordings retained for 12 months, then deleted
- Candidate resumes retained for X months depending on jurisdiction
- Finance records retained for 7 years
- Project temp folders archived after project close plus 90 days
Even partial automation helps.
Continuous monitoring
Run a monthly report that flags:
- newly public files
- new external shares
- files owned by newly offboarded users
- large files not accessed in a long time
- folders with explosive growth
This turns dark data from a yearly panic into routine maintenance.
A simple “dark data audit” checklist you can run this week
If you want something concrete, here’s a starter checklist that works even if your tooling is limited.
- Export a list of all files in shared drives / team sites with owner, size, last modified, sharing settings.
- Filter for public links and external shares. Fix the worst ones first.
- Filter for owners who are no longer employed. Transfer or move those files.
- Sort by size. Identify the top 50 largest items. Ask if they are still needed.
- Filter for files not accessed in 18 months. Pick one department and archive a batch.
- Choose a sensitivity label scheme (even just 4 levels) and label the top risk areas first.
- Set one default sharing policy change that reduces risk without breaking workflows.
- Schedule this audit monthly, even if it’s manual at first.
That is enough to start creating light.
Wrapping up (what “solved” actually looks like)
You do not solve dark data by deleting half your cloud storage in a weekend.
You solve it when:
- You can answer what you have and where it is.
- You can identify sensitive content quickly.
- You can see who has access, especially externally.
- Orphaned content has owners.
- Stale content has a lifecycle.
- New dark data gets flagged early, not discovered during a crisis.
And the vibe shift matters too. People stop treating the cloud like a junk drawer.
It becomes a system.
Not perfect. Still a little messy, because humans. But visible. Managed. Safer.
That’s the win.
FAQs (Frequently Asked Questions)
What is dark data and why does it accumulate in cloud storage?
Dark data refers to files stored in your cloud environment that have low visibility and usefulness due to lack of proper indexing, tagging, clear ownership, or connection to workflows. It accumulates because files often exist in inconsistent locations without metadata or retention rules, such as old exports, duplicates, orphaned folders, and unused meeting recordings. The ease and low cost of cloud storage encourage this sprawl, making dark data multiply over time.
Why is dark data more than just digital clutter?
Dark data poses real risks beyond mere clutter. It creates security exposures since forgotten files with improper sharing settings can lead to breaches. It complicates compliance with regulations like GDPR or HIPAA because organizations cannot easily track where sensitive data resides or who accesses it. Additionally, it causes a productivity tax by wasting employees’ time searching for files, recreating lost documents, and managing multiple versions.
What are the key consequences of unmanaged dark data?
Unmanaged dark data leads to three main consequences: 1) Security exposure due to invisible files with potentially risky permissions; 2) Compliance and legal risks as unknown data hampers adherence to retention policies and regulatory requirements; 3) Productivity loss as employees spend excessive time locating or duplicating information, resulting in operational inefficiencies.
How should organizations start addressing dark data in their cloud environments?
Organizations should begin by building a comprehensive file inventory before attempting any cleanup. This involves gaining visibility into all files to answer questions about their location, content type, ownership, access permissions, sensitivity level, and status (active, stale, duplicate). Having this map allows for informed decisions on which files to archive or delete without risking critical business information.
What is an effective approach to scoping a dark data cleanup project?
To avoid project failure from trying to tackle everything at once, start with a manageable scope such as a single department (e.g., HR or Finance), the largest storage repository (like Google Drive or SharePoint), the highest-risk file types (spreadsheets or PDFs), or one cooperative business unit. Prioritize shared drives and team sites first before moving on to personal drives where political challenges may arise.
What does crawling and indexing cloud content entail in managing dark data?
Crawling and indexing involve systematically listing all files across your cloud platforms—not just searching—to gather detailed metadata including file path, name, type, owner/editor information, creation/modification/access dates, sharing settings and permissions, size, and file fingerprints for duplicate detection. This process provides the necessary visibility into your cloud environment’s contents to identify hidden or orphaned files effectively.
