Re-enable de-duplication on a filesystem larger than 64 TB

Question

Hi there!

We have a situation where an overly large filesystem on a Server 2019 was set up for de-dupe (first at 64TB, than grown to 74TB). Somehow the local SysAdmin managed to crash/stop/remove the DeDupe Enablement while trying to remove the Deduplication from the drive.

In order to remove dedupe, we'd need to re-enable the de-dupe service so we can reclaim the lost disk space (46TB) The type of data is not suitable for dedupe, we lost more disk space to dedupe than it saved so the plan was to remove the dedupe functionality completly. Which went horribly downhill and we're stuck with a 74TB drive where 46TB are blocked in /SysVolInf/Dedup/ChunkStore

In order to clean up we need to re-enable dedupe but we're only getting the error "system too large"This might be because the volume is larger than the maximum supported volume size"

Yes, we know that, it's 74TB in total... but how can we free up the space without the GarbageCollection which needs DeDupe enabled?

Any pointers/ideas? Last resort would be ditching the online backup data (25TB) and start from scratch. But this would be the very last option.

Thanks for help!!

Best, Martin

Answer 1

Hmm, no-one?

We'd just need to "re-enable" dedupe in order to eradicate it from the filesystem... nothing else.
There are currently 33TB blocked in the SysVol/Dedup subfolder, the backup dataset is "only" 25,5 TB large...

Enabling Dedup was clearly the wrong decision here, but getting rid of it seems even more problematic due to the filesystem size now.

Answer 2

Hello,

Thank you for your question and reaching out.

I’d like to let you know that if you any additional storage avaialble then add it to Dedup drive then re-enalale and dislable dedup.

also , please try to run the tool (DDPEVAL.exe) if you can further anayalese the drive.
https://learn.microsoft.com/en-us/archive/blogs/klince/evaluate-savings-with-the-deduplication-evaluation-tool-ddpeval-exe

Please have a look similar topic :
https://social.technet.microsoft.com/Forums/ie/en-US/2fe66f71-1956-4cec-8419-fec7ebe1ae57/dedup-chunk-store-large-garbage-collection-doesnt-work?forum=winserverfiles

If the reply was helpful, please don’t forget to upvote or accept as answer.

Thanks,

Prakash

Answer 3

Sorry, I think you missed the point completely...

HOW do I (re-)enable dedupe on a filesystem LARGER than 64TB? The filesystem is 74TB right now!

Volume "d:" cannot be enabled for Data Deduplication. Data Deduplication does not support volumes larger than 64TB. Error: 0x8056530b, Deduplication could not be enabled on the specified volume. This might be because the volume uses an unsupported file system, is larger than the maximum supported volume size, is read-only, is formatted with an unsupported cluster size, or is not a fixed drive. Deduplication is supported on fixed, write-enabled ReFS, NTFS, CSVFS_ReFS, or CSVFS_NTFS volumes smaller than or equal to 64 TB, with cluster sizes less than or equal to 64 KB.
.

THIS is the problem I'm fighting... there is enough free disk space for the un-optimisation.

Answer 4

Hello,

Thanks for your reply.

Unfortunately it is not supported by Windows server as the limit is 64 TB.

Please have a look on this Microsoft article mentioning size limit.
https://learn.microsoft.com/en-us/windows-server/storage/data-deduplication/whats-new

On the other note if you have some additional space Please try to move the data to different Drive and make the Original drive to less than 64 TB so that you can re-enable Deduplication.

Thanks,

Prakash

Answer 5

I really just wanted to get RID of the whole dedupe function... shame that this seems to be blocked by the the hard-coded filesystem size limit... we'd just need the "garbage collection" in order to free up the disk space.

So I feel it will be a large data move and a filesystem rebuild w/o dedupe. And smaller than 64 TB in order to keep VSS running...

Best, Martin