Eli, the technology you describe in this Introduction is was used to be called single-instance storage (it's not the current definition). It's what Microsoft Exchange and other products use when storing messages and attached files. Only a single instance of a message or file is saved with pointers to each user in the distribution list. Modern data deduplication is performed at the block level so if, say, a single bit of a store file is change, only a single block with the new bit is saved.
Microsoft (and others) implements Data Deduplication at a much lower level - not the file level. Each 4K chunk of data (regardless of what file it belongs to) is compared to other 4K chunks and if a duplicate is found, a pointer is created and a counter is increased for that chunk. This video describes what Microsoft calls "Single Instance Storage" - a less efficient form of deduplication.
Data deduplication on SAN and NAS is performed automatically or on a scheduled basis (weekly, monthly) to recover space already used. There is a modern form of storage called single instance storage (the new definition), which performs in-line deduplication in real time, never saving blocks in a datastore which already exist in the store. This is perfect for things like virtual desktops, where individual VMs are 95% identical to all other VMs. Single instance storage is currently very expensive.
Nice video! Thanks for the explanation! Does all vendors like NetApp, Dell EMC achieve the same performance in terms of data duplication, or could varies based on how they use this technology?
Eli, that was super interesting, thank you. One question though: As far as you said, data deduplication works on file level (you talk about folders etc). Would this also be of use if for instance i store a lot of backups from different users / machines eg in disk images? Like - if i have 10 or 20 backups from different windows 7 machines would it recoginze the "similarities" within the disk images or does it strictly work on a file system level?
After many years of using data deduplication, I can honestly say that I have never lost more data to any other computing technology. I've lost entire RAID 5 arrays (RAID 5 is a scam) but I haven't lost any more data there than I have using data deduplication. It sounds good on paper but in practice, it always fails. Wendell from Level 1 Techs says this: "Friends don't let friends do data deduplication."
Does the DL160 have an iLO built in? If so and it's licensed, you can use virtual media to install Server 2012. Put the CD in your PC's DVD-ROM drive and mount the drive virtually through iLO. Reboot the DL160 and if it's set to boot from DVD, it should boot from the DVD in your PC.
If they have a pointer to an identical file, just their individual pointer goes away. If they have some identical file that other people have a pointer to, the original will be moved to another user, and the pointers will be redefined.
i brought a hp prolaint DL160 Gen 8 it doesn't come with cd I brought windows server 2012 cd how can i transfer it to a usb Do a video how to do it Then do video how to install it on the hp prolaint DL160 Gen 8
Eli, your drawing looks like someone jump and split his legs & arms, am not saying that is not good I am normal user who learn alof of things from you, personally it's just the most and easiest explanation this way. BUT if you have add icon's like 4 computers and use the mouse as link then we'll know exactly and more specifically what you talking about to be clean and clear ,,, circles are not like squares. Anyway it's just a suggestion & still the best of the best, thanks alot man
I have never seen such a clear and plain explanation. Simply put it was supercalifragilisticexpialidocious elaboration.
You're the best, monsieur! I'm not the youngest, but new to this. There's so much to learn, and you make it simpler. Thanks!
Eli, the technology you describe in this Introduction is was used to be called single-instance storage (it's not the current definition). It's what Microsoft Exchange and other products use when storing messages and attached files. Only a single instance of a message or file is saved with pointers to each user in the distribution list.
Modern data deduplication is performed at the block level so if, say, a single bit of a store file is change, only a single block with the new bit is saved.
Microsoft (and others) implements Data Deduplication at a much lower level - not the file level. Each 4K chunk of data (regardless of what file it belongs to) is compared to other 4K chunks and if a duplicate is found, a pointer is created and a counter is increased for that chunk. This video describes what Microsoft calls "Single Instance Storage" - a less efficient form of deduplication.
Other companies also have their own name for such a technology.
Data deduplication on SAN and NAS is performed automatically or on a scheduled basis (weekly, monthly) to recover space already used. There is a modern form of storage called single instance storage (the new definition), which performs in-line deduplication in real time, never saving blocks in a datastore which already exist in the store. This is perfect for things like virtual desktops, where individual VMs are 95% identical to all other VMs. Single instance storage is currently very expensive.
Gotta LOVE Eli!!! Now this guy knows his stuff!!!
Freaking excellent explanation as always Eli🎉🎉🔥🔥👏🏿👏🏿👏🏿👏🏿
Nice video! Thanks for the explanation!
Does all vendors like NetApp, Dell EMC achieve the same performance in terms of data duplication, or could varies based on how they use this technology?
Eli, that was super interesting, thank you. One question though: As far as you said, data deduplication works on file level (you talk about folders etc). Would this also be of use if for instance i store a lot of backups from different users / machines eg in disk images? Like - if i have 10 or 20 backups from different windows 7 machines would it recoginze the "similarities" within the disk images or does it strictly work on a file system level?
Eli, you simply rock man !! Great stuff
After many years of using data deduplication, I can honestly say that I have never lost more data to any other computing technology. I've lost entire RAID 5 arrays (RAID 5 is a scam) but I haven't lost any more data there than I have using data deduplication. It sounds good on paper but in practice, it always fails.
Wendell from Level 1 Techs says this: "Friends don't let friends do data deduplication."
Does the DL160 have an iLO built in? If so and it's licensed, you can use virtual media to install Server 2012. Put the CD in your PC's DVD-ROM drive and mount the drive virtually through iLO. Reboot the DL160 and if it's set to boot from DVD, it should boot from the DVD in your PC.
great job nice and easy explanation
Perfect and thorough explanation.. lots of thanks from Calgary! Sorry for the smoke on your visit to my backyard Eli lol
not only exact, variations also possible
I like how says exact :) 3:23
is this concept same as single instance storage? so far it sounds like a exact same thing. Please can someone explain?
Could you talk about Global deduplication across the entire server environment? Thank you...
Its called SketchBook Express from Autodesk, its an app for Windows 8.
What are you using for your whiteboard (hardware/software)?
So what if any user deletes the data from the file server ?...
If they have a pointer to an identical file, just their individual pointer goes away. If they have some identical file that other people have a pointer to, the original will be moved to another user, and the pointers will be redefined.
Thanks Eli
i'd say dedust man. also why are your books upside down? great introduction though
great Eli!
Sketchbook!!! Thank you so much. :-)
Lot of duplication in Explaining the De-duplication concept. Just for Fun.
easy busy :) thanks mate.
i brought a hp prolaint DL160 Gen 8 it doesn't come with cd
I brought windows server 2012 cd how can i transfer it to a usb
Do a video how to do it
Then do video how to install it on the hp prolaint DL160 Gen 8
incorrect.... dedup does not work at file level.. it works at chunk level.
+chander prakash Jaiswal No, there is block-level & file level deduplication.
Eli, your drawing looks like someone jump and split his legs & arms, am not saying that is not good I am normal user who learn alof of things from you, personally it's just the most and easiest explanation this way.
BUT if you have add icon's like 4 computers and use the mouse as link then we'll know exactly and more specifically what you talking about to be clean and clear ,,, circles are not like squares.
Anyway it's just a suggestion & still the best of the best, thanks alot man
I need this for my 200GB's of music..... cuz i know theres so many copies!!
1080p for me
u look like jason statham
why daaata I thought you were American