self-host the INTERNET! (before it's too late)
Вставка
- Опубліковано 15 тра 2024
- Ugreen NASync Private Cloud Storage Solution： bit.ly/3IRQY3T
40% OFF on (Mar 26th 8: 00 am PDT-28th 8：00 am PDT)-48Hours
40% OFF on (Mar 26th 11:00 am EDT-28th 11:00 am EDT)-48Hours
Up to 35% OFF during 3/28-5/9
People who pays a $5 deposit on the pre-sale will receive 40% off from 3/26-5/9
Using a few tools, we can self-host much of the important data the internet has to offer.
RESOURCES
Docker Guide: • WATCH THIS to get star...
Kiwix Server: github.com/kiwix/kiwix-tools/...
ZIM Files: library.kiwix.org/#lang=eng
OpenStreetMaps Tile Server: github.com/fsievers/openstree...
OpenStreetMaps Data: download.geofabrik.de/
Docker Volume Tip: forums.docker.com/t/how-to-ma...
👏SUPPORT TECHHUT
UA-cam MEMBER: bit.ly/members-techhut
TECHHUT.TV: bit.ly/techhut-join
DONATE PAYPAL: bit.ly/paypal-techhut
$100 LINODE CREDIT: bit.ly/linode-techhut
$5 PBCWAY CREDIT: bit.ly/techhut-join
🏆FOLOW TECHHUT
JOIN DISCORD: bit.ly/discord-techhut
TWITTER: bit.ly/twitter-techhut
MASTODON: bit.ly/mastodon-techhut
INSTAGRAM: bit.ly/personal-insta
📷MY GEAR (PAID LINKS)
ASUS ROG M16: amzn.to/3t8Xgpo
DeepCool MATREXX 40: amzn.to/3q3K8Qn
AMD 3700x: amzn.to/31AKX9N
ASRock B550M: amzn.to/3qeymTv
G.Skill Trident Z Neo: amzn.to/3JRbeSF
Radeon RX 580: amzn.to/3n7Ax9g
Cannon M50: amzn.to/3xLfhuA
00:00 - WHY
01:15 - UGREEN NAS (Sponsor)
03:00 - prerequisites
03:58 - Network Share
05:02 - Kiwix
06:50 - Kiwis Server Setup
09:28 - OpenStreetMaps Server Setup
14:53 - ty - Наука та технологія
Just think, a AAA modern game is over 100 gigs and yet the entire Wikipedia is only 109GB!!!
thats because wikipedia is mostly text, and games are mostly binaries, textures, audio, amongst others.
That's because: Textures are pretty damn high-res nowadays, and there are several layers instead of just having a single texture (like back in 2002-ish).
You've got diffuse textures, specular textures, normal maps (bump mapping), emissive textures, ambient occlusion maps, detail textures, reflection and refraction maps, subsurface scattering maps, metalness maps. So take the size of a single texture and 10x it to cover all the different types of extra textures needed for special effects.
THAT is why games are 100+GB. Binaries (.dll and .exe files), audio, map data and others fade into naught compared to textures.
I mean famous Titanfall 's 35GB of uncompressed audio are thing.
What else you want? Maybe remove gigabytes of unused resources and locations? Or beta test gameplay?
Workers of multibillion corporation have much more important things to do, like triple check working on in game shop or implement new gatcha.
Trash takes up less space than I thought.
But 100gb are "maxi" with media, text only are far less
Next project fitting the entire internet on a 1.5 TB sandisk mini SD card.
All human knowledge in my thumbnail
We would need hella compression for that since the internet is many zettabytes big
Only the most essential of the internet, like Wikipedia,dictionary, and world time software stuff like that , compress it so it fits.
sure. backup entire WWW onto a smart pregnancy test. what can possibly go wrong.
@teknastyk Don't be a dummy. I mean the essentials.
MOM?! WHERE DID YOU PUT MY INTERNET?!
Your sister took it
"I uh, put it away"
You don't need to get Internet, we have Internet at home
(internet at home = 56k dial up, or even better, an old timey tv arial on the roof)@@Honeybearsphone
I printed it, and then sold the gear
It's getting more important for people to collectively start saving as much information as possible, in bulk, than ever before. I very recently just had a look at my UA-cam playlists and rediscovered one I made during Covid times called "important historical context" which would have been filled with videos of certain political developments. ALL of the videos in that playlist were removed from UA-cam. All of them. The scary thing is, while we now have the "easiest" way to access information, the ministry of truth also now has the easiest way to curate and re-write history.
Very smart
The winners rewrite history.
I love that reference
It’s insane that I can not find a fraction of the documentaries that I used to watch on controversial topics anymore. I wish more people were aware that this type of thing is happening.
@@jayman94fly the losers can also rewrite history to pretend like they won does that make them the winners?
You forgot the most important piece of the internet!
Stack Overflow.
real
Just localize gpt 4 turbo bro.
@@mahpell7173 you can do that?
@mahpell7173 Your pfp in junction with your comment makes it funnier than it would otherwise be because it looks like you're snickering at your own joke.
Oh, nm. My mind went a complete different direction.
Literally what the guy in half life Alyx did
"You downloaded the Internet?!"
"Well, most of it anyway"
Still gotta play this game, hopefully i'll get a better gpu soon
@@antonshotpoteita Alyx is a good optimised game actually, I've ran it on my 1050 ti, it was stable 30 fps, but there may be places where it will drop
@@stehouse I played Alex on 1650 Super and in some moments the FPS was really good, but in some moments, due to the low FPS, I started to feel sick, although I had never felt sick in VR before.
@@antonshotpoteita Absolutely worth it.
This video, like many others, created an overwhelming feeling along the lines of, "oh my gosh of course, he's right. I need to save at LEAST the survival information, then everything else in rank of importance". Something like that. It urges me to start prepping again. Which honestly is probably something like trying to create a safety to cope with the anxiety. But you know what, friends, I realized that even if SHTF as big and as hard as some of us prep for, I'm okay with probably dying. I mean I would give it my all, there certainly would be a lot of stress and pain. And in that scenario, most preppers would say going into it unprepared will likely spell demise. And they're probably right. But that's okay.
I recently was blessed to be able to somehow manage to put money down and start paying mortgage on a house (escape the rent trap am I right?) and over several months of living @ the new crib I realized it really is a house of cards. Between the chaos in government, ambitions of larry fink and the 1%er's, natural disasters, changes in economy, being stretched thin financially, almost getting laid off at work, fire, flood, powerful HOA's (mine is pretty chill TG), family stresses etc etc, you quickly realize that literally anything and everything can take you down and lose you everything you've ever worked for.
Life can wipe you out in more ways than you can imagine. We can certainly insure ourselves against common things and we do, but we can never prep for it all. Even if we had the most hunk of a bunker we could walk outside tomorrow, and tree branch falls on us and we get the speed nap. Make your home a home, that is, make it comfortable. But damn we've got to live our lives and not create suffering for ourselves by worrying about whether or not we have the head knowledge to find and make kindling in apocalyptic urban setting #2.
~an exaggeration that goes way beyond what this video even touches just to make a point written by a guy that will probably die someday.
sorry to be so serious but hoping this posture helps someone like it helps me.
Well said..
you know what, i appreciate the honesty
The first thing is make sure you have electricity to power your self-hosted internet if world, national or local disaster occurs.
I feel the same, except I have kids, so I have to keep them alive.
bro its such a mental hurdle to compile everything needed and back it up twice. but having a llm suite to answer questions post event would be too valuable.
dude, this is such a good idea!! you should expand on this.
come up with a whole doomsday essentials pack - not just wiki, but also books, etc. key resources to survive and thrive
Would love to get my hands on a printed Encylcopaedia Britannica from say 1970 and a bunch of yearly updates.
The Next Step on Prepperism.
It already exists, it is called a Cyberdeck.
@@Storin_of_Kel But cyberdecks are mostly done for the challenge and the cyberpunk aesthetic. This has a more serious orientation to it.
I kinda suspect that in a doomsday scenario, being able to reliably power on any of those devices would be quite a difficult problem. If it's a man made disaster, tech infrastructure and power delivery would be an immediate target. If it's a natural disaster, I could sorta see stuff working? Idk, solar power and a whole lot of low energy devices can help.
Ah, the IT crowd moment :) Jane, this is the internet!
i immediately thought of that, lol.
The ELDERS of the INTERNET? The ELDERS of the INTERNET KNOW MY NAME?!
We are bunking off.....!!!!! 😂😂😂
Well…. If it’s ok with the Hawk
Jen not Jane.
what about out of copyright books? project gutenberg type stuff? Store an entire library of stuff to read, that would be awesome
I've been doing a lot of archiving old books over the past six months- mainly on first and second hand accounts of the Civil War. Interesting stuff.
Internet Archive and Project Gutenberg both have a ton, also PDFdrive is your friend, but you can download a lot of these things and store them on a solid state drive (or a few of them if you’re a hoarder) quite easily.
@@Hypnotically_Caucasianactual hero
Or in copyright books, LibGen comes to mind :P
libgen? Annas archive? Hey @Hypnotically_Caucasian, just check there.
Hmm.. only 109GB for Wikipedia? I'll add that to todo list for my server. That would have been helpful back in early 2000s when I had slow Internet and doing school work.. lol 😅
But if you started downloading 109 GB back in 2000, you'd be finished just about.... now :P
That's just the text of the English Wikipedia, with images and the other languages it's like 1TB
@@ANT-jm4qxjust the text of English Wikipedia compressed is like 35gb
My man would have graduated before the download finished
@wrOngplan3t you wouldnt be finished, since youd be stuck with 2000's outdated info 😂
Me: "Mom, can we have Internet?"
Mom: "We have Internet at home"
Internet at home:
Coming home after a long day at school to relax with some wikipedia articles.
internet archive is another cool place to download stuff
Unfortunately being sued into oblivion by book publishers, so use it while you can
Very true
Local mapping data is such a godsend. Years ago we were crossing from argentina to chile through a small crossing in northern san juan, middle of the mountains aka no internet and no phobe signal, OSM was super useful for navigation :D
It's a nice to know that wikipedia is small enough for anybody to make a backup of.
You'd better shield that NAS from magnetic pulse/solar flare events or it will be wiped/fried. You MUST GROUND YOUR SHIELD (screen or copper metal fabric) !!!
Tomorrow I will download the internet, and then cut the cable! Take that, ISP!
unfortunately the internet constantly updates...
also im in australia so it'll take about 8 hours to download wikipedia as it is
I think I want to integrate one of those ChatGPT OpenAI AIs and run the AI offline with this offline data set.
@@zonk1477 the AI would need crazy amounts of RAM and data to be usefull.
@@Dave-cx3dr llama2 lore
"Internet's most important resources: Arch Linux Documentation." What am I going to do with Arch Linux if the world's internet disappears lol. I think at that point there will be more important things to worry about. "Okay, all the cities are in shambles.. i'm starving.. I'm cold.. I've got it, let me grab my localized Arch Linux docs. At least I won't have a peasent operating system like Ubuntu."
You're not going to do anything with Arch Linux. That's not the point.
The point of the arch docs is that they are the most concise and consistently useful set of Linux documentation yet made.
*Critically*, the vast majority of the information in those docs is distro-agnostic.
I don't use Arch Linux on any of my devices. Everything in my house runs something that's Debian-based. Even with that, having access to a good-quality set of mostly system-agnostic documentation is absolutely invaluable, so I've always got the arch docs bookmarked, and it's one of the first things I'd save if I were building one of these.
I of course would also save the Debian documentation, and perhaps dump the AskUbuntu forums, in addition to Stackoverflow and Serverfault.
That being said though, what the Arch Wiki has that those don't, is brevity.
*That* is why I still use the Arch Wiki Constantly even though I haven't used Arch in years.
>decide to back up resources to be prepared when the world ends
>don't download anything technical because "i'll have more important things to worry about than my computer"
>computer breaks
>can't fix it because i didn't download the relevant resources
mfw
@@nibbletrinnal2289> be too stingy to download anything because “there are bigger issues”
> have no technology that you can use
> be in the stone age
> mfw
Kinda my thoughts too. Especially frustrating if needed to download additional packages in order to solve a problem
in a worst case scenario; internet will still be operational. all of the core infrastructure is built deep underground. we also use satellites. to say that internet wouldn't exist anymore is a myth. go play "outriders" . humanity took "all of internet" with them; when they left earth.
perfect application for my 6W TDP N100 Proxmox sever running on solar power
This reminds me of that IT Crowd episode where they convince people the internet is in a black box.
Because it will be
I have a Boox Palma and bought a 1 Tb micro SD card. I’ve got almost all of what Kiwix offers, my entire state maps downloaded, and numerous videos, documents, books, libraries, etc.
All of this in an e-reader the size of a smartphone with a battery that can last weeks, if used sparingly. It’s a pretty cool setup.
Sweet
Love it, awsome knowledge and content. Thank you for the share.
This is the best! So glad I subscribed to this channel!
Not sure if I can afford my little bits of the internet.
Being a Venezuelan, I totally understand you and backup your actions. Thanks you very much for sharing your expertise with those of us who may need it in the future. (God forbid it though, because that would mean the SHTF).
What a neat concept! I'd heard of Kiwix before but didn't realize what you could do with it. Cool that you can load mapping data too.
Hi , sorry for the of topic. I would advise to turn down the fill ligh by about 2 stops. So there will be more volume. Used to work as second DP , or as a gaffer in cinema production for almost a decade .
Best wishes
I'll give it a try. 😎
This is the best tech channel ive seen and im still subscribed
For some reason I love watching videos about setting up servers and networks.
Oh so very cool, this would be a very good VM for me to try thanks so much for a great video!!
Dude, I'm so proud of you. 198,000 subscribers? Crazy. Keep killing it dude. from Warpiggies, to techhut.
Thanks again.
My idea of a nas is plugging an external drive into the CASA server
This is actually a great idea. Similar to having a library of information in your private collection.
This has been something I've been planning to do for a long time but wasn't quite sure how. I really wanted to build a sort of STC (Standard Template Construct) server for myself if basically everything went to shit. Thanks very much! I'll be setting this up ASAP! :)
Better put it in a faraday cage and keep it off direct contact with the net.
I was going to back green, but I'm concerned that the software will either have my data hit their server or they will somehow have access to the data. I wanted it for work but can't risk client data confidentiality.
Born and raised in Sunnyside, WA! Spent my life bouncing around the West but mostly WA and AZ. Thank goodness, projects like this mean that society will be much easier to restart if, God forbid, the worst case scenario does occur.
I love where you’re going with this… for some of us.. Web 3.0 being decentralized… if you think about it… initially we all were decentralized before off the cloud. ☺️ I see that becoming a thing again. Loved the video.
Wow, great h/w specs!
Need this! Especially I have a raspberry pi tower sever along Raspberry Pi 3 client.
Every company should store there sites everyday on a hard drive and then have a second feed source like satellite land line and signal so it stays active no matter what happens
Thanks now my bunker is complete.
I already do this with a series of XHDD's. Toshiba and Seagate sell reliable hardware at a [mostly] decent price. Every time they knock the price down to 100 USD i grab one of the 4TB drives and fill it up. Between work software (IDE's mostly), movies, shows, music & games, archived websites and internet videos; ive got enough entertainment to out last the apocalypse!
7:50 lmao the notification got me
This is unironically the best present for Mother’s Day. Hell yeah
They need to make a Vault (or even multiple) for the internet like the Svalbard Global Seed Vault.
The main purpose would be for re-uploading in the event of unforeseen events causing the internet to go offline, such as major Solar Storms if they are ever to occur!
As for viability:
Estimates suggest that it contains hundreds of exabytes (1 exabyte = 1 billion gigabytes) of data. This includes everything from web pages, images, videos, documents, emails, social media posts, and more.
Our largest yet known to the public is Microsoft’s Azure Data Center, which has 15 exabytes of storage.
Microsoft also came up with extremely resilient storage disks that can withstand major damage of just about any kind, called Project Silica.
These can already store 7 TB of data per 75mm * 75 mm * 2 mm
Right now it would require about a billion of these disks to store it all
But allegedly Microsoft is planning to apply this tech to their 15 exabyte Azure Data Center, it might be feasible after all!
I've been downloading my vidoes. and PDF's in my collection but this is next level!
Did you install Proxmox on the UGreen NAS?
Wouldnt be better to use NFS for the storage?
The investigative reporter - Whitney Webb - highly recommended people doing this type of thing.She predicted (based on evidence she found) that there is a plan in place to take down the internet (and later do a "3.0 reboot of it ) to aid in forcing everyone on CBDCs and massive censorship / blocking access to certain information. Although, I think the option featured here is more sophisticated than what she had in mind for grabbing stuff off the internet.
The absolute muscle memory of clicking the word trains.
I have self hosted many things, but not yet the Internet
Time to archive the internet archive
I think that a GPT-4 trained model (a recent version of it got leaked, so it's widely available if you search) would be an excellent addition!
Besides being trained into the larger context of today's Internet, it's "knowledge" is stored into the trained model, so theoretically you wouldn't need Wikipedia (although I REALLY recommend it).
Great youtuber on the Internet 😊
is it necessary to go via VM & docker with Proxmox when there is also option for like lxc?
Curiously, today at work I needed to do something like this and I think the best option is wget with --mirror --convert-links and other flags
can you please explain why ?
Hey! I have a question. I have Jetson Nano running Xubuntu, that is running Portainer, a Docker Manager, could I have my Jetson Mount my TrueNAS Server for storage and run my own internet and hold the data on my truenas server, but have the Jetson Nano access it?
Out of curiosity. noticed you are using Edge on Linux ... is the hardware decoding working for you on the browser, vp9, h.264 and av1 or is it using CPU when watching videos, like on UA-cam?
Do you have any recommendations for more time tested equipment? I’m not doing all that and putting it on a kickstarter product ESPECIALLY when they have that weird software lol.
Better idea. Wifi or Bluetooth compressed file data sharing that connects automatically to multiple people. Maybe 2 other devices or more. Theres a similar idea in bluetooth walkie talkies, but no one has been able to get it to work well
Good video!
Honestly if space travel ever becomes more mainstream each ship is going to need its own ‘internet’ that can be updated when a connection is available.
Perfect for meshnet
Totally doing this
Always thought about doing this
Hi! What about websites that are not on kwix? I'd love to host offline Word Reference as I use it for work and sometimes the internet is off
plenty of times ive needed offline mapping because I was out of range of cellular or in a hole that wasn't covered by cell service.
you look exactly what you sound like, and you look exactly like someone who would do this
i'm genuinely impressed
good video
I'm both offended and honored. Thank you 🙏
Reminds me of the Brit comedy TV series 'The IT Crowd.' episode when they convince one staff the whole internet is in a little black box.
so, it's limited to Kiwix right now? any other source platforms like that? or self-hosted scraping+compression maybe?
How do you keep the kiwix data updated?
Super YT channel 👍💪
Yet another UGreen NAS promo? Yikes!
Kindest regards, neighbours and friends.
what is that beatyfull linux distro YOU are using in this vid? Also nicest vid about this topic that ive seen so far. :D
Jellyfin is another great option to put into this
what about having endless os as a liveboot flashdrive?
Save everything you find important to your life. (Pictures, music, movies, research, books, etc) aside from what you like. You don’t have a prayer of storing the entire internet. Just focus on what you can’t live the rest of your life without. I personally have a few drives containing everything I love about the digital age. If the apocalypse comes, I’ll be set in terms of entertainment 😂
Wow! So cool!
i wish i had a sponsor to think like u
Currently downloading youtube.
What's the usefulness of hosting some Linux distribution wiki without the repositories?
Honestly I would do this for peace of mind. Just knowing when it’s offline the knowledge it can no longer be messed with negatively. Technically the amount of knowledge that you get from this would be next to impossible for an individual to get through it all. You’re only going to fact check and update interests of information.
This should actually be the way PCs evolve into. The OS automatically comes with the most important information and education from the internet. The rest could be added if you choose to. Just an extra drive already setup on machines with access. It would also make sense energy wise to. Less power to run access over and over again. Each time you access that website a ton of infrastructure needs to turn on just to get it to you. The bonus of this too is that companies can offer faster speeds for cheaper because they are not being bogged down by consumers. Of course that would be a balancing act. Now I just need to figure out communications like phone and texting so I don’t need to rely on companies for it. This internet download solution is a huge help to my goals.
The only way to do your own comms right now is LORA, so I hope all your friends live close by. Well, there's also HAM.
Then again, having friends close by is the best thing in a disaster.
@@joemerino3243 Thanks for the info. I’m going through a challenge where trying to figure out if it’s possible to make everything you would pay a bill for could be substituted for something self sufficient. So the most challenging one is comms. Everything else can be done as long as sufficient power is available through things like solar and wind. Water is provided by Rain and Dehumidification. Food through having a garden and taking care of animals. I mean you could do all these without power but it’d take ten times the amount of labour and time.
@@daniellapain1576 I think as you research you'll find solar and wind to be extremely insufficient for your purposes.
They both are very uneven in output and thus require batteries. Power storage wears out much faster than people think. Especially if you're trying to get your drinking water through dehumidification.
What you'll want is to be by a stream and dam it, then build a small hydroelectric setup. These can be built by hand from old washing machines or similar electric motors. This will give you more stable power that doesn't fluctuate so much. Your energy would still be quite restricted compared to metered power, though.
@@joemerino3243 It’s awesome that you just replied. I was on another thread and me and a few others are discussing a heating alternative using friction. Using a simple Dewalt battery drill a smooth piece of bar on concrete. I was able to get 200 degrees Fahrenheit in a short amount of time. There is something to it but it will take a lot of trials with different configurations. My input to this idea is to trial this concept inside a Sand Battery. Originally the fellow working on this idea was working out how to heat water with a wind turbine.
I think we’re close to the point where we could ask an AI to use Wikipedia to make an update to the last 2010 print version of the encyclopedia with updated information. One that would be printed out. You could even get it to make a 1 book summary or 5 book or what ever. Depending on how much you actually want to print or read. But training an AI on the old encyclopedia Britannica might require uploading multiple years so the AI can learn what type of information gets added or removed over the years. So the goal being that the AI would know to add important scientific advances or major political events. With the Wikipedia data the AI could cross reference those changes with the much larger Wikipedia data set. Obviously it would be copyright infringement unless Brittanica signed off on it or participated. I think the company still exist but is much smaller and just does online: But that would be a cool project.
A simpler solution might be to just use Wikipedia to make an update book or books for any version of encyclopedia you own. So if you have the 2010 you would have a book that has the updated events since that edition came out. If you own a 1980 set you might need 3 update books for example.
Already had this in mind for over 10 years but it's expensive. All movies TV shows and games id want during the apocalypse.
Idea: Have a collection of the offline installers for all the most important softwares aswell as iso files for various operating systems
A version of this with an LLM and RAG would be nice.
what linux distro do you use?
"Do this if you dont trust your government." Well looks like WE ALL have a new weekend project lol!
Immediately remember IT Crowd and showing Jen “the internet”
What's your go-to for downloading satellite maps?
Hey did you find any good sat maps?
That's actually a great idea!
I won't have to ping mirrors so heavily when I'm bootstrapping and stuff.
I'm not a doomer about the state actor grid-down scenario anymore though. If they didn't believe shutting it down was not survivable, they would've done it by now.
Next video get a list of github project to backup along side, for linux enthusiasts, hackers and anything useful that is on github that has to be there on an old hdd. I will be using 1 Tb old sshd for this, no nas or anything facy just putting something to work. Any tips and tricks to do with that are welcome follow :) #theendisnear
It would be nice if they made a rugged one and water proof case
it would be very nice to the front end users if there was a an app of an offline browser that lists all the databases currently downloaded on your computer. I can sorta handle the set up process but constantly remembering what to type in to access the database makes head hurt
Also add an offline ai model in the mix. It's surprising how much data it can provide with how small they are and it truly feels like still having access to the internet.
Is it simple to keep the zims updated?
UA-cam redirected me from this video to some Google page about consumer something or other...
It would be amazing if you could use an LLM for searching this, sorta like a local Copilot.
you might want to look into setting up local research agents, were at a point now where this as become quite effective, lama3 already effectively removes the need for the Wikipedia part but it would still be useful to setup local access to more abstract resources like his linux documentation
Can it host large language models and Llama files?
Next you're going to have to get an EMP shielded case 😂
You look exactly like Kaleb Cooper for clarksons farm
This , Jen , is the Internet.