@@ServeTheHomeVideo But why?! Is it cheaper than sourcing 1 Gbit/s ports? Are there any real world situations where a 10 Gbit/s management port is advantageous over a 1 Gbit/s port? Are there any firmware updates that are so big that having 10 Gbit/s just makes sense?
@@MrMartinSchou By that argument, there's also no need to actually have two management ports. This switch has some pretty beefy RAM and FLASH compared to other switches, so there's a bunch of things you can do with the management ports. Anything that has to pass data through the CPU could potentially work on it, even routing. But I think my favorite answer would be "because the CPU supports it." In contrast, I have a CRS-518-16G-2S+IN, which uses a Wi-Fi SOC for its CPU...but it doesn't have Wi-Fi. Would be fun if they added the features supported by the SoC.
Love the option to divide the 100G to 4 pcs 25G. We use it in our datacenter with very expensive Cisco LEAF switch, so I'm excited to try this one from Mikrotek. :)
Mikrotik back at it again, making me go "One more piece of hardware, then it should be good for a while... OOO SHINY" thankfully for me, little bit out of the price range I can swing for homelab and not something for any upcoming work deployments.... for now. 25/100G came down a lot quicker than I thought it would and really making me wish I'd waited a bit longer to kit out my homelab, 10G/40G is fine but the specific equipment could have waited and done better. Really looking forward to the continued trickle down
keen to see where that product line is going in the next 3-5 years. they entering the switch market on serious numbers now with ther marvell partnership something one could only have dreamed about 5-8 years back where they started to use ARM technology
What a coincidence. I was just pricing out Dell S5232F-ON switches for a new small DC install (6 Proxmox hosts. 2 Storage Servers, and 10Gbps internet).
For me it’s important to have Roce/rdma features and ptp support. If Microtik does it I’m all over it. Use it for storage or high fps camera applications
Yep this switch is cool, but I love the 48 GbE / 4 SFP+ / 2 QSFP monster. Perfect to stuff around in a home lab and learn, and amazing flexibility to do weird and wonky protocols on every port!
Holy crap. I immediately think "What kind of loads would generate this sort of traffic?" in small environments? 12.5 GB/s max... I can only imagine the kinda SAN/NAS you'd need to push that bandwidth, number one, and to multiple clients at the same time on top of that? Woo! I'm dreaming of anything faster than my old ass Synology with hard drives... I know, I'll see myself out haha Thanks for the vid! This is really awesome.
@@levygaming3133 Sure, but do you personally have a SAN with this capability? I sure don't. How many people really do? That was my point - that this is way overkill for most users - even though I could have made it a bit better.
@@ServeTheHomeVideo That's wild. Holy hell. I can't even imagine the cost of the SAN plus enough drives of the right capacity and generation to push that much data to multiple users.
Yes, I just upgraded everything to 2.5gb ethernet, although one connection goes over a MOCA 2.5gb link (works really well). Paid the extra for a wifi 7 router with 2 10gb ports and 4 2.5gb ports. Now waiting for the fiber connection that was laid to my yard so I can sign up for Google fiber. I have been waiting since December.
So it isn't something I would do now but something I would like to do. In the professional video space there is something called IPMX. It allows for extremely low latency video reproduction. I would love if companies like microsoft and parsec implemented it into their remote desktop protocols. It uses a lot more bandwidth but for that you can get sub ms latency at least as far as the encoding goes.
I'm still rocking the Mellanox MSB-7890 36-port 100 Gbps Infiniband externally managed switch that I picked up in 2018 for $2950 CAD (at the time, which works out to be something like $2300 USD, used). For a homelab AI/HPC cluster, I'm definitely using 100 Gbps.
First mikrotik only adds that H3 ofload will be expanded. L2 works as it should. 2. Winbox case if you know it is a great interface (cli is always available) 3. L3 is executed by ARM processor and this is where mikrotik works to do as much as possible through hardware support by switch system Decent performance in mikrotik only in L3 in L2 we have high performance
Hey man, can you review the QNAP QSW-M7308R-4X-US? I think that one is potentially better than this one, especially for a home lab. It costs 1k, has 4 ports at 100Gbps, and 8 at 25Gbps which is great for lots of people.
We have tested 50+ 2.5GbE switches on the main site and made an Ultimate Buyer's Guide for them. You can now get unmanaged 4-port 2.5GbE and 2-port SFP+ for $45 or less.
I wonder if this switch can be used as router as it was mentioned somewhere. BTW Mellanox SN2700 which is 32 port 100G switch has similar power consimption to this Mikrotik. I have this Mellanox SN2700 at home and it draw about 45w idle with no transceivers plugged in. Now with 11x QSFP28 power consumption is 96w. I bought few of these Mellanox SN2700 for an ISP in my area for little less than $1000 each. They gave me 1 for free as a compliment :) -Edit- Actually we weren't aware of the power consumption of this Mellanox before we buy and our expectations were higher and this was a pleasant surprise :)
For those who think this switch is expensive... Have a look at the new black magic design 2110 switch... Patrick, why don't you have a look at that switch? Would like to know what your thoughts are
0:23 yeah, THAT "feature" is the reason, as to why i am not really looking at mikrotik as my first option. (Well that is mostly due to one of the exercises one could get on my exam was inter vlan routing, and setting that without any documentation is pure pain)
I think this is pretty nice option for many small businesses, which want to have new switch with ability to upgrade servers down the road in next 1-2 years to use 100Gbps for CEPH for example. I think even i could use it for 2-3 server cluster with CEPH under proxmox (yeah, 2 servers for quorum is great idea /s), so that i would still have 10 ports left for other stuff down the road (and most likely get another one to have HA after some time). Even "cheap" Supermicro servers with U.2 slots can be bought new for like 15-18k, which easily could hit stable 12GBps or more, so price-wise i think it have everything what users would need - like You said. 2k is a bargain for me.
You can use 2 servers + Qdevice so it has quorum for Proxmox (but not Ceph), and with just two nodes you are better off with zfs and replication or even the linbit drbd block device replication than Ceph. At least imho
@@marcogenovesi8570 I`m starting to learn about CEPH, replication, HA and stuff in this area - just today got second Lenovo SFF with i5-6500T so that i can create 2 proxmox nodes and learn about replication without touching my main server (Epyc 7D12) so that if brown hit the fan i can just wipe it and start from scratch. Main goal is to learn it and use at work, so that i can create "proper" production environment or just baseline so that someone after me could continue it, to get it done "cheap" and make it fly :)
Full Mesh Network (with frr+fabricd) is a better option for a small 3 node cluster then to use a single SAN Switch. pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Routed_Setup_(with_Fallback) If you are on the budget, you can look for used 40/56GbE Mellanox ConnectX-3 NICs (like MCX354a-FCBT) and QSFP+ or QSFP28 DAC cables.
that one must have been released very recently, I checked last week when looking for a 25G routers and they didn't have switches with such specs yet. but having 100G bandwidth capacility within a LAN is quite nice actually. if you think about it, the latest gen NVMe drives go up to 12.5 GB/s, which is about 100 Gbps. you could setup a compute server with a bunch of GPUs and a data server with a bunch of Gen5 NVMe's. then the compute server could receive data from the data server at the same bandwidth as having NVMe drives internally...
thank you for this wonderfull review. lets see if the prices will drop next year :) one question: This intel xeon pillow - where did you get it? Would love to have one for my homeoffice / homelab. Daniel.
Hi Daniel - At Supercomputing 2022 in Dallas I was sitting and chatting over a drink with an Intel VP as the event closed. The people were coming around to clean up, and the pillows were on the chairs. It sounded like they were going to the dumpster, so I asked if I could take one, and the VP said yes. Xeon Max CPU on one side, GPU Max on the other.
CPU in this switch seems to be more powerful than the one installed in CCR2004-1G-12S+2XS. You can't make it sweat with two 10G copper ports. I should happily route and NAT few gigs of Internet uplink.
I upgraded to 10G 7 years ago using Juniper switches in my homleab, I think its now time for 100G but I want to make the transition smooth so Id need 10G ports and this switch has none :( - Im not super happy seeing router-os either as Id like L3 in the access switch, JunOS is so much more robust. I do have one L2 10G Mikrotik switch today and its .. special
Nice review. Had mine at home for a couple of weeks now, running it in my loft, even when the ambient is over 30 up there, the switch is quiet enough that it's not noticeable at night. I was tempted with the crs504-4xq, glad I waited as I'd have run out of ports already. Using this as my fast switch with 2 bonded ports to a crs510-8xs-2xq for the slower 25g devices. Now I just need to tweak TrueNAS to fill the bandwidth :)
Would love to know if it can do MACSEC to any decent degree. I have three buildings interlinked via dark fiber at 10GbE, and encrypting those links is required. I'd love to upgrade to 100GbE but any gear to do that with MACSEC support is too insanely expensive.
Mikrotik RouterOS 7 (their current firmware) has an immature MacSEC. It's still in development and hardware acceleration or other features are missing. Check again in a year or two
LOL. I would love to have something like that for my NAS Plex Server, laser printer, 2 laptop computers and a PlayStation. Oh yeah, and a wireless hub...
I think the usecase mentioned is pretty mouch the prime usage example for a switch of this kind. Cameras like the URSA 4k can get up to 20gbps for a single camera and when editing a multicam setup of five of these then even a single user can saturate 100gbps. Of course this is a usecase that you will almost never see in the wild (As it is pretty stupid), But even with 2-3 editors in a reasonable use case you could saturate a QSFP28 port.
@@ServeTheHomeVideo I am currently designing a 25gbps SSD NAS for me to use at home. I am on a pretty tight budget so it probably will be a moded old Dell Poweredge. But I can get two sfp28 network cards, a server and the switch for about 800-1000USD plus the price of the SSDs.
Does it function with all its fan modules removed or does it automatically shut down then? I’d like to add a fan duct to external 140 mm fans to cool it this way to reduce the noise level.
I only have 1 computer and 1 phone. No NAS, no real LAN, no other internet devices. I can't afford one. So why do I want one of these?! Am ... am I an addict?
Hey. Is there a such thing as a sfp28 rj45 cat8.1 PHY? I have seen cat8 cable being advertised as a lower cost solution to fiber or Biax but no hardware to support it. Or is it a future proofing technology as in install the cable now so you can use it later.
@@ServeTheHomeVideo Thanks. I find it strange that it is marketed as a replacement for twinax but there are no appliances that actually use it outside of as a replacement for cat7 in the few applications it is better for, like around industrial equipment with alot of emf noise. I was wondering if anything actually used it.
It is affordable for what it is, and it is also absolutely affordable for anyone at or above a middle class (first world) income. There are many hobbies with toys way more expensive than this, sports in particular.
16x 100Gbps vs Mikrotik... 😅 I mean, there's market for them, and I am sure they improved through the years, but I had past experiences that, as much as I want to shell out cash for a few of these switches, I don't know if I should.
@@boneappletee6416 Security: a lot of their devices still ship with hardcoded credentials, or with web/Winbox interfaces buggy AF. Wireless: keeps disconnecting with no reason (unless packet offloading is off). Routing: unit decides to drop static routing tables at will. Switching: won't sustain SFP/QSFP+ speeds for longer periods of time if you have a really busy switch, without dropping packets as if QoS was applied to certain ports. Cooling: Unless in a controlled environment, running on a warmer than standard datacenter thermal recommendations causes some of the troubles above. Those are the most severe ones that comes to mind. And I've tested quite a few hardware products, from small to large carrier switches, as per my day job. Can't trust a device to certain critical functions if they don't offer a resiliency buffer.
we are currently renovating the house and i managed to convince my dad to run cat7 ethernet everywhere, so now we have like 30 runs of cable prolly gonne be more when we starting to renovate the garage and garden shed. What kind of switch would u guys recommend for my usecase? i have a nas with 10gbit would hook my pcs up to 10gbe aswell and the rest could chugg along with 2,5gbit or 1gbit maybe, aps, cameras, tvs and such. i would like to experiment with vms and different vlans in the future maybe have guest wifi and normal wifi and stuff like that
I wonder what kind of support these things have for redundant or “virtual chassis” type setups. Having dual path configuration, even if it’s active/standby is rather critical.
If I was still chasing speed I'd be all over that thing.. but I'm actually down to 1Gb LAN and 200Mb internet and.. stuff just takes a little longer the few times I'm moving gigabytes 🤷
Total noob here, is it possible to connect this switch to another one that uses qsfp+( The 40gbps one) like their flagship 48port switch that has 2 qsfp+ connections? Or more generically (q)sfp28 to (q)sfp+.
Yes, but you will be limited to 40Gbps on that link. Likewise, you can use 10G SFP+ optics in the 25GbE SFP28 ports, but they will only run at 10G speeds.
That is fair. I think for a lot of folks who need/want this much 100GbE they will think this is cheap. 100GbE NICs, optics, DACs and more can be costly.
The day is coming when home users get hardware of this quality. Id love to have 100 gig networking at home, i really would, i already have 10GE and GF, and its here ill be stuck for a long as time, cos nothing above comes with a god damn ethernet uplink port anymore, i might have 2.5 Gb internet, but i have a poxy stupid fiber setup they have used, with transceivers you cant remove and only 1 2.5GB Ethernet out port. I can only sit and prey
so for right now, today, I think that price is just too much. this is the kind of thing I'm going to look back on in 3 or 4 years on ebay when we FINALLY have 100gb nics that aren't 16 lanes (we'll need revs of pcie for that) because cpu manufacturers STILL are not giving us enough pcie on high end consumer cpus. once you stick a gpu in your system (16 lanes), and a few ssd's at 4 lanes a pop (some of them even shared with the onboard components and the system chipset), you're left with MAYBE a 4x slot you can use for a nic... this is fine i guess (if you don't mind sharing that nic's pcie bandwidth with a high speed nvme drive for maybe 10gb networking, but NOT enough for 25, 50, 100g networking and THAT SUCKS! - and even with 10g, there are so few boards that include onboard 10g right now it's dumb and manufacturers are trying to shove 2.5g down our throats despite 10g being available) - and at that point we might have pcie 7 at which time that x4 slot might just have enough bandwidth for 50 or 100g. sure, there's threadripper and the various server platforms, and that's great for backend use, but i'm not about to buy a system with a $1000 premium on top of what it costs for a high end consumer cpu today in order to get a few more pcie lanes and a couple of additional memory channels. as for what I could do with a switch like this, the first thing that comes to mind, honestly, is high bandwidth low latency ceph backend. load up some machines with a combination of hard drives and flash with 100g nics and you've got yourself a very high performance storage system to the point of overcoming one of ceph's primary difficulties (slow write performance due to the way that pg's work) just by throwing some raw network and distributed flash performance at it. one question though (I'm not well versed in the high performance stuff), how would one go from something like this used as a high performance server backend switch to a 10g aggregation switch? is there a qsfp module that can negotiate to multiple sfp+ on the other end for inter-operating with an existing high density 10g network?
@@marcogenovesi8570 there are people who do homeprod and homelab who utilize enterprise class hardware at home. that's my market. I'm also included in the video production market (at home). my current network stack downstairs consists of 5g fiber into an atom based quad sfp+ port pfsense router and then on to a netgear xs728t 10g base-t 24 port enterprise network switch, i've got a 5 node proxmox cluster (each with dual 10g base-t networking), and a 36 hard drive 4u NAS (with 6 nvme ssd's as well), which has dual port 10g networking. upstairs I'm using a zyxel 1930-10 8 port 10g base-t switch (with 2 ports lagg'd to the downstairs netgear switch, also 2 ports lagg'd to my netgear rax80 wifi router (in ap mode) for home wifi access). my workstation (a ryzen 5950x (soon to be upgraded to a 9050x) / 4090 machine has onboard 10g base-t, i've got a secondary workstation (5950x/3090) that I use largely for cg rendering and ai projects, sometimes video encoding). I've looked at expanding my nas setup by going with a multiple machine system with ceph but i'm not quite there yet, probably another 3 or 4 years. while there isn't a HUGE market for doing what I do, it's HARDLY unique.
@@joshhardin666 If you use that equipment for work, you are small businness. If you don't, you probably need help. I have a home lab with a 2 node proxmox cluster with 10gbit and two redundant switches with MLAG to run experiments and stuff but I can't really justify 5 nodes and a large NAS just to play in my spare time.
@@marcogenovesi8570 how do you get quorum with only 2 nodes in your proxmox cluster? (they're generally built out in odd numbers of at least 3). also my proxmox nodes are very low power sff desktops (dell optiplexes) i7-8700, 64gb ram, 512gb sata m.2 to boot and a 2tb nvme for vm storage) and I use it for work a little bit (doing break/fix testing on various software configurations and/or documentation of fixes - i'm a full-time IT consultant and work from home 3/5 days a week.) the big NAS is part plex server, part stream archive (I stream both for fun and for work sometimes doing AV IT and I archive pretty much everything for editing later for both personal and work streams). it's also where I keep vm backups from my proxmox server, general backups from my desktops, etc. centralized archival storage... it's also a bit of a torrent seedbox. my proxmox cluster hosts several websites for personal use, sometimes some game servers, and just recently I started learning about kubernettes so i've been doing a lot of creating and destorying whole k8s clusters recently.. so yeah, it's part homelab, part homeprod.
Regarding nic's first thing the smallest u could go for 100gbe is PCIe v4 x8 for single 100gbe, or PCIe v5 x4 but there is no even NIC for that... IIRC Mellanox (nvidia) has MLX5 with pcie x8 v4
100 GB/s is faster than an Intel Core X memory bus... and we are very close to Terrabit Ethernet. I don't get it. We need an open source consumer mainframe OS
1.6Tbps Ethernet will be deployed in 2027. We just did a video on a 64-port 800GbE Marvell switch two weeks ago. 800GbE would require a PCIe Gen6 x16 NIC just to handle a single port.
Does 100gbe behave like a link aggregation of 25gbe? If it does then you cant get a single file transfer going between systems at 100gb it will only be good for 25gb. Is that true or false?
The term homelabber came after STH which has been around for 15+ years at this point. It has been the largest server, storage, and networking review site for around a decade. Most of the folks getting into homelabbing are doing so to learn things for work. For even a small business this is cheap, and we had an example of that in this video
sfp/+ switches were ALWAYS affordable. it's the transceivers that do the work/cost alot. but the knock off brands do fine for home use. carry on, nothing new here.
You know what would get me to buy Mikrotik hardware again? Not having to use RouterOS with it. Seriously, to _hell_ with that. It's more of a pain than nftables.
That would be a waste, MT finally made a switch that can be a router too. So with routeros on it, you can set up the 10gbit ports as internetconnection (two connections, so dual wan is possible too), use the fast cpu as a router and have it directly providing fast internet to all your devices on the switch!
@@stephanszarafinski9001 You can already do that with several Mikrotik switches, such as the crs312-4c+8xg-rm. The thing is that would suck because RouterOS sucks. It would be one thing if you could just run OPN- or PFsense on the device, but you can't. You're stuck with the horrible, sweaty ass that is RouterOS, which would mean missing basic functionality like DNSSEC support. And yes, you can always run a separate DNS server, but if you're doing that, you might run a separate router and firewall, too, making RouterOS even more pointless and unnecessary and shit. I used a Mikrotik switch as a distribution switch for a year. It was a constant headache and source of anxiety. Moving it downstream into purely an access role in a completely trusted environment where it can basically be a dumb switch was such a relief.
3 місяці тому
Okay... I think cheap might be a bit of a stretch. Affordable, sure, but not cheap 😂
Although I bought the crs520, I still couldn't accept the code quality of the mikrotik team after repeated consideration. So I sold it without even unboxing it.
Yea. The challenge with 10GbE is that it is now so slow that it gets very little investment by companies for new chips. Many of the Realtek switches we are seeing were designed for things like automotive
I mentioned the story in the video about someone who recognized me in LA as the MikroTik Guy that wanted to know about this switch. He has a video production studio and needs a dozen 100GbE links for editors plus a few more for storage.
There are desktop and workstation motherboards with 10Gbps ethernet and pcie lanes supporting 25, 40, and 100Gbps on the consumer market. Its a bit of a trade off between storage, networking, and gpu but I love my MSI x670e ace.
10 gigabit management port lol
More like two :-)
@@ServeTheHomeVideo But why?! Is it cheaper than sourcing 1 Gbit/s ports? Are there any real world situations where a 10 Gbit/s management port is advantageous over a 1 Gbit/s port? Are there any firmware updates that are so big that having 10 Gbit/s just makes sense?
@@MrMartinSchou Mikrotik sometimes does things because they can, not because it makes sense :)
@@MrMartinSchou Yes cheaper. The ARM SoC already has the ports built into it
@@MrMartinSchou By that argument, there's also no need to actually have two management ports.
This switch has some pretty beefy RAM and FLASH compared to other switches, so there's a bunch of things you can do with the management ports. Anything that has to pass data through the CPU could potentially work on it, even routing.
But I think my favorite answer would be "because the CPU supports it." In contrast, I have a CRS-518-16G-2S+IN, which uses a Wi-Fi SOC for its CPU...but it doesn't have Wi-Fi. Would be fun if they added the features supported by the SoC.
Love the option to divide the 100G to 4 pcs 25G. We use it in our datacenter with very expensive Cisco LEAF switch, so I'm excited to try this one from Mikrotek. :)
Can you connect 4 devices to one port that way?
@@Ethan-gu9hm Yes
@@Ethan-gu9hm Yes, That’s the beauty of this port. 🙂
Mikrotik back at it again, making me go "One more piece of hardware, then it should be good for a while... OOO SHINY" thankfully for me, little bit out of the price range I can swing for homelab and not something for any upcoming work deployments.... for now. 25/100G came down a lot quicker than I thought it would and really making me wish I'd waited a bit longer to kit out my homelab, 10G/40G is fine but the specific equipment could have waited and done better. Really looking forward to the continued trickle down
Totally feel you on that. I already have the four port switch but here I am looking for a reason to pick this up -.-
Clean internal design. Thanks for sharing.
Thanks for watching!
keen to see where that product line is going in the next 3-5 years.
they entering the switch market on serious numbers now with ther marvell partnership
something one could only have dreamed about 5-8 years back where they started to use ARM technology
What a coincidence. I was just pricing out Dell S5232F-ON switches for a new small DC install (6 Proxmox hosts. 2 Storage Servers, and 10Gbps internet).
I think we did a S5232F-ON video a few years ago
I love MikroTik switches. ❤
Yes
For me it’s important to have Roce/rdma features and ptp support. If Microtik does it I’m all over it. Use it for storage or high fps camera applications
They really need to get on that. That's the one major factor holding these back
@@cjchico mikrotik and date center switches hahaha
They just introduced PTP support in the latest beta (7.16)
Yep this switch is cool, but I love the 48 GbE / 4 SFP+ / 2 QSFP monster. Perfect to stuff around in a home lab and learn, and amazing flexibility to do weird and wonky protocols on every port!
Looking forward to the day a low power 8 port 100G switch is $500 or less, maybe in a couple of years 😄
Getting there slowly
@@ServeTheHomeVideowhat's the point of this why would anyone in their home would need this
@@chriswright8074 NAS
@@chriswright8074 I do to hook up all of my home lab ProxMox servers with CEPH!
@@chriswright8074 why would anyone ever need more than 640kb of RAM...
Holy crap. I immediately think "What kind of loads would generate this sort of traffic?" in small environments? 12.5 GB/s max... I can only imagine the kinda SAN/NAS you'd need to push that bandwidth, number one, and to multiple clients at the same time on top of that? Woo! I'm dreaming of anything faster than my old ass Synology with hard drives... I know, I'll see myself out haha
Thanks for the vid! This is really awesome.
Blackmagic Design has a 100GbE NAS as an example.
easy when the SSD's they have push like 7.5GB/d each...
I mean, isn’t a PCIe Gen 5 SSD 12-14GB alone?
@@levygaming3133 Sure, but do you personally have a SAN with this capability? I sure don't. How many people really do? That was my point - that this is way overkill for most users - even though I could have made it a bit better.
@@ServeTheHomeVideo That's wild. Holy hell. I can't even imagine the cost of the SAN plus enough drives of the right capacity and generation to push that much data to multiple users.
And here i am dreaming of a 2.5gb/10gb switch..
i was so happy to finally have 2.5G SW and 10Gb to my hypervisor .... now it feels so outdated in comparison lol
Yes, I just upgraded everything to 2.5gb ethernet, although one connection goes over a MOCA 2.5gb link (works really well). Paid the extra for a wifi 7 router with 2 10gb ports and 4 2.5gb ports. Now waiting for the fiber connection that was laid to my yard so I can sign up for Google fiber. I have been waiting since December.
So it isn't something I would do now but something I would like to do. In the professional video space there is something called IPMX. It allows for extremely low latency video reproduction. I would love if companies like microsoft and parsec implemented it into their remote desktop protocols. It uses a lot more bandwidth but for that you can get sub ms latency at least as far as the encoding goes.
I'm still rocking the Mellanox MSB-7890 36-port 100 Gbps Infiniband externally managed switch that I picked up in 2018 for $2950 CAD (at the time, which works out to be something like $2300 USD, used).
For a homelab AI/HPC cluster, I'm definitely using 100 Gbps.
That is a great switch, I have their crs510 another great product.
Totally. This is just a step up in capacity. I think it is good to have different price/ capacity options
Now someone needs to go make a pcie 5.0 x4 100gbe adapter that can go in a pcie or m.2 slot
Yes! Actually one of the big issues is power for something like this
@@ServeTheHomeVideo Makes sense, but I doubt anyone would mind having to sacrifice a sata power plug for something like that.
That price is very reasonable for this capacity, I have paid a lot more for inferior products. I want one of these now 🤔
This is perfect for my 50Mb internet speed and my 1Gb homelab!
First mikrotik only adds that H3 ofload will be expanded. L2 works as it should.
2. Winbox case if you know it is a great interface (cli is always available)
3. L3 is executed by ARM processor and this is where mikrotik works to do as much as possible through hardware support by switch system
Decent performance in mikrotik only in L3 in L2 we have high performance
It would be nice to see mikrotik switches that can be stacked.
they support 2-node MLAG but people that tried it said it's currently scuffed so i would not recommend for that until they fixed it.
Why? It's 2024, eVPN is where its at this decade.
I don't suppose it supports RDMA/RoCE at this price point?
no, afaik mikrotik does not usually support those in any switch
If you have zero touch rdma cards you dont need that.
Not yet....
MikroTik finally dropping MIPS for management. Pour one out for MIPS.
They have been on this path for some time
Yup, price is reasonable
And me happy to just moved to 2.5Gbe.
Its not bad, getting close to 300mb/s from my NAS.
Faster than single hdd.
Far away from average ssd.
Hey man, can you review the QNAP QSW-M7308R-4X-US? I think that one is potentially better than this one, especially for a home lab. It costs 1k, has 4 ports at 100Gbps, and 8 at 25Gbps which is great for lots of people.
That is on the list. We also have one of their 10Gbase-T and SFP+ switches
Looking at my 5 port 2.5g unmanaged switch that cost me $120 angrily 🤨
We have tested 50+ 2.5GbE switches on the main site and made an Ultimate Buyer's Guide for them. You can now get unmanaged 4-port 2.5GbE and 2-port SFP+ for $45 or less.
I would love a 16 port 2.5gb switch with 8 10gb ports. Sadly can’t find anything close to this at a decent price.
i dont have any 100g capable clients or any use case but i want one so badly
Man! once upon a time, we could only afford 10Mbps for workstations and the 100Mbs was only for the servers! We just added a few zeros to the right :)
I wonder if this switch can be used as router as it was mentioned somewhere. BTW Mellanox SN2700 which is 32 port 100G switch has similar power consimption to this Mikrotik. I have this Mellanox SN2700 at home and it draw about 45w idle with no transceivers plugged in. Now with 11x QSFP28 power consumption is 96w. I bought few of these Mellanox SN2700 for an ISP in my area for little less than $1000 each. They gave me 1 for free as a compliment :)
-Edit-
Actually we weren't aware of the power consumption of this Mellanox before we buy and our expectations were higher and this was a pleasant surprise :)
For those who think this switch is expensive... Have a look at the new black magic design 2110 switch...
Patrick, why don't you have a look at that switch? Would like to know what your thoughts are
That is in my shopping cart.
@@ServeTheHomeVideo did you also put an SDI to 2110 converter in there? 😎 Looking forward to the review
lol I was so proud of my 2.5 gig network with nic giga switches I finished setting up yesterday
Mikrotik comes in swinging, I was suspecting they would since Marvell acquired AHEM made that 100Gbe switch chip technology
This is *exactly* what I'm looking for.
0:23 yeah, THAT "feature" is the reason, as to why i am not really looking at mikrotik as my first option.
(Well that is mostly due to one of the exercises one could get on my exam was inter vlan routing, and setting that without any documentation is pure pain)
I think this is pretty nice option for many small businesses, which want to have new switch with ability to upgrade servers down the road in next 1-2 years to use 100Gbps for CEPH for example. I think even i could use it for 2-3 server cluster with CEPH under proxmox (yeah, 2 servers for quorum is great idea /s), so that i would still have 10 ports left for other stuff down the road (and most likely get another one to have HA after some time). Even "cheap" Supermicro servers with U.2 slots can be bought new for like 15-18k, which easily could hit stable 12GBps or more, so price-wise i think it have everything what users would need - like You said. 2k is a bargain for me.
You can use 2 servers + Qdevice so it has quorum for Proxmox (but not Ceph), and with just two nodes you are better off with zfs and replication or even the linbit drbd block device replication than Ceph. At least imho
@@marcogenovesi8570 I`m starting to learn about CEPH, replication, HA and stuff in this area - just today got second Lenovo SFF with i5-6500T so that i can create 2 proxmox nodes and learn about replication without touching my main server (Epyc 7D12) so that if brown hit the fan i can just wipe it and start from scratch. Main goal is to learn it and use at work, so that i can create "proper" production environment or just baseline so that someone after me could continue it, to get it done "cheap" and make it fly :)
Full Mesh Network (with frr+fabricd) is a better option for a small 3 node cluster then to use a single SAN Switch.
pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Routed_Setup_(with_Fallback)
If you are on the budget, you can look for used 40/56GbE Mellanox ConnectX-3 NICs (like MCX354a-FCBT) and QSFP+ or QSFP28 DAC cables.
that one must have been released very recently, I checked last week when looking for a 25G routers and they didn't have switches with such specs yet.
but having 100G bandwidth capacility within a LAN is quite nice actually. if you think about it, the latest gen NVMe drives go up to 12.5 GB/s, which is about 100 Gbps.
you could setup a compute server with a bunch of GPUs and a data server with a bunch of Gen5 NVMe's. then the compute server could receive data from the data server at the same bandwidth as having NVMe drives internally...
thank you for this wonderfull review. lets see if the prices will drop next year :) one question: This intel xeon pillow - where did you get it? Would love to have one for my homeoffice / homelab.
Daniel.
Hi Daniel - At Supercomputing 2022 in Dallas I was sitting and chatting over a drink with an Intel VP as the event closed. The people were coming around to clean up, and the pillows were on the chairs. It sounded like they were going to the dumpster, so I asked if I could take one, and the VP said yes. Xeon Max CPU on one side, GPU Max on the other.
CPU in this switch seems to be more powerful than the one installed in CCR2004-1G-12S+2XS. You can't make it sweat with two 10G copper ports. I should happily route and NAT few gigs of Internet uplink.
yeah but it's a bit pricey for "a few gigs of uplink"
@@marcogenovesi8570No switch is perfect.
Maybe a bit much for most home lab environments. But bad ass switch like most things MT does.
Awesome!
One question though...
Does it come in black :p
It does if you want to spray paint it :)
It always is Mikrotik.
Mikrotik are good.
Are there any low power switches for home use that are less enterprise/rack mount looking? Like if you just need maybe 4 ports as fast as possible?
Check out the CRS504 review linked in the description.
I upgraded to 10G 7 years ago using Juniper switches in my homleab, I think its now time for 100G but I want to make the transition smooth so Id need 10G ports and this switch has none :( - Im not super happy seeing router-os either as Id like L3 in the access switch, JunOS is so much more robust. I do have one L2 10G Mikrotik switch today and its .. special
Unbeatable wow
Nice review. Had mine at home for a couple of weeks now, running it in my loft, even when the ambient is over 30 up there, the switch is quiet enough that it's not noticeable at night. I was tempted with the crs504-4xq, glad I waited as I'd have run out of ports already. Using this as my fast switch with 2 bonded ports to a crs510-8xs-2xq for the slower 25g devices. Now I just need to tweak TrueNAS to fill the bandwidth :)
I need a switch with 16x 100Gig, and 24x 10Gig 8P8C (RJ45), with layer3 switching... or 8x 100Gig and 32x+ 10Gig 8P8C (RJ45) with layer3 switching
Would love to know if it can do MACSEC to any decent degree. I have three buildings interlinked via dark fiber at 10GbE, and encrypting those links is required. I'd love to upgrade to 100GbE but any gear to do that with MACSEC support is too insanely expensive.
Mikrotik RouterOS 7 (their current firmware) has an immature MacSEC. It's still in development and hardware acceleration or other features are missing. Check again in a year or two
LOL. I would love to have something like that for my NAS Plex Server, laser printer, 2 laptop computers and a PlayStation. Oh yeah, and a wireless hub...
I think the usecase mentioned is pretty mouch the prime usage example for a switch of this kind. Cameras like the URSA 4k can get up to 20gbps for a single camera and when editing a multicam setup of five of these then even a single user can saturate 100gbps. Of course this is a usecase that you will almost never see in the wild (As it is pretty stupid), But even with 2-3 editors in a reasonable use case you could saturate a QSFP28 port.
I think a future Mac Studio will go 100GbE. Blackmagic has 100GbE NAS or at least footage dump options already
@@ServeTheHomeVideo I am currently designing a 25gbps SSD NAS for me to use at home. I am on a pretty tight budget so it probably will be a moded old Dell Poweredge. But I can get two sfp28 network cards, a server and the switch for about 800-1000USD plus the price of the SSDs.
The moment when your network storage is faster than your internal PCIe5 M2.0 ...
Does it function with all its fan modules removed or does it automatically shut down then?
I’d like to add a fan duct to external 140 mm fans to cool it this way to reduce the noise level.
I only have 1 computer and 1 phone. No NAS, no real LAN, no other internet devices. I can't afford one.
So why do I want one of these?! Am ... am I an addict?
For the business you are starting!
Can you please check its speed with NICs supporting iwarp instead of Roce.
Hey. Is there a such thing as a sfp28 rj45 cat8.1 PHY? I have seen cat8 cable being advertised as a lower cost solution to fiber or Biax but no hardware to support it. Or is it a future proofing technology as in install the cable now so you can use it later.
25G and higher you are better off just using fiber. Realistically over 10G folks seem not to be pushing copper outside of short-range DACs.
@@ServeTheHomeVideo Thanks. I find it strange that it is marketed as a replacement for twinax but there are no appliances that actually use it outside of as a replacement for cat7 in the few applications it is better for, like around industrial equipment with alot of emf noise. I was wondering if anything actually used it.
Jeez! why do I always get suckered into clicking the link when I see affordable in the description.
It is affordable for what it is, and it is also absolutely affordable for anyone at or above a middle class (first world) income. There are many hobbies with toys way more expensive than this, sports in particular.
Will the GUI allow you to apply the same set of changes to more than one interface in one go?
Does this support stacking and ToR functionality? If so then it is seriously cheap.
Annapurna Labs, sheeesh that thang gonna overheat
Do they still lack Openflow support?
still no word on the size of the TCAM? their L3HW help page still doesn't list this switch.
now we need offorable mainboards/cpus with tons of PCIe lanes to have a ton of NVME drives to saturate the 100 gbit 🙂
100GbE is enough for only one PCIe Gen5 M.2 SSD.
@@ServeTheHomeVideoYep, :)
16x 100Gbps vs Mikrotik... 😅 I mean, there's market for them, and I am sure they improved through the years, but I had past experiences that, as much as I want to shell out cash for a few of these switches, I don't know if I should.
What sort of issues did you have? Genuinely curious
@@boneappletee6416Me too
@@boneappletee6416 Security: a lot of their devices still ship with hardcoded credentials, or with web/Winbox interfaces buggy AF.
Wireless: keeps disconnecting with no reason (unless packet offloading is off).
Routing: unit decides to drop static routing tables at will.
Switching: won't sustain SFP/QSFP+ speeds for longer periods of time if you have a really busy switch, without dropping packets as if QoS was applied to certain ports.
Cooling: Unless in a controlled environment, running on a warmer than standard datacenter thermal recommendations causes some of the troubles above.
Those are the most severe ones that comes to mind. And I've tested quite a few hardware products, from small to large carrier switches, as per my day job. Can't trust a device to certain critical functions if they don't offer a resiliency buffer.
16:57 How does one read these tables? Does 36424 Mbps (36 Gbps) mean it didn't hit even close to the 100 Gbps?
we are currently renovating the house and i managed to convince my dad to run cat7 ethernet everywhere, so now we have like 30 runs of cable prolly gonne be more when we starting to renovate the garage and garden shed.
What kind of switch would u guys recommend for my usecase?
i have a nas with 10gbit would hook my pcs up to 10gbe aswell and the rest could chugg along with 2,5gbit or 1gbit maybe, aps, cameras, tvs and such.
i would like to experiment with vms and different vlans in the future maybe have guest wifi and normal wifi and stuff like that
I wonder what kind of support these things have for redundant or “virtual chassis” type setups. Having dual path configuration, even if it’s active/standby is rather critical.
They support bonding, LACP for basic failover/bonding and MLAG for using two switches as redundant with bonding
Does it generate netflow through the management port?
can we get a "smaller" one with like 4x 100gbit and 8x 10/25gbit? :D
Check the links below for the CRS504, CRS512 and so forth
Check the links below for the CRS504, CRS512 and so forth
@@ServeTheHomeVideo cheers much appreciated
i am still looking for an afordable vlan/trunk switch with at least 8 downstream ports 1gig and one uplink port of at least 2.5 gig.. any suggestions?
CRS310-8G+2S+IN ?
Can you please review this cheap switch
Xiaomi Mi 10G Ethernet Switch
Has 2 sfp + ports and rest 2.5gb
If I was still chasing speed I'd be all over that thing.. but I'm actually down to 1Gb LAN and 200Mb internet and.. stuff just takes a little longer the few times I'm moving gigabytes 🤷
Total noob here, is it possible to connect this switch to another one that uses qsfp+( The 40gbps one) like their flagship 48port switch that has 2 qsfp+ connections? Or more generically (q)sfp28 to (q)sfp+.
Yes, but you will be limited to 40Gbps on that link. Likewise, you can use 10G SFP+ optics in the 25GbE SFP28 ports, but they will only run at 10G speeds.
The SFP´s will make it expensive :D
I mean you can get SR optics for under $60 new
SFPs make everything expensive
cheap it is but well beyond my price point for a switch.....that is for sure.
That is fair. I think for a lot of folks who need/want this much 100GbE they will think this is cheap. 100GbE NICs, optics, DACs and more can be costly.
I just half switched to 2.5gig....what the hell is going on
The day is coming when home users get hardware of this quality.
Id love to have 100 gig networking at home, i really would, i already have 10GE and GF, and its here ill be stuck for a long as time, cos nothing above comes with a god damn ethernet uplink port anymore, i might have 2.5 Gb internet, but i have a poxy stupid fiber setup they have used, with transceivers you cant remove and only 1 2.5GB Ethernet out port.
I can only sit and prey
By ethernet do you mean rj45?
Is it backwards compatible to 2.5 gig? LOL
Ha!
No RDMA/DCB? C'mon Mikrotik
so for right now, today, I think that price is just too much. this is the kind of thing I'm going to look back on in 3 or 4 years on ebay when we FINALLY have 100gb nics that aren't 16 lanes (we'll need revs of pcie for that) because cpu manufacturers STILL are not giving us enough pcie on high end consumer cpus. once you stick a gpu in your system (16 lanes), and a few ssd's at 4 lanes a pop (some of them even shared with the onboard components and the system chipset), you're left with MAYBE a 4x slot you can use for a nic... this is fine i guess (if you don't mind sharing that nic's pcie bandwidth with a high speed nvme drive for maybe 10gb networking, but NOT enough for 25, 50, 100g networking and THAT SUCKS! - and even with 10g, there are so few boards that include onboard 10g right now it's dumb and manufacturers are trying to shove 2.5g down our throats despite 10g being available) - and at that point we might have pcie 7 at which time that x4 slot might just have enough bandwidth for 50 or 100g. sure, there's threadripper and the various server platforms, and that's great for backend use, but i'm not about to buy a system with a $1000 premium on top of what it costs for a high end consumer cpu today in order to get a few more pcie lanes and a couple of additional memory channels. as for what I could do with a switch like this, the first thing that comes to mind, honestly, is high bandwidth low latency ceph backend. load up some machines with a combination of hard drives and flash with 100g nics and you've got yourself a very high performance storage system to the point of overcoming one of ceph's primary difficulties (slow write performance due to the way that pg's work) just by throwing some raw network and distributed flash performance at it.
one question though (I'm not well versed in the high performance stuff), how would one go from something like this used as a high performance server backend switch to a 10g aggregation switch? is there a qsfp module that can negotiate to multiple sfp+ on the other end for inter-operating with an existing high density 10g network?
It's obviously not targeted at consumers but more to small-medium businness
@@marcogenovesi8570 there are people who do homeprod and homelab who utilize enterprise class hardware at home. that's my market. I'm also included in the video production market (at home). my current network stack downstairs consists of 5g fiber into an atom based quad sfp+ port pfsense router and then on to a netgear xs728t 10g base-t 24 port enterprise network switch, i've got a 5 node proxmox cluster (each with dual 10g base-t networking), and a 36 hard drive 4u NAS (with 6 nvme ssd's as well), which has dual port 10g networking.
upstairs I'm using a zyxel 1930-10 8 port 10g base-t switch (with 2 ports lagg'd to the downstairs netgear switch, also 2 ports lagg'd to my netgear rax80 wifi router (in ap mode) for home wifi access). my workstation (a ryzen 5950x (soon to be upgraded to a 9050x) / 4090 machine has onboard 10g base-t, i've got a secondary workstation (5950x/3090) that I use largely for cg rendering and ai projects, sometimes video encoding). I've looked at expanding my nas setup by going with a multiple machine system with ceph but i'm not quite there yet, probably another 3 or 4 years.
while there isn't a HUGE market for doing what I do, it's HARDLY unique.
@@joshhardin666 If you use that equipment for work, you are small businness. If you don't, you probably need help.
I have a home lab with a 2 node proxmox cluster with 10gbit and two redundant switches with MLAG to run experiments and stuff but I can't really justify 5 nodes and a large NAS just to play in my spare time.
@@marcogenovesi8570 how do you get quorum with only 2 nodes in your proxmox cluster? (they're generally built out in odd numbers of at least 3). also my proxmox nodes are very low power sff desktops (dell optiplexes) i7-8700, 64gb ram, 512gb sata m.2 to boot and a 2tb nvme for vm storage) and I use it for work a little bit (doing break/fix testing on various software configurations and/or documentation of fixes - i'm a full-time IT consultant and work from home 3/5 days a week.) the big NAS is part plex server, part stream archive (I stream both for fun and for work sometimes doing AV IT and I archive pretty much everything for editing later for both personal and work streams). it's also where I keep vm backups from my proxmox server, general backups from my desktops, etc. centralized archival storage... it's also a bit of a torrent seedbox. my proxmox cluster hosts several websites for personal use, sometimes some game servers, and just recently I started learning about kubernettes so i've been doing a lot of creating and destorying whole k8s clusters recently.. so yeah, it's part homelab, part homeprod.
Regarding nic's first thing the smallest u could go for 100gbe is PCIe v4 x8 for single 100gbe, or PCIe v5 x4 but there is no even NIC for that... IIRC Mellanox (nvidia) has MLX5 with pcie x8 v4
100 GB/s is faster than an Intel Core X memory bus... and we are very close to Terrabit Ethernet. I don't get it.
We need an open source consumer mainframe OS
1.6Tbps Ethernet will be deployed in 2027. We just did a video on a 64-port 800GbE Marvell switch two weeks ago. 800GbE would require a PCIe Gen6 x16 NIC just to handle a single port.
Does 100gbe behave like a link aggregation of 25gbe? If it does then you cant get a single file transfer going between systems at 100gb it will only be good for 25gb. Is that true or false?
why are you trolling?
No, it does not. You get a single 100GbE link unless you split qsfp port intentionally.
@@mvs3246 Nice. Thanks.
Now that makes it an interesting choice between the 4*100gbe mikrotik or the 16*25gbe mikrotik switch for me.
that thing costs more then my server xd
😂😂 First? The vibe couple months ago was the best 2.5G switch and now we are talking in 100G.
I mean last week we did 800GbE :)
@@ServeTheHomeVideo honestly all these speeds make me giddy
This channel really needs to change its name. Homelabbers aren't looking to spend 2 fucking grand on a switch.
The term homelabber came after STH which has been around for 15+ years at this point. It has been the largest server, storage, and networking review site for around a decade. Most of the folks getting into homelabbing are doing so to learn things for work. For even a small business this is cheap, and we had an example of that in this video
"100 gig networking is not even fast anymore"
Its fast enough unless your a hyperscaler.
Motherf*** I literally just got the last piece for my 10g homelab in. F*** right off with this bullshit 😢
I still didn't get a switch with enough POE++'es and SFP's so no need for 100 gig..
We have the new CRS320 review recorded. It will go live in Aug
sfp/+ switches were ALWAYS affordable. it's the transceivers that do the work/cost alot. but the knock off brands do fine for home use.
carry on, nothing new here.
Our last firmware update downgraded random ports to 10 or 100 mbps tx. It took some time to get a fix.
Mikrotik lost some of my trust with this
around 2.460,80 EUR are not cheat, cheat is realated to around 100-400EUR or $.
There are sellers with these under 2,000 EUR. For this class of switch that is cheap
You know what would get me to buy Mikrotik hardware again? Not having to use RouterOS with it. Seriously, to _hell_ with that. It's more of a pain than nftables.
That would be a waste, MT finally made a switch that can be a router too. So with routeros on it, you can set up the 10gbit ports as internetconnection (two connections, so dual wan is possible too), use the fast cpu as a router and have it directly providing fast internet to all your devices on the switch!
@@stephanszarafinski9001 You can already do that with several Mikrotik switches, such as the crs312-4c+8xg-rm. The thing is that would suck because RouterOS sucks. It would be one thing if you could just run OPN- or PFsense on the device, but you can't. You're stuck with the horrible, sweaty ass that is RouterOS, which would mean missing basic functionality like DNSSEC support. And yes, you can always run a separate DNS server, but if you're doing that, you might run a separate router and firewall, too, making RouterOS even more pointless and unnecessary and shit.
I used a Mikrotik switch as a distribution switch for a year. It was a constant headache and source of anxiety. Moving it downstream into purely an access role in a completely trusted environment where it can basically be a dumb switch was such a relief.
Okay... I think cheap might be a bit of a stretch. Affordable, sure, but not cheap 😂
Nothing to see here since it doesn't have usable OS. Roce/rdma, MLAG et. not working or non existent. Good hardware, but's about it.
Although I bought the crs520, I still couldn't accept the code quality of the mikrotik team after repeated consideration. So I sold it without even unboxing it.
@actckactck2042 and that is the reason I went for other vendors. Either used or new, they are better than Mikrotik.
I cannot imagine shelling out 3 grand CAD for a microtik device, I am sorry but I'd take any used appliance from another company over that lol
Why would anyone need a 100 gig
Yet any decent managed 10G switch with 10+ ports costs 50% of this in 2024 ... 🤮
Yea. The challenge with 10GbE is that it is now so slow that it gets very little investment by companies for new chips. Many of the Realtek switches we are seeing were designed for things like automotive
Since when small office need 16 x 100GbE ?! we can barely get 1 Gbe Wan connection and all our pc run 1GbE
I mentioned the story in the video about someone who recognized me in LA as the MikroTik Guy that wanted to know about this switch. He has a video production studio and needs a dozen 100GbE links for editors plus a few more for storage.
There are desktop and workstation motherboards with 10Gbps ethernet and pcie lanes supporting 25, 40, and 100Gbps on the consumer market. Its a bit of a trade off between storage, networking, and gpu but I love my MSI x670e ace.
"Affordable" 🤣
Lost me at $2,000 lol. I'm cheap when I see cheap I think $100 or less.
Cheap? LOL, I got a Questone2 for $160