Thanks to Maximum Settings Cloud Gaming for sponsoring today's video. Get started with your very own Cloud Gaming VM or Bare Metal Machine at bit.ly/3wxfUuB!
I can't comment anything useful, it seems UA-cam is auto-deleting it. We shall see if it is going to you for review or if UA-cam decides to magically allow 5-6 comments in a few minutes.
@@KiraSlith nock to tip. The length of an arrow. At least thats what I hope he meant cuz I dont think he has the leeway to measure things in dog D1ks on youtube.
I would pull the watch cell battery to reset all settings, and move JP9 to 2-3 for the recovery bios, also remove all sticks of memory expect 1 to remove timing retaining or bad memory channel. Also double check your VGA cable is still good just encase.
One of the largest server OEMs on earth. Think bigger then Dell or HP. They could care less about individual consumers. They sell exclusively to large enterprises and hyperscalers.
Pretty sure I dealt with a desktop type system made by them many years ago. From what I remember, I wasn't impressed with the build quality or their support. I think that it was bought through Costco Wholesale, and probably returned for a refund, after I tried to fix it for the customer. I also think that I might have been the third or fourth tech to attempt to repair it.
Did you notice the CONSOLE SELECT SW (lower part of the screen at 13:13)? Options are BIOS or BMC. I would guess it's set to BMC right now. Maybe switching to BIOS will give you access to the bios that you want.
Put a GPU into each slot. My Asrock AM4 server board has this same issue. You can disable the onboard VGA which connects to the ASPEED BMC. Get a dummy plug into a port on each GPU and I bet it might post
This is why my homelab is all Dell PowerEdge. Plug the service tag into their support site and you get ALL the manuals, drivers, and information you need, even the build sheet for your unit's factory configuration. The only things they seem to lock down behind an account are their XC series, but those are just rebadges of other models so you just have to look up the equivalent regular model. HP almost got it right, but they constantly break their support pages and practically disowned everything made before the split.
In that case I think I'll just buy Dell and just Dell computers only. The toolless designs of the older OptiPlexes and the newer Precision SFF models are my favorite, especially with their blue case lock.
@@CraftComputing did you read? @duncandubick7535 vor 23 Stunden Did you notice the CONSOLE SELECT SW (lower part of the screen at 13:13)? Options are BIOS or BMC. I would guess it's set to BMC right now. Maybe switching to BIOS will give you access to the bios that you want.
This makes me appreciate my old Supermicro servers even more. Standard cables, standard board holes, standard behavior, straight forward BMC (well, as far as bmcs go anyway) everything just works.
At work I mostly deal with Supermicro servers, sometimes Gigabyte and Tyan as well. I much prefer Supermicro out of these three, very pleasent to work with. Thoughtfully built hardware and firmware, I don't know how they compare in other regards like cost, but they sure are nice to work with.
The Quanta T22HF-1U is a 1U rackmount server manufactured by Quanta Cloud Technology Inc. It supports up to 400GbE networking bandwidth in x16 PCIe 5.0 slots, and has enhanced serviceability with tool-less, hot-swap designs. The server is NEBS compliant for Telco/5G data center deployments, and liquid cooling is supported with up to 385W CPU TDP. Might need some Verizon/ATT server guys to chime in...
In the video you tried a Naples CPU. Have you tried a Rome or Milan CPU? The firmware ("BIOS") on those systems contain one or more AGESA packages. Each AGESA is for only one generation. I don't know how Quanta does it (I've only worked with Quanta systems with Intel CPUs) but other barebones manufacturers often will have two AGESA in their firmware. e.g. so you could only use a Naples or Rome, unless you reflashed a different supporting version, then you could use a Rome or Milan CPU. When Milan came out, we had the fun time of having to install a Rome just to flash, then remove it and install the final Milan. First gen Epyc boards didn't have enough flash memory to hold an image with both AGESAs, so they were Naples only. tl;dr: Try a Rome and a Milan CPU to check for compatibility.
I bought a Quanta Server T42S-2U on ebay and can confirm I feel the same pain :). I was able to install a skylake cpu with ecc and was able to boot the node and use IPMI to reset the bmc password. in my example I was only able to boot once I had both cpu sockets (of one blade) installed with a genuine intel cpu. ES or QS was not working as was only one cpu. If you want to have a look at something on that system let me know I can see what I can do. as for your blade I guess that usb and vga are disabled (which is something a hypervisor would request) so getting into the bios here would be ..... a challenge. As for documentation, the best option would be way back machine. also my system had different names for the blade (which had documentation) and the chassis which had less so try to see what the blade name is and google on that (tip might be a part list of the chassis part number).
tried that, they dont seem to have html links to their files its just a bunch of gibberish in the download link. Was hoping it was possible to just substitute the filename thinking they might just be lazy enough to leave the files there and delete the links. Doesnt seem to be the case. Gonna take someone more skiled than I to poke around. Wayback machine seeems to be a bust too
@@doomboots yeah from what I can find my model Quantaplex T42S-2U is still findable which led me to the fru part list for a motherboard / blade: T42S MB Assy (LBG-1G) but trying the same search this for the T22HF makes me find nothing. It also looks like the quanta site even in 2018 all the way up to 2024 only has 4 node multimode systems, I was also not able to find ANY T22 based models from 2018-now in product slides or support links so think its a hidden or special sku which sucks. I will see if the bmc can change bios settings so at least you can install an os. I will see if I can find the manual from my Quantaplex T42S-2U since I think the blade had an BMC and BIOS reset header on the board.
i have the same Server here, everything working fine. But it was pain in the ass to get it working. To log in to the BMC i had to update the bmc firmware via a windows update tool cli-thing... I sent mails to their support (germany) and they were very helpful. There are some usernames/passwords, that i've found on the internet admin/admin admin/password ADMIN/ADMIN ADMIN/qct.admin qct.admin/qct.admin admin/cmb9.admin hope i can help
Some things: 1. Quanta is a vendor for HyperScaler companies. Meaning Google, Oracle, Microsoft, Meta, etc. This is typical of a quanta machine. 2. Are you absolutely sure these blades support AMD EPYC 7601 CPUs? Those are 180W TDP CPUs and are power hungry. 3. What wattage are those PSUs rated for? Typically, PSUs need to be 2-3x more powerful than CPU TDP (180 x 2 or 180 x 3). Then multiply that times 2. You have two nodes. Have you tried plugging in both PSUs? It's possible it wasn't configured for Single PSU use and machines won't boot unless both PSUs are present. 4. Quanta removes documentation due to the fact they mostly only sell to larger hyperscalers. 5. Do the nodes get IPs for IPMI? You have the BMC yes. But each node ALSO gets it's own IPMI as well. Try looking into that. 6. Do you have any storage plugged in? 7. Remove all RAM except for 1 DIMM. See if that makes a difference. That's all I can think of honestly. Oh. Last point. You get what you pay for....
Worked on many many Quanta systems, and they are pretty "cost engineered" but also generally fairly solid for what they were meant to do - BUT also very much designed to spec for the company that "engineered" them. (often in lots of 10K+ units min) You won't find any specs at all for the systems I typically used unless you work for the company that initially requested them contract manufactured. The flipside, is typically they are ALSO typically used so hard by the time they are EOLed there isn't really a lot of life left typically. I know one hyperscaler I worked for, we ran ours at least 50%, and the DCs weren't terribly cold for environmental reasons, so those servers cooked for those 5-7 years. At this point, they are mostly e-waste, yes. But everything eventually ends up there. But ya I'd bet its the wrong CPU "family" AGESA FW wise.
If/when i become God-Queen-Emperor, my first decree will be that beer shall be free. My second decree shall be that lights out management requiring a cloud login shall be punishable by surrender and liquidation of all company assets to charitable causes.
To reset the BIOS from the BMC command line on a Quanta server, you can try log in to the BMC's CLI and issue the command reset sp. This will reset the BMC module, which in turn might reset the BIOS to its default settings
I have a similar purchase that is going slightly better. I found a SuperMicro 2U/4-node system of the X8 generation, so Intel, older, and slower, but each node has 2 sockets, so I actually have 8 sockets in a 2U chassis, but the intriguing thing for me was the boards all included 40Gig Infiniband (10Gig Ethernet) ConnectX-2 network cards. The down side is the onboard storage. No SATA connectors on the board, so no SATA-DOM. No M.2 slots. No SAS. Each node has access to 3 LFF drives on the front backplane and that's it. _But_ with the 40Gig IB, (I already have a Mellanox switch that supports 40gig ETH and IB) I figured I could probably work out some sort of iSCSI solution that would be plenty fast, but that has turned out to be a challenge. Trying to get a modern version of FlexBoot (Mellanox's PXE/iSCSI boot firmware) onto one of these older ConnectX cards (let alone one built onto a server board) has turned out to be a challenge. I'm probably going to abandon that and just boot from an iPXE ISO and let it handle the iSCSI connectivity, but that's a new development that I haven't had a chance to play with much. I did manage to wreck the pins on one of the CPU sockets, but I fortunately found a pair of motherboards on ebay for a reasonable price. They came with an unexpected surprise... proprietary BIOS. No interesting features, so I tried to flash to the stock BIOS but that was a no-go, so just swapped the BIOS chip from the damaged board and all was well. Eventually I hope to get XCP-NG running on all of the nodes and then mess around with some of the HA features, but I'll cross that bridge when I get to it.
Im using an old X9 supermicro 1U as a deployment server. Works really great. It has an i3 in it and 3 gigabit nics. two internal and one in the expansion slot. The two 40mm fans in it is really bad.. supernoisy :) but.. not worth replacing for now.
Goddamnit Jeff, you beat me to this! I was researching this 3 months ago to house some of my extra 7451's. If you need, I ended up getting the Quanta QVL for this from the support team. It only shows support for the 7xx2 series, but it does have other information, so I never ended up buying.
You could try to use ipmitool remotely to directly change settings with your credentials or to get access to SOL to get access to the BIOS of the board
@mer I meant more like the Serial-over-LAN access, but the system event log might help find issues, yup Serial-over-LAN allows the content of the BIOS to be streamed over SSH(ish) during boot, so you can make BIOS settings without VGA output (and thus potentially enable VGA)
If the fans are running full speed like that all the time, chances are its not posted. Re-seat RAM, CPU, and find a bios reset jumper. Or could be a quirk that both blades need to be running in order for it to post?
Always enjoy the videos, success or not so successful, they are all a good watch. I had John Coltrane playing in the background durring the vid. That kinda jazz fits right in to experimental homelab electronics. Cheers
1) So I tried looking for the manual -- no cigar. I did turn up that the model of the motherboard itself is known as "S5H(F) MB". You see the s5h referenced a bunch of times in your dmesg output and console logs. 2) The other thing that I noticed was that right next to the big red reset button, there's a dip switch (SW1) which says "CONSOLE_SELECT_SW". The picture, in that area, doesn't have enough resolution for me to be able to read the silk screening on the switch itself (I think that I see a '4', but I can't tell if the other thing is a '10' or something else.) So, maybe you can try playing with that to see if that might change stuff. 3) The other thing that you might also try is do drop down to only one stick of RAM. That might be a challenge because without the manual, you don't know which is the primary DIMM slot, so there is a chance that you might not get very far with this. 4) This is why I buy Supermicro. Yes, they might be more expensive, but their stuff well documented. 5) I think that there's been discussions about Supermicro paywalling their IPMI behind a subscription, but it hasn't happened yet. But yes, this would be very frustrating.
@@marcogenovesi8570 They paywalled it behind their subscription-based business model. Which is fine for their clients/customers that are okay with that, but the fact that you can't log into the IPMI/BMC without a cloud subscription -- that's just stupid. I mean...I thought that HPE was bad by requiring a license for said IPMI, but to paywall it behind a cloud subscription model -- that's even worse.
Other comment said that their was a selection in the boot log between 1 and 4 which were BIOS and BMC, so that might be it. Change the DIP switch to the other number to try to get BIOS and change the VGA settings and everything.
The BMC has GPIOs and 5 serial ports, one is used by the BMC kernel (you connected to that one) but the other 4 are candidates for the BMC serial connection to the main computer. I would look at the code on the BMC and look for how it controls and monitors the base system: GPIO and/or serial along with the settings should be in there somewhere.
I might suggest contacting a reseller like "the server store" or something similar to see if they have documentation that they would be willing to send you. If nothing else you might get a hard copy.
Hardware locked behind a subscription should be illegal. There is no reason for it. Also it should be mandatory to open source your software/firmware when you end of life a product. If you aren't going to support it let other people do it.
maybe you're gonna have to figure out which chip on the motherboard contains the bios, connect to it with some sort of pin connector, extract the bios rom, and reverse engineer it, or try to use a flasher (like a SOIC16 flash chip) and try to flash a different bios on it (like coreboot), keep the original bios rom in case you need to flash it back (and perhaps brick the board). But since you have no docs to go off of, this will be extremely hard to figure out.
You bought this from the server store. The CPU they pulled from those systems was 7351 They do offer custom configuration if you reach out to them they will get the CPU and cables to you By design and has fuses on their CPUs that vendor lock the CPU to a mobo. Due to this your system may not be posting due to a vendor lock of CPU to another vendor In short if you talk to the server store they can send to known good cpus for this system and the u.2 power cables
I'm actually tempted to buy one of these and a used 64-core Rome CPU just for the (relatively cheap) compute power to replace two nodes of 2x E5-2695 v4 CPUs. But it seems like a gamble with the lack of support and documentation. The difficulty with using it is probably why it's cheap to begin with.
What a great video, sometimes the videos fail, but I am like you Jeff I love to experiment as that is how we learn! I really hope someone in the comments can help you figure this out seems like a ton of compute power to just let go to waste!
I had a 1U server about that long from supermicro, what was weird is that it was a dual node, but it wasnt a blade, it used blade style dual socket boards, but instead of a power finger like a normal blade board, it used a 12VO 24 pin(and no CPU 8 pin) also, it had two of them, and while the manual said not to chain them, you could totally chain 2 boards from one PSU if you are using L series processors rather than X series, but keep an eye on that RAM, if you used ECC RDIMMs rather than UDIMM just 2 sticks could use more power than the L series CPU and with 12 memory slots, you could have ~250w in just RAM(edit on a single board) A rats nest of cables. odd considering how much room was inside this massive chassis with these tiny supermicro X8DTT-IBQF boards. It was also wider than a normal serve rack, so i ended up pulling the boards and PSU's out to put them in some long ITX cases. like the coolermaster 130
Have you tried to put the cpu on the other Blade? Is bios the same? If each motherboard has its own bios maybe settings are different and you can have video signal on the other motherboard
I have a 7601 and a H11 board and it randomly decides to drop a single memory channel as well. It usually returns to 8 channels when I reseat the CPU. Happens a couple of times a year.
So glad ive hust standardized to supermicro. Alrhough i dont currently have a use case for blades ive been curious. I do run dual epyc 7742s in supermicro 2u servers for my cpu mining when im not teaching myself how to build ai/render nodes.
Get a SOIC clip and pull the ROM, for posterity, and also open it up with a BIOS tool to change the default options. Then perform a reset and it will run with the options you set. Also as others have said, throw a vidoe card in it. Waiting for round 2.
I had a server like this before. One think I did note. It will not work with any type of modern monitor and I need to use a old VGA one and once you did this the screen worked and you could see what was going on with the node server. Also type swapping the node around could be a faulty slot.
37" is long for a server. I thought my SuperMicro 6047 72-drive model was bad - but it's only 35". On the other hand, it's a 4u and a LOT heavier - even before I put drives in it!
Probably a custom board with a bios that supports only one or two specific cpu models. Ppl keep mentioning the 7351. Seems like the easiest way forward.
We've seen them opening a package with a Klingon knife, now we see that he also has a Bat'Leth (spelling?). Makes me want to ask him which series he prefers.... DS9 here....I have fond memories of watching it with my mom. A local station with cycle through TOS, TNG, DS9, and first season of VOY before starting all over. Came on sometime after the news went off at 10:30pm .
@@CraftComputing *GRIN* It is the grittiest, the best written, I'm the best acted. It also has the USS Defiant!! They also didn't kill Data off in it!!!!
7:07 do you know that "special" AMD screwdriver that limits torque applied to socket screws is actually made so that users won't strip socket screws, and not to apply the specified force to CPU? I've built more than a dozen of AMD servers using nothing but standard screwdrivers, you just need to bottom out all screws and everything will work fine
Some people online I saw talking about Dell blade servers said their KVM didn't recognize it. Have you tried direct connecting a monitor? Also one other post said something about using the print screen key to switch active blades, not sure if that's a dell thing only or if quanta used it too or if that even matters if each one has its own IO.
I havent had experience with this server but perhaps you might be able to get a copy of the bios to download over the serial connection and edit options in with ami bios tool. But i would worrie about flashing it back as im guessing that its checksumed or something of that nature and turn it into a paper weight.but thats from some guy in his shed.
In addition to trying a GPU card or doing a clip chip reflash of the BIOS, could go old school and try shorting the BIOS nv chip while booting should go into defaults. Once you're visible/in the BIOS you can lift the short and save new settings
@@sebastiantruswell5879 That assumes the bios wasn't already set to PCI graphics, but we don't know if it was or wasn't. Not being set to VGA may mean it had a GPU for AI. We don't know what it was used for, and if they reset the bios before selling it.
@@NdxtremePro So here is my thinking as an ex big 3 server manufacturer R&D tech at 0305AM.... We know that the BIOS is Good and that the BMC is good, and we know that both have been pinned out to reset now this doesn't reset the persistent side of the BMC in many cases, it will remove users, addresses, rack location etc all user inputted info, but can keep the BIOS settings and other settings for the hardware... with it being reset and not defaulting back to the VGA output we know a setting is persistent (two reasons for this and we will get on to this) so we could throw in a GPU into the slot a 5-minute job and then wait for 20 minutes potentially, so a waste of 25 minutes... Bearing mind Geoff is a pretty smart cookie and has been at this for hours.... So the options are 1) Try a GPU - waste of 25 mins based on known variables 2) pay for a licence to the BMC - potentially fruitless as may have been reduced, or no longer available for this server and may even still be registered to the original purchaser so not transferable 3) wipe the BMC with OpenBMC - known serial access to the BMC, new BMC OS wont be vulnerable to Pantsdown and not vendor locked... So in 2009 when I left said company we were looking at cloud provisioning using the BMC to harden on login/registration with cloud services, so as it got internet, the machine was hardened, further to this from looking at this vendor they never fixed vulnerabilities such as Pantsdown so going opensource would be the better bet... so if its not been decommed in the cloud and the service is still active then tough unless you can call the original owners and get them to remove it for you. so flashing a BMC at 30 mins or failing at trying a GPU at 25mins as a consultant I have thought of all the possibilities and presented the one that is time efficient and has a high degree of success, based on past life experience. Looking at the chip and its supporting components it is vendor supplied OS/firmware on pretty generic hardware and with a known Serial connection flashing the BMC is the logical choice to offer as a potential solution.
Were you able to edit files in the serial connection? (I may not have been paying enough attention when you were talking about it) If so, would it be possible to hack the bmc webui and bypass the cloud login part?
howdy. great content. regarding the question you pondered in the beginning : you may have the same allergy I have. I found once I had the first drink, the phenomenon of craving occurred and I became unable to control the amount I drank
So excited to see this vid drop. Been looking at these for a while given current epyc motherboard prices. Shame that they act like scumbags like that. They clearly arent interested in something they cant sell the service for anymore.
Well done 👍 you have brought this to everyone’s attention . I know this type of hardware has a life span 🤔😞 But borking it so it can’t be used. Right to repair is a thing Right to repurpose? Is that a thing? Those chip are awesome 😎
Thanks for sharing! Now I have similar problem with Brocade 1020 network card. I run it on Arch Linux but most of my SFP+ modules are failing on it, even 8G & 10G Brocade modules. So now I am digging and trying to find firmware for that thing :(
Oh God, i found the same one AT AN EVEN BETTER PRICE!!! Man, $299 is tempting even with all the failure. Does anyone know if Quanta vendor locks their cpus?
7:24 Not often you see servers with T-topology that doesnt matter if you use the furthest/closest slot in a channel, good to see. Though in all honesty, unless you're dunning DDR4 4000 it shouldnt matter much anyway even if its not T-topology
Shame this project didn't fan out as planned, hope this one can be solved and doesn't end up as another Brocade. Now then, if you don't mind, I'm gonna go put a few more binges on the cloud gaming series.
The login credentials for the BMC over Serial or SSH are NOT the same as you would use in the Web UI or over IPMI. And Quanta does not require special licensing or "cloud login" to access their BMCs. The default IPMI login is admin/admin, have you tried that?
@@CraftComputingThe server model you have is likely a custom server built by QCT for a large customer, which is why you won't find it on their website. Most of my old QCT servers are still listed on their site, although marked as EOL. Have you tried running a different, lower power CPU. It could be that the BIOS is locked to run specific CPU's or the board only supports a lower maximum TDP than your current CPU. Once you have access an operating system running, you will be able to change the BMC password.
I’m kind of curious if this OOB manages the system similarly to how a Bluefield DPU works, where you can get into the host system via an rshim console. Have you looked for a running rshim service in the BMC environment? If the rshim service is running, then I would imagine that you can use minicom or similar to connect to the rshim device in /dev.
Not sure if it would work but I know some dell boards even when you didnt have the idrac unit in would still support command line IPMI commands. May apply here for getting passed the license issue
So Quanta requires that your IMPI network has access to the internet? That seems like a security risk waiting to happen. I'm not sure if this is standard practice or not, but if I ran a data centre and had the option of IMPI being intranet-only, that's how I'd do it.
Thanks to Maximum Settings Cloud Gaming for sponsoring today's video. Get started with your very own Cloud Gaming VM or Bare Metal Machine at bit.ly/3wxfUuB!
"From knot to tip", that's uncharacteristically saucy. 😳
I can't comment anything useful, it seems UA-cam is auto-deleting it. We shall see if it is going to you for review or if UA-cam decides to magically allow 5-6 comments in a few minutes.
It is annoying when something does not work but I am sure someone has used these products before. Good luck :)
@@KiraSlith nock to tip. The length of an arrow. At least thats what I hope he meant cuz I dont think he has the leeway to measure things in dog D1ks on youtube.
I would pull the watch cell battery to reset all settings, and move JP9 to 2-3 for the recovery bios, also remove all sticks of memory expect 1 to remove timing retaining or bad memory channel. Also double check your VGA cable is still good just encase.
So Quanta will be a fine addition to my list of "never buy or even look at"
Top e-waste provider
One of the largest server OEMs on earth. Think bigger then Dell or HP. They could care less about individual consumers. They sell exclusively to large enterprises and hyperscalers.
Pretty sure I dealt with a desktop type system made by them many years ago. From what I remember, I wasn't impressed with the build quality or their support. I think that it was bought through Costco Wholesale, and probably returned for a refund, after I tried to fix it for the customer. I also think that I might have been the third or fourth tech to attempt to repair it.
total crap. Support drop after this short time...
I have one of their switches. they have the most obtuse manuals ever and are almost impossible to use.
Did you notice the CONSOLE SELECT SW (lower part of the screen at 13:13)? Options are BIOS or BMC. I would guess it's set to BMC right now. Maybe switching to BIOS will give you access to the bios that you want.
Noticed that too. It's worth a shot!
My goodness your right looks like choices are 1 and 4.... Probably a combination of switches to get that value.
lets hope he read your comment
Upvoting everyone so hopefully Jeff sees it!!!
Good eyes. Even after you told me exactly where it was, i didn't see it for a long time.
Put a GPU into each slot. My Asrock AM4 server board has this same issue. You can disable the onboard VGA which connects to the ASPEED BMC. Get a dummy plug into a port on each GPU and I bet it might post
That’s has worked for me with an asrock rack motherboard
"don't use any cpus we don't recommend. also, we're not gonna tell you which cpus we recommend."
"we dn't talk to anybody that does not have an expensive support contract, and we don't say much even to our clients"
Kafkaesque, isn't it?
If you say what your using, and get that reply, it's means I support CPU...
But they do...
This is why my homelab is all Dell PowerEdge. Plug the service tag into their support site and you get ALL the manuals, drivers, and information you need, even the build sheet for your unit's factory configuration. The only things they seem to lock down behind an account are their XC series, but those are just rebadges of other models so you just have to look up the equivalent regular model.
HP almost got it right, but they constantly break their support pages and practically disowned everything made before the split.
Yep, this seems to be the way
In that case I think I'll just buy Dell and just Dell computers only. The toolless designs of the older OptiPlexes and the newer Precision SFF models are my favorite, especially with their blue case lock.
That "dead" memory channel coming back was the ultimate "it works at my place" or "it works for me"...
Nice shirt Jeff! 😉 Looking sharp! That's too bad about the BMC! I hope the community comes through!
Tell your wife 'Good Job' :-D
@@CraftComputing did you read?
@duncandubick7535
vor 23 Stunden
Did you notice the CONSOLE SELECT SW (lower part of the screen at 13:13)? Options are BIOS or BMC. I would guess it's set to BMC right now. Maybe switching to BIOS will give you access to the bios that you want.
This makes me appreciate my old Supermicro servers even more. Standard cables, standard board holes, standard behavior, straight forward BMC (well, as far as bmcs go anyway) everything just works.
At work I mostly deal with Supermicro servers, sometimes Gigabyte and Tyan as well. I much prefer Supermicro out of these three, very pleasent to work with. Thoughtfully built hardware and firmware, I don't know how they compare in other regards like cost, but they sure are nice to work with.
@@titaniummechanism3214 the only complaint I'd have is that I bought their servers rather than their stock shares at this point haha
The Quanta T22HF-1U is a 1U rackmount server manufactured by Quanta Cloud Technology Inc. It supports up to 400GbE networking bandwidth in x16 PCIe 5.0 slots, and has enhanced serviceability with tool-less, hot-swap designs. The server is NEBS compliant for Telco/5G data center deployments, and liquid cooling is supported with up to 385W CPU TDP.
Might need some Verizon/ATT server guys to chime in...
In the video you tried a Naples CPU. Have you tried a Rome or Milan CPU? The firmware ("BIOS") on those systems contain one or more AGESA packages. Each AGESA is for only one generation. I don't know how Quanta does it (I've only worked with Quanta systems with Intel CPUs) but other barebones manufacturers often will have two AGESA in their firmware. e.g. so you could only use a Naples or Rome, unless you reflashed a different supporting version, then you could use a Rome or Milan CPU. When Milan came out, we had the fun time of having to install a Rome just to flash, then remove it and install the final Milan. First gen Epyc boards didn't have enough flash memory to hold an image with both AGESAs, so they were Naples only.
tl;dr: Try a Rome and a Milan CPU to check for compatibility.
I found an excel list for t22hf that listed Rome cpu only
If you archive the stuff on quanta's site now, in a few years you'll be glad you did
Doesn't work internet archive for this website?
Maybe you can do a request on the way back machine do back it up??
@@montecorbit8280host the archive box docker I believe it's called. Be your own archive service
@@montecorbit8280 that's exactly what i was wondering! ill mention that in his discord
Dude likes blades so much his rack features a convenient side mounted blade to defend the internal blades
Batt'leth not included
This is the sort of content I've been missing. "Bought this... and now I'm drinking..." sort of things.
I bought a Quanta Server T42S-2U on ebay and can confirm I feel the same pain :). I was able to install a skylake cpu with ecc and was able to boot the node and use IPMI to reset the bmc password. in my example I was only able to boot once I had both cpu sockets (of one blade) installed with a genuine intel cpu. ES or QS was not working as was only one cpu. If you want to have a look at something on that system let me know I can see what I can do. as for your blade I guess that usb and vga are disabled (which is something a hypervisor would request) so getting into the bios here would be ..... a challenge. As for documentation, the best option would be way back machine. also my system had different names for the blade (which had documentation) and the chassis which had less so try to see what the blade name is and google on that (tip might be a part list of the chassis part number).
tried that, they dont seem to have html links to their files its just a bunch of gibberish in the download link. Was hoping it was possible to just substitute the filename thinking they might just be lazy enough to leave the files there and delete the links. Doesnt seem to be the case. Gonna take someone more skiled than I to poke around. Wayback machine seeems to be a bust too
@@doomboots yeah from what I can find my model Quantaplex T42S-2U is still findable which led me to the fru part list for a motherboard / blade: T42S MB Assy (LBG-1G) but trying the same search this for the T22HF makes me find nothing. It also looks like the quanta site even in 2018 all the way up to 2024 only has 4 node multimode systems, I was also not able to find ANY T22 based models from 2018-now in product slides or support links so think its a hidden or special sku which sucks. I will see if the bmc can change bios settings so at least you can install an os. I will see if I can find the manual from my Quantaplex T42S-2U since I think the blade had an BMC and BIOS reset header on the board.
i have the same Server here, everything working fine. But it was pain in the ass to get it working. To log in to the BMC i had to update the bmc firmware via a windows update tool cli-thing...
I sent mails to their support (germany) and they were very helpful.
There are some usernames/passwords, that i've found on the internet
admin/admin
admin/password
ADMIN/ADMIN
ADMIN/qct.admin
qct.admin/qct.admin
admin/cmb9.admin
hope i can help
Some things:
1. Quanta is a vendor for HyperScaler companies. Meaning Google, Oracle, Microsoft, Meta, etc. This is typical of a quanta machine.
2. Are you absolutely sure these blades support AMD EPYC 7601 CPUs? Those are 180W TDP CPUs and are power hungry.
3. What wattage are those PSUs rated for? Typically, PSUs need to be 2-3x more powerful than CPU TDP (180 x 2 or 180 x 3). Then multiply that times 2. You have two nodes. Have you tried plugging in both PSUs? It's possible it wasn't configured for Single PSU use and machines won't boot unless both PSUs are present.
4. Quanta removes documentation due to the fact they mostly only sell to larger hyperscalers.
5. Do the nodes get IPs for IPMI? You have the BMC yes. But each node ALSO gets it's own IPMI as well. Try looking into that.
6. Do you have any storage plugged in?
7. Remove all RAM except for 1 DIMM. See if that makes a difference.
That's all I can think of honestly. Oh. Last point. You get what you pay for....
Worked on many many Quanta systems, and they are pretty "cost engineered" but also generally fairly solid for what they were meant to do - BUT also very much designed to spec for the company that "engineered" them. (often in lots of 10K+ units min) You won't find any specs at all for the systems I typically used unless you work for the company that initially requested them contract manufactured.
The flipside, is typically they are ALSO typically used so hard by the time they are EOLed there isn't really a lot of life left typically. I know one hyperscaler I worked for, we ran ours at least 50%, and the DCs weren't terribly cold for environmental reasons, so those servers cooked for those 5-7 years.
At this point, they are mostly e-waste, yes. But everything eventually ends up there.
But ya I'd bet its the wrong CPU "family" AGESA FW wise.
180 watts is not a lot for an EPYC.
@@bricefleckenstein9666 true, but 180w for Naples generation is on the high end of the spectrum.
What you accomplished was keeping my attention and doing the tech stuff that I would like to do myself. Thanks for the video!
A bat'leth on your server rack? It's a server blade to go with your blade server.
CLabRetro here on youtube here has a knack for getting into BMCs, but he's usually getting into older servers.
If/when i become God-Queen-Emperor, my first decree will be that beer shall be free. My second decree shall be that lights out management requiring a cloud login shall be punishable by surrender and liquidation of all company assets to charitable causes.
ALL HAIL GOD-QUEEN-EMPEROR RUBYROKS!
HERE, HERE!
State sponsored beer rations? I hope you have a budget plan to pay for that.
I don't like beer much....any chance that would extend to whiskey?!
Or better yet, extend it to Pepsi and Dr Pepper as well!!
@@KiraSlith I have a feeling halving my country's military budget would free up plenty of money for beer rations
To reset the BIOS from the BMC command line on a Quanta server, you can try log in to the BMC's CLI and issue the command reset sp. This will reset the BMC module, which in turn
might reset the BIOS to its default settings
I have a similar purchase that is going slightly better. I found a SuperMicro 2U/4-node system of the X8 generation, so Intel, older, and slower, but each node has 2 sockets, so I actually have 8 sockets in a 2U chassis, but the intriguing thing for me was the boards all included 40Gig Infiniband (10Gig Ethernet) ConnectX-2 network cards. The down side is the onboard storage. No SATA connectors on the board, so no SATA-DOM. No M.2 slots. No SAS. Each node has access to 3 LFF drives on the front backplane and that's it. _But_ with the 40Gig IB, (I already have a Mellanox switch that supports 40gig ETH and IB) I figured I could probably work out some sort of iSCSI solution that would be plenty fast, but that has turned out to be a challenge. Trying to get a modern version of FlexBoot (Mellanox's PXE/iSCSI boot firmware) onto one of these older ConnectX cards (let alone one built onto a server board) has turned out to be a challenge. I'm probably going to abandon that and just boot from an iPXE ISO and let it handle the iSCSI connectivity, but that's a new development that I haven't had a chance to play with much.
I did manage to wreck the pins on one of the CPU sockets, but I fortunately found a pair of motherboards on ebay for a reasonable price. They came with an unexpected surprise... proprietary BIOS. No interesting features, so I tried to flash to the stock BIOS but that was a no-go, so just swapped the BIOS chip from the damaged board and all was well.
Eventually I hope to get XCP-NG running on all of the nodes and then mess around with some of the HA features, but I'll cross that bridge when I get to it.
Im using an old X9 supermicro 1U as a deployment server. Works really great. It has an i3 in it and 3 gigabit nics. two internal and one in the expansion slot.
The two 40mm fans in it is really bad.. supernoisy :) but.. not worth replacing for now.
Goddamnit Jeff, you beat me to this! I was researching this 3 months ago to house some of my extra 7451's. If you need, I ended up getting the Quanta QVL for this from the support team. It only shows support for the 7xx2 series, but it does have other information, so I never ended up buying.
President Skroob : The ship is too big. If I walk, the movie will be over.
Snotty beamed me twice last night. It was wonderful.
@@CraftComputing Hello, President Skroob.
Hello, Charlene.
I'm Marlene.
Hello, Marlene.
I'm Charlene.
Chew your gum!
You could try to use ipmitool remotely to directly change settings with your credentials or to get access to SOL to get access to the BIOS of the board
Yeah that's what I was thinking. Get the system event log
@mer I meant more like the Serial-over-LAN access, but the system event log might help find issues, yup
Serial-over-LAN allows the content of the BIOS to be streamed over SSH(ish) during boot, so you can make BIOS settings without VGA output (and thus potentially enable VGA)
Thanks for informing me that Quanta should be 100% on my avoid list.
I love the effort, sometimes things don't workout but the leason learned is the value gained.
IPMI on boot tried to register into dropbox DNS, that's a clue where this system came from.
If the fans are running full speed like that all the time, chances are its not posted. Re-seat RAM, CPU, and find a bios reset jumper. Or could be a quirk that both blades need to be running in order for it to post?
Always enjoy the videos, success or not so successful, they are all a good watch. I had John Coltrane playing in the background durring the vid. That kinda jazz fits right in to experimental homelab electronics. Cheers
Shoutout for the spongbob cards. little things like that make me lol
1) So I tried looking for the manual -- no cigar.
I did turn up that the model of the motherboard itself is known as "S5H(F) MB".
You see the s5h referenced a bunch of times in your dmesg output and console logs.
2) The other thing that I noticed was that right next to the big red reset button, there's a dip switch (SW1) which says "CONSOLE_SELECT_SW".
The picture, in that area, doesn't have enough resolution for me to be able to read the silk screening on the switch itself (I think that I see a '4', but I can't tell if the other thing is a '10' or something else.)
So, maybe you can try playing with that to see if that might change stuff.
3) The other thing that you might also try is do drop down to only one stick of RAM. That might be a challenge because without the manual, you don't know which is the primary DIMM slot, so there is a chance that you might not get very far with this.
4) This is why I buy Supermicro. Yes, they might be more expensive, but their stuff well documented.
5) I think that there's been discussions about Supermicro paywalling their IPMI behind a subscription, but it hasn't happened yet.
But yes, this would be very frustrating.
even Dell is less of an ahole than this. You can still find manuals for ancient stuff
@@marcogenovesi8570
They paywalled it behind their subscription-based business model.
Which is fine for their clients/customers that are okay with that, but the fact that you can't log into the IPMI/BMC without a cloud subscription -- that's just stupid.
I mean...I thought that HPE was bad by requiring a license for said IPMI, but to paywall it behind a cloud subscription model -- that's even worse.
Other comment said that their was a selection in the boot log between 1 and 4 which were BIOS and BMC, so that might be it. Change the DIP switch to the other number to try to get BIOS and change the VGA settings and everything.
I resemble the remark about being "crazy enough to have a server rack in the garage"
The BMC has GPIOs and 5 serial ports, one is used by the BMC kernel (you connected to that one) but the other 4 are candidates for the BMC serial connection to the main computer. I would look at the code on the BMC and look for how it controls and monitors the base system: GPIO and/or serial along with the settings should be in there somewhere.
I might suggest contacting a reseller like "the server store" or something similar to see if they have documentation that they would be willing to send you. If nothing else you might get a hard copy.
Hardware locked behind a subscription should be illegal. There is no reason for it. Also it should be mandatory to open source your software/firmware when you end of life a product. If you aren't going to support it let other people do it.
I believe open-sourcing firmware or software at end of life should absolutely be mandated by all countries.
My simple thought, which is by no means a solution: use as few RAM slots as possible in the beginning to minimize problems.
maybe you're gonna have to figure out which chip on the motherboard contains the bios, connect to it with some sort of pin connector, extract the bios rom, and reverse engineer it, or try to use a flasher (like a SOIC16 flash chip) and try to flash a different bios on it (like coreboot), keep the original bios rom in case you need to flash it back (and perhaps brick the board). But since you have no docs to go off of, this will be extremely hard to figure out.
You bought this from the server store.
The CPU they pulled from those systems was 7351
They do offer custom configuration if you reach out to them they will get the CPU and cables to you
By design and has fuses on their CPUs that vendor lock the CPU to a mobo. Due to this your system may not be posting due to a vendor lock of CPU to another vendor
In short if you talk to the server store they can send to known good cpus for this system and the u.2 power cables
Have you tried to put in a video card in to see if that would display video out and bypass the disabled VGA port?
Videos like this reminds me why I run Dell.
I'm actually tempted to buy one of these and a used 64-core Rome CPU just for the (relatively cheap) compute power to replace two nodes of 2x E5-2695 v4 CPUs. But it seems like a gamble with the lack of support and documentation. The difficulty with using it is probably why it's cheap to begin with.
What a great video, sometimes the videos fail, but I am like you Jeff I love to experiment as that is how we learn! I really hope someone in the comments can help you figure this out seems like a ton of compute power to just let go to waste!
What was that big red button on the board? The one that reset?
In similar servers, running with just one PSU can stop it booting. Try plugging both in.
I had a 1U server about that long from supermicro, what was weird is that it was a dual node, but it wasnt a blade, it used blade style dual socket boards, but instead of a power finger like a normal blade board, it used a 12VO 24 pin(and no CPU 8 pin) also, it had two of them, and while the manual said not to chain them, you could totally chain 2 boards from one PSU if you are using L series processors rather than X series, but keep an eye on that RAM, if you used ECC RDIMMs rather than UDIMM just 2 sticks could use more power than the L series CPU and with 12 memory slots, you could have ~250w in just RAM(edit on a single board)
A rats nest of cables. odd considering how much room was inside this massive chassis with these tiny supermicro X8DTT-IBQF boards.
It was also wider than a normal serve rack, so i ended up pulling the boards and PSU's out to put them in some long ITX cases. like the coolermaster 130
Have you tried to put the cpu on the other Blade? Is bios the same? If each motherboard has its own bios maybe settings are different and you can have video signal on the other motherboard
I have a 7601 and a H11 board and it randomly decides to drop a single memory channel as well. It usually returns to 8 channels when I reseat the CPU. Happens a couple of times a year.
So glad ive hust standardized to supermicro. Alrhough i dont currently have a use case for blades ive been curious. I do run dual epyc 7742s in supermicro 2u servers for my cpu mining when im not teaching myself how to build ai/render nodes.
Get a SOIC clip and pull the ROM, for posterity, and also open it up with a BIOS tool to change the default options. Then perform a reset and it will run with the options you set. Also as others have said, throw a vidoe card in it. Waiting for round 2.
I had a server like this before.
One think I did note. It will not work with any type of modern monitor and I need to use a old VGA one and once you did this the screen worked and you could see what was going on with the node server.
Also type swapping the node around could be a faulty slot.
Probably already thought of this, but maybe use the Way Back Machine to recover is documents?
37" is long for a server.
I thought my SuperMicro 6047 72-drive model was bad - but it's only 35".
On the other hand, it's a 4u and a LOT heavier - even before I put drives in it!
Probably a custom board with a bios that supports only one or two specific cpu models.
Ppl keep mentioning the 7351. Seems like the easiest way forward.
I really wish they did these in a 2u form factor stacked so I can put them in a narrower rack.
Working with used servers can be a HUGE pita.
*Thumbs Up* for a good Guy - i do same stupid stuff at home, so Fun to watch 🙂
We've seen them opening a package with a Klingon knife, now we see that he also has a Bat'Leth (spelling?). Makes me want to ask him which series he prefers....
DS9 here....I have fond memories of watching it with my mom. A local station with cycle through TOS, TNG, DS9, and first season of VOY before starting all over. Came on sometime after the news went off at 10:30pm .
DS9 is best Trek :-)
DS9 is TNGs mentally disabled cousin.
@@CraftComputing
*GRIN*
It is the grittiest, the best written, I'm the best acted. It also has the USS Defiant!! They also didn't kill Data off in it!!!!
7:07 do you know that "special" AMD screwdriver that limits torque applied to socket screws is actually made so that users won't strip socket screws, and not to apply the specified force to CPU? I've built more than a dozen of AMD servers using nothing but standard screwdrivers, you just need to bottom out all screws and everything will work fine
Try putting a live OS that will boot headless on various media USB, nvme and see if it boots and is visible on the network.
Very interested in the end of this story!
Some people online I saw talking about Dell blade servers said their KVM didn't recognize it. Have you tried direct connecting a monitor? Also one other post said something about using the print screen key to switch active blades, not sure if that's a dell thing only or if quanta used it too or if that even matters if each one has its own IO.
I havent had experience with this server but perhaps you might be able to get a copy of the bios to download over the serial connection and edit options in with ami bios tool. But i would worrie about flashing it back as im guessing that its checksumed or something of that nature and turn it into a paper weight.but thats from some guy in his shed.
In addition to trying a GPU card or doing a clip chip reflash of the BIOS, could go old school and try shorting the BIOS nv chip while booting should go into defaults. Once you're visible/in the BIOS you can lift the short and save new settings
Shove a graphics card in the x16 slot?
will need to enable PCI graphics in the BIOS which he can't get into at the moment...
@@sebastiantruswell5879Unless you try. It wont hurt.
@@sebastiantruswell5879 That assumes the bios wasn't already set to PCI graphics, but we don't know if it was or wasn't. Not being set to VGA may mean it had a GPU for AI. We don't know what it was used for, and if they reset the bios before selling it.
@@NdxtremePro So here is my thinking as an ex big 3 server manufacturer R&D tech at 0305AM....
We know that the BIOS is Good and that the BMC is good, and we know that both have been pinned out to reset now this doesn't reset the persistent side of the BMC in many cases, it will remove users, addresses, rack location etc all user inputted info, but can keep the BIOS settings and other settings for the hardware... with it being reset and not defaulting back to the VGA output we know a setting is persistent (two reasons for this and we will get on to this)
so we could throw in a GPU into the slot a 5-minute job and then wait for 20 minutes potentially, so a waste of 25 minutes... Bearing mind Geoff is a pretty smart cookie and has been at this for hours....
So the options are
1) Try a GPU - waste of 25 mins based on known variables
2) pay for a licence to the BMC - potentially fruitless as may have been reduced, or no longer available for this server and may even still be registered to the original purchaser so not transferable
3) wipe the BMC with OpenBMC - known serial access to the BMC, new BMC OS wont be vulnerable to Pantsdown and not vendor locked...
So in 2009 when I left said company we were looking at cloud provisioning using the BMC to harden on login/registration with cloud services, so as it got internet, the machine was hardened, further to this from looking at this vendor they never fixed vulnerabilities such as Pantsdown so going opensource would be the better bet... so if its not been decommed in the cloud and the service is still active then tough unless you can call the original owners and get them to remove it for you.
so flashing a BMC at 30 mins or failing at trying a GPU at 25mins as a consultant I have thought of all the possibilities and presented the one that is time efficient and has a high degree of success, based on past life experience.
Looking at the chip and its supporting components it is vendor supplied OS/firmware on pretty generic hardware and with a known Serial connection flashing the BMC is the logical choice to offer as a potential solution.
Were you able to edit files in the serial connection? (I may not have been paying enough attention when you were talking about it) If so, would it be possible to hack the bmc webui and bypass the cloud login part?
howdy. great content.
regarding the question you pondered in the beginning : you may have the same allergy I have. I found once I had the first drink, the phenomenon of craving occurred and I became unable to control the amount I drank
Generating E-waste in this way is evil.
Thanks for the info. I'll never buy Quanta. Wow, imagine removing all docs! What are they thinking!
Must admit i'd have a good chuckle if one video You were there chugging a red bull or a milkshake or similar
OK you are logged into the Onboard KVM that switches between the blade servers. So you have not gotten into the Blades.
have you tried booting of sibling socket in case there is some weird topology with video output wired via specific PCIe lanes?
10:14 klingon bat'leth ... just in case .. always be prepared..😎
GLORY TO YOU, AND YOUR RACK!
@@CraftComputing Are you saying that to Lursa or B'ethor? 🙂
So excited to see this vid drop. Been looking at these for a while given current epyc motherboard prices. Shame that they act like scumbags like that. They clearly arent interested in something they cant sell the service for anymore.
Maybe try an EPYC 7352 as those are on the list from the Qanta spreadsheet and have similar performance characteristics and prices as the 7601.
I’ve been looking at these like crazy on the server store… oooo
Have you contacted the eBay seller, might have some info 🤔
Also, try installing PCIE low profile GPU to output video and go into bios settings
You might like to try a pcie video card and get screen output from that but after getting into bios you can set prams for vga output. Good luck!
Well done 👍 you have brought this to everyone’s attention .
I know this type of hardware has a life span 🤔😞
But borking it so it can’t be used.
Right to repair is a thing
Right to repurpose?
Is that a thing?
Those chip are awesome 😎
What did we learn?
Do not buy anything from Quanta.
That server is pretty... EPYC
Oh...is that why AMD named it that! It all makes sense now. 😉
"Sometimes all you need is a trip to another location". I cant count on my hands how many times that has fixed a clients computer or server.
Thanks for sharing!
Now I have similar problem with Brocade 1020 network card. I run it on Arch Linux but most of my SFP+ modules are failing on it, even 8G & 10G Brocade modules. So now I am digging and trying to find firmware for that thing :(
Your tech mojo is Jedi level, when a Padawan is ready a master will be found. To begin my training, I will subscribe and ring the the bell 🧐.
This is the way
Oh God, i found the same one AT AN EVEN BETTER PRICE!!! Man, $299 is tempting even with all the failure. Does anyone know if Quanta vendor locks their cpus?
3:13 That was a faaar fetched reference. One could call it a pizza plate. For the hut
7:24 Not often you see servers with T-topology that doesnt matter if you use the furthest/closest slot in a channel, good to see.
Though in all honesty, unless you're dunning DDR4 4000 it shouldnt matter much anyway even if its not T-topology
Craft Computing :
Come for the server content
Stay because you need to defend your honor and need a Bat'leth
On the bright side, the fans work.
You made a slight error, SP3 supports Naples, Rome, and Milan it just depends on whether the BIOS on that server will take all of them.
Shame this project didn't fan out as planned, hope this one can be solved and doesn't end up as another Brocade.
Now then, if you don't mind, I'm gonna go put a few more binges on the cloud gaming series.
The Brocade problem was EASY to solve though.
This one is like fist-fighting a ghost...
The login credentials for the BMC over Serial or SSH are NOT the same as you would use in the Web UI or over IPMI. And Quanta does not require special licensing or "cloud login" to access their BMCs. The default IPMI login is admin/admin, have you tried that?
Of course I've tried admin/admin ; ADMIN/ADMIN ; root/admin ; any other combo I could find.
@@CraftComputingThe server model you have is likely a custom server built by QCT for a large customer, which is why you won't find it on their website. Most of my old QCT servers are still listed on their site, although marked as EOL.
Have you tried running a different, lower power CPU. It could be that the BIOS is locked to run specific CPU's or the board only supports a lower maximum TDP than your current CPU.
Once you have access an operating system running, you will be able to change the BMC password.
Love these kind of videos!
@2:40 - If you end up having any issues with the x11 board, my money will be on a slightly warped chassis.
Percussive maintenance achieved
I’m kind of curious if this OOB manages the system similarly to how a Bluefield DPU works, where you can get into the host system via an rshim console. Have you looked for a running rshim service in the BMC environment?
If the rshim service is running, then I would imagine that you can use minicom or similar to connect to the rshim device in /dev.
I would try to install a video card in one of the PCI slots since it's a new device It wouldn't be disabled in the BIOS.
Not sure if it would work but I know some dell boards even when you didnt have the idrac unit in would still support command line IPMI commands. May apply here for getting passed the license issue
So Quanta requires that your IMPI network has access to the internet? That seems like a security risk waiting to happen. I'm not sure if this is standard practice or not, but if I ran a data centre and had the option of IMPI being intranet-only, that's how I'd do it.