Hetzner's DIY Datacenter Tour — Full Breakdown & Transcript

[00:02] Falkenstein is going to build me a server and put it online in, and I quote, "a few minutes." Okay, that sounds pretty far-fetched, but that's exactly what Hetzner's business is, and they sponsored us to come all the way

[00:15] here in Germany so they could show me what I would describe as the most professionally DIY facility that I think I have ever seen. Seriously though, this place is incredible. If you can do it yourself, they do it. I'm talking

[00:30] building their own HVAC systems, building their own custom server racks and enclosures. They even designed their own unique steep-roofed DC buildings, and they're working on a DIY solar deployment that would transition them

[00:43] from already 100% renewable energy to even cleaner renewable energy. And the best part is they gave me an unprecedented level of access to just like climb the exhaust structures, explore under the false floor. I even

[00:57] built an actual server on the production floor that will be deployed to one of floor that will be deployed to one of their customers in just a few minutes if their customers in just a few minutes if all of this goes according to plan.

[01:18] place, like Gimmel Stomp, but I assure you guys, it's real and it is absolutely you guys, it's real and it is absolutely beautiful. But that is not why Hetzner decided to build Germany's largest data center here. That comes down to the

[01:32] climate. To keep their costs, their water consumption, and their energy consumption low, Hetzner relies heavily on air cooling, and the cooler the air, Hence the decision to build their other

[01:45] data center in Helsinki, Finland. This desire for maximum efficiency drives pretty much every decision around here. Like if I saw something weird or unusual, the [music] answer invariably was, "Well, it's very efficient that

[01:58] way. Take for instance the layout of their buildings. Most DCs build up using multi-story structures with complicated heat management systems. Here, land is cheap. So, Hetzner instead is building out, and they devised an unusual

[02:12] shed-style roof for their modular buildings that takes advantage of natural convection in order to more efficiently move heat. Here, let's go inside so I can show you guys. Under my feet right now is pretty standard data

[02:25] center stuff. A raised floor with chilled air that is forced into the cold aisles and then exhausted through customer systems, keeping the systems cool and the interior of the DC at a consistent 22° C.

[02:39] These filtered HVAC systems use Hetzner's own design to cool the incoming air as needed. So far, so pretty standard. But, here's where things get interesting. You can really feel the cooling up here,

[02:52] or rather the heat. The shape of this roof allows natural convection to carry roof allows natural convection to carry the hottest air in the DC all the way up and over here where it gets exhausted directly outside. I got to say that is

[03:07] something that I have never seen before. A DC that is just open to outside air. But, apparently they never have issues with rain or snow getting in because they keep the data center at a positive internal pressure using powerful turbine

[03:21] fans. Now, let's head down and take a look at some of the system racks. See, in most DCs, you would find rows and rows of standard 19-in racks. Hetzner does offer those for folks who want, say, a virtualized cloud server,

[03:35] but the vast majority of what we saw during our visit used Hetzner's own custom bare metal solutions that offer up some key advantages, with the biggest being flexibility and cost. We're going to take a closer look at some of these

[03:48] in a second, but first, some history. Hetzner realized many years ago that consumer grade hardware was getting powerful enough to be used for cloud applications and reliable enough to expect a 7 to 8 year lifespan or

[04:02] >> even more. And all of this at a fraction of the cost of enterprise solutions. So, by passing those savings along to their customers, they created a really cool niche for themselves. Now, their older systems actually use tower cases on

[04:16] standard warehouse shelving, but over time they asked themselves, "Do we need cases again?" And that is how the ES or easy stack system was born. Basically, what you're looking at here are motherboard trays on little metal

[04:30] sliders that have a bunch of modular parts that can be bolted onto them depending on the customer's needs. Do you want a couple of hard drives? Cool. SSDs? Ditto. Oh, wait. You need way more hard

[04:45] drives? Well, they have a dedicated hard drive shelf that will double your rack space consumption, but allows for up to 15 hard drives off of a single system. So, you could, say for instance, build your own cloud backup. Now, obviously,

[04:59] you don't get the kind of density from this that you might get from one U or from blade servers, but again, here comes the E word. Hetzner found that for their customers, the efficiency of being able to cool their systems with a single

[05:13] standard case fan versus like a bunch of high-powered 40 mm screamer fans outweighed the benefit of more density. And the power consumption of a fan might sound like a insignificant thing, but to give you some idea how obsessive these

[05:27] guys are, they even disable motherboard RGB because across hundreds of thousands of systems, every single watt counts. One of the most striking things about this place is the total mishmash of different gear in here. Like, some of it

[05:40] is frankly kind of entry-level. Like, this is just a consumer Ryzen system with this cute little what they call a reset board to help with remote management. See, consumer boards don't have IPMI for, you know, network access

[05:55] to the BIOS. So, this is a way for customers to hard reset or power down a come and do it every time. And then meanwhile, two racks under it, we've got an Ampere ARM-powered server, and then over here, an Epic server on a proper

[06:11] enterprise motherboard with like high-capacity NVMe SSDs. It's a total mix. Sometimes we see large blocks of the same system that were clearly all ordered together, but not that often, and that's just due to the

[06:24] nature of Hetzner's business, where they accommodate many smaller customers who might order a single system today, three more the month next, and then another 10 a year later. That means they need to stay nimble, and their modular approach

[06:37] lets them do just that. If there's ever a problem with your system, every DC has these little workstations kind of sprinkled about, where you can speak directly with a technician who can repair your system either using parts

[06:50] that are on hand, or just if it's more challenging issue, they'll just replace all the memory from the memory shortage. Actually, it's kind of nice that there's permissiveness and flexibility extends to networking as well. So, at the top of

[07:06] every rack, you can see that there's a switch. So far, so normal. But, while many network configurations will use VLANs to help customers create virtual networks of their systems, Hetzner goes a step farther and actually lets you

[07:20] just manage the top of rack switch, and then hardwire all of your own systems together on their own private physical network, if you want to. So, in a nutshell, the vibe I'm getting from this place is, well, what I really

[07:33] service hosting and cloud infrastructure, but man, wouldn't it be nice to do that in a place that also has data center grade redundant cooling and power. That's Hetzner, baby. Speaking of, why don't we actually head over and

[07:47] our way over there, I did promise that we could go under the floor and now seems like as good a time as any, right? That's chilly. Ah.

[08:02] I don't think I've ever done this before. Now, Hetzner is expanding pretty modular construction is they want to be customers are going to put in their new building. So, everything around me right

[08:18] now is just like cold air. And so, if they want to create a new cold aisle, all they do is just lift up some floor tiles like we just did, replace those with grills, and then throw up some racks and an enclosure around me.

[08:32] And all the building materials they need to do all of that are here on site. See, this is what I mean by like DIY professional. And I get the same kind of vibes in here. This is their battery backup.

[08:47] Like, to be clear, I'm not saying it's not professional quality. If utility power fails, these lead-acid batteries are going to kick in immediately and can run the building that we're in for about 10 to 15 minutes until the diesel

[08:59] generator outside can fire up. And that has enough fuel on site to run for several more days before they'll need to truck in more. And these batteries are kind of a really sensible choice. They're good for 10 to 15 years, which

[09:13] makes this a low maintenance and economical solution. It's just not what Same goes for these. Normally, when I see an evaporative cooling tower, it's literally that. It's It's a tower. It's enormous. But, these are just like

[09:28] practical little heat exchangers. And the reason they can get away with that is that it's only about four or five days out of the year that Falkenstein gets too hot. So, they've got these little banks of evaporative coolers, and

[09:40] then they build them on the north side of the building in the shadow of like the tall end of the roof here. Again, to help them operate more efficiently by Now, I think it's time for me to get out of the sun and see where the servers are

[09:55] of the sun and see where the servers are built. Hetzner builds over 1,000 servers per week here on site. And when you're dealing with that kind of volume, you've got to consider, you guessed it, efficiency. And I love the way this

[10:07] place is laid out. The assembly area sits right in their main shipping and have their main troubleshooting and diagnostic stations here, and okay, this is really cool. They have such an enormous volume of hardware to test

[10:21] around here that they actually created their own automated hard drive testing platforms. All you do is chuck a drive in, it turns purple to indicate that it's testing, and then when it's done, green means good, and red means

[10:37] less good. Hey, I can read that. Decommissioning of older servers happens and then right across from there is all the hardware that they need to build their standard systems. So, new orders come in on the screens over there, then

[10:52] the parts get picked into a bin, put on this conveyor, and a technician grabs it and goes to town. Speaking of, you guys said that I could build a real good on that, right? >> Sure, but you have to do it the German

[11:05] way. >> What's the German way? Oh, I see. Well, let's get to work.

[11:17] no instructions, but uh this is my chassis, right? Well, it's no LTT screwdriver, but I suppose it'll do. Okay, honest question, honest question. Wera quality tools, no hate. Are German people's hands shaped

[11:29] like this? Like, whose hands are these handles made for? Anyone? German people? begin with a quick inventory. We've got our motherboard, we've got our SSD, RAM, [music] and CPU, FSP power supply, we got a couple of cooling fans. This I

[11:44] assume [music] is a heat sink. You better believe it. And okay, this is actually pretty cool. Hetzner used to order shrouds like this, whether [music] third-party suppliers, and then they realized after a while, why are we doing

[11:56] this when we can just 3D print them? It doesn't necessarily save that much in terms of cost, but what it does do is it gives them way better flexibility something on demand, and if they need something tweaked, they just ask the

[12:10] buys back at HQ, hey, could [music] you, you know, fix that? And it's done. Oh, underesti- underestimated [music] the size of my Canadian butt. >> You can adjust the behind. >> You Oh, you can adjust it.

[12:23] Well, I didn't. Most of this I expect to be pretty intuitive, you know, like be pretty intuitive, you know, like building a normal system. There we go. Cooltra 265. Wow, you guys got RAM? 64 gigs.

[12:38] Must be nice to ball however hard this guy does. us redundancy in the event of a failure, which there probably won't be, but you screws are here. So, M.2 assembly screw? Actually,

[12:56] that LTT stubby screwdriver. It's just it's magnetic, so All right, I need to try and figure out what the deal is with these. These are going to go somewhere. Do you have a reference server I can look at? Oh yeah, there we

[13:10] go. Oh, I was really close. Okay. Yeah, so that just goes in there kind of like exactly how I did it. Nice. Ooh, I like that. That's a pretty satisfying little snap. Aw, cute. This

[13:24] gives us SSD cooling. I wonder who this is for. Hi Hetzner me build an actual production system. It's amazing. You know, I've asked to do stuff like this many times during tours and they've never let me build an actual

[13:37] prod product. Micron let me build my own RAM kit, but >> [music] >> and they like explicitly forbade me from

[13:49] ever giving it to anyone else or letting anybody else use it. Um [music] whereas this time it's going to actually go onto a rack. This is sick. Oh, this is cute. So no screws to assemble their like test bench shelf things. They've just got

[14:05] got these little holes and it just kind of like sits in there. Now this is interesting. You guys have your motherboard standoffs standardized and they're all out, but I noticed that there's some extras

[14:21] that I do not want. Is there something to cover them or do I >> Bend them out of the way? >> Yeah. >> Just with a screwdriver and just >> Just just go for it. Okay. Let's get rid

[14:34] of this one. Oh, well that was easy. I don't even need a screwdriver for that. Don't need you. Are these threaded? They're not.

[14:47] We're self-tapping. Efficient. I have never used self-tapping screws to mount a motherboard before. So congratulations Hetzner. This is a first for me. really wanted to do. Makes me a little uncomfortable. But you talk about

[15:03] efficiency. Who wants to futz about with brass threaded standoffs coming out of just generally being a pain in the butt. This is great. I could build so many servers with self-tapping screws.

[15:15] assume this sticker goes somewhere. protractor? No, I'm kidding. Okay, there we go. We're going to call that good enough. Cable management, not efficient. So, we're just going to go ham.

[15:28] >> Cuz we need to grab the power cable, and it needs to go in before this cooling okay. I made my first mistake. There's a little bit of cable management that matters. Hold on. This

[15:41] has to go through a grommet like this, and it has to run through here so that we can get all the power run to the front. There we go. All right, now our power supply can be

[15:54] connected. But wait, there's more. There's airflow management foams that go under the motherboard to keep the air from accidentally being sucked under the guess we're going back to the beginning. Okay.

[16:07] And then a T right there. >> Yeah. >> Okay. All right, brilliant. Oh, dude. We've got it this time. You we found the longer self-tappers that are for these little SSD cooling fans.

[16:27] "Nah, I should be fine." Do you have the concept of hubris in Germany? Oh, wait. No, it was right before. Basically, this goes in here.

[16:39] Okay, there we go. And then we actually just use a zip tie to mount it. Efficient. Apparently, I was supposed to leave the elastic on the cables. This is going to be an exhaust. So, we're going to do it right like that.

[16:55] cooling shroud on, but not quite. We've still got to hook up that front reset switch thing, which Oh, okay. Just comes like this, and I guess I just give it the old

[17:07] This is the part where I hopefully don't break this tab, because then everything get a new one of these pieces. This has got to bend this way, right? Yeah, Okay, got to bend this way, right? Yeah, Okay, this has got to bend this way.

[17:31] goes like that. All right. Okay. We can dance when we want to. Uh, I'm going to to plug this in now. I'm here all week.

[17:45] >> I mean, we're over 30 minutes, by the way. >> Okay, listen, listen. Okay, what header is this plugging into? Oh, no. I cannot tell. Okay, what that tells me is it's one under here. So,

[17:57] that's self-explanatory. Goes here. Okay. Get that there. Oh, nice. Oh, dude, quality. Tools away, like a German would do. Now, we just got to figure out how the devil this thing goes on. So, the cables go up through here

[18:12] getting a longer cable. Thank you. Go around it. And then Okay, fair enough. And then this sits around the RAM. All right, that seemed like a good sound for a computer to make. There you go. Nice. So, you got a little bit of air

[18:26] intake over your RAM sticks here, but the vast majority of it is pulled through the heat sink. And that's it. That's all the cooling for the system. aisle. So, just a tiny bit of air flow from these little 40 mm fans is going to

[18:39] be plenty for our SSD. And then the last bit is to keep that shroud in place. There's a couple of cute clever little tabs here. Come here, you cute clever little tab. Look at that.

[18:54] on. Now, it's time for QC. Those are just loose. Just the way I did them. That's it. Pencils down. I I think this is built now. All right, there it is. Some random customer's getting this almost as good

[19:07] as a regular Head Snapper built system. >> In 36 minutes. bad. Those look pretty much the same, right?

[19:22] >> All right. Where are we going next? system, the one I built today is not the same as the ones that they'll be

[19:38] building here in a month or especially in a year. In the DC space, the R&D never ends. So, let's have a look at the test rack, which Okay, it's actually more of like a whole end of one of the DCs, but the test rack is part of it. In

[19:53] this rack is pretty much any standard config that you saw somewhere else today. And they need one here in order to test new firmware updates or software anything more broadly if they roll it out. It also contains experimental

[20:08] systems that might be deployed in the future. For example, Headwater doesn't future. For example, Headwater doesn't do much in the way of multi-GPU stuff yet, but a customer asked them to experiment with building an eight-GPU

[20:21] machine for them. So, they've actually got two different versions of what that might look like here. One that uses a PCIe switch and one that doesn't. And they're just going to kind of roll with it and see which one makes more sense.

[20:34] In the nearby top-secret experiment room, you'll find all kinds of fun goodies. The hot box here is for stress testing and it's packed with experimental cooling solutions. Oh, hey, I know this one.

[20:48] Leak shield. These guys have like negative pressure on the loop or something like that so that liquid can't come out if it accidentally springs a leak or whatever. Now, Headwater doesn't really do

[21:00] anything with liquid cooling yet, but as power density keeps increasing, I'm sure So, next time I come to visit, there's going to be all kinds of new stuff, >> Yay! >> Yeah, of course.

[21:13] fun. And if you guys enjoyed it, maybe check out the tour we did of a more conventional supercomputer build at Simon Fraser University. It uses a very different approach, and I think shows

[21:25] one-size-fits-all [music] when it comes to connected computing. bringing us all the way out here, and for sponsoring this video. You can learn colocation options at the link down

[21:39] colocation options at the link down below.