In the quest to increase power utilization efficiency and to achieve the computation of more data in smaller spaces, the computing industry investigated alternative solutions to forced air-cooling. Cold wall and baseplate cooling methods, helped by liquid or gas exchangers have been used for decades and from a laptop, datacenters and on to radio base stations, a well-established technology to extract calories from dissipating components has been achieved. The technology worked well, but to jump from 40kW per rack to 250kW and more, even that technology reached its limits.
Figure 1 – Mineral oil cooled PC ▶
(Picture: PRBX / Courtesy of Puget System)
How to get more computing power from a datacenter with safety and efficiency has been the concern of many engineers, and the idea to get the full benefit of liquid cooling by immersing heavy computing systems into fluid became an interesting option. After more than 10 years of experimentation, business cases and trials, where does that industry stand in 2021 and how will power supplies adapt and develop to accommodate that technology?
From fish-tank to super high-density datacenter
If you are a fan of online gaming requiring huge levels of computing power, you may remember PC conventions where geeks presented oil immersed computers in a fish-tank (figure 1). Anecdotal as it may seem, beginning in 2005, the idea to benefit from deep liquid cooling technology has been explored by the gaming community but the biggest interest for that technology emerged from Bitcoin mining requiring massive computing power.
At the origin of Bitcoin mining, many companies took advantage of the cold Nordic environment and the locally produced renewable hydroelectric energy to setup datacenters. Nordic countries began many projects to support those initiatives. One example is the Swedish project, ‘The Node Pole’ promoting an abundance of stable and competitively priced electricity from renewable energy, inviting datacenter operators to benefit from this specific environment.
Figure 2 – KnC Miner 10 megawatt data center filled with high-powered computers mining for cryptocurrency in Boden (Sweden) (Picture: PRBX / KnC Miner-2014)
Many leading companies launched datacenters in Nordic countries e.g. in Boden, Sweden. We could mention the Bitcoin company KnC Miner, who in 2014 opened a 10-megawatt data center filled with high-powered computers mining for cryptocurrency, capitalizing on the benefits of hydroelectric power and natural cooling. Although the source of energy powering a Bitcoin farm was renewable, nonetheless the energy dissipated was lost and concern was emerging regarding ‘Energy Utilization’. Many Bitcoin mining datacenters all over the world were operating in huge halls, with thousands of computing units cooled by forced air, without any heat recycling (figure 2).
If Bitcoin mining centers operating in Nordic conditions could ‘get by’ using forced air cooling, the methodology was definitely not a long-term solution, and where massive computing units operating in the rest of the world – and not benefiting from natural cold air – it was not a solution at all. In any case, considering the environmental aspect and impact, wasting energy became a major concern and even in Nordic countries local communities placed high demand levels on datacenters to improve Power Usage Effectiveness (PUE) and to optimize and re-use calories produced during the computing process, for example to heat water for public usage.
Figure 3 – Allied Control 2-phase Immersion Cooling – (Picture: PRBX / Courtesy of LiquidStack)
Besides Bitcoin mining, the growing demand for mass computing architecture for simulations and future networks of autonomous vehicles motivated datacenter operators to consider alternative methods to deliver extremely high computing power in smaller spaces with a PUE close to ONE. The idea to immerse the heavily computing parts of datacenters in fluid grew within the engineering community, with functional systems being tested in 2010.
The road for immersion cooling was opened!
When Bitcoin meets Big Data
We could name many experiments performed all over the world to design immersed, high computing power machines in fluids, but it’s worth mentioning the 1.4MW container data center and its 240kW flat racks launched by the Hong-Kong based company Allied Control (now LiquidStack) and rewarded with the ‘Best Green ICT Award’ in 2014 (figure 3).
From the first generation launched in 2012 to the third generation launched in 2015, cooperating with 3M under a project named 2PIC (2-phase Immersion Cooling), Allied Control increased the Total Watts Per Square Foot from 0.25kW to 3.23kW while maintaining a Power Usage Effectiveness (PUE) of 1.02. This has been made possible by optimizing the immersion cooling technology with the 3M coolant Novec 7100.
Figure 4 – Comparison of air cooling vs. fluid cooling performances (Picture: PRBX / Courtesy of 3M)
Presented at many conferences, e.g. Open Compute Project Summit (OCP Summit), immersion cooling offers unprecedented benefits in terms of performance. As shown in figure 4, the power density per rack can increase from 40kW to 250kW (and more), the computing power from 10kW to 100kW per square meter, and the energy used for cooling to reduce from 2.0 pPUE (partial Power Usage Effectiveness) to below 1.02 pPUE.
In addition to improving performance, immersion cooling is also considered by datacenter managers as a possible solution to reduce the risk of fire. We are all able to recall the fire at the French OVH datacenter in Strasbourg and the collateral damage that impacted their customers. Despite all caution and measures in place to prevent a fire, the increased power density of existing datacenters cooled by conventional methods remains a high concern for datacenter managers.
The dielectric coolants used in immersed datacenters have a dielectric strength that’s thousands of times higher than air, so even if there’s a short circuit in the coolant, there’s no spark or ignition, which clearly greatly reduces the risk of fire. As well, immersed data centers are using a very limited number of fans, mainly used outside the computing environment in the heat exchanger.
All considered, immersed datacenters have lots of benefits and following on from Google, Alibaba and many others, the recent announcement from Microsoft to use two-phase immersion cooling in its Quincy, Washington Azure datacenter confirms the demand for Big Data computation taking over from the original Bitcoin experimentation phase.
Making datacenters more powerful and better in Power Usage Effectiveness is great but is cooling working and what will that mean for power supplies manufacturers?
Figure 5a – Single-phase Liquid Immersion Cooling (SLIC) basic principle (Picture: Powerbox (PRBX))
SLIC and TLIC!
Two technologies are commonly used for immersion cooling: Single-phase Liquid Immersion Cooling (SLIC) and Two-phase Liquid Immersion Cooling (TLIC). Both technologies make it possible to achieve more than 200kW per rack with impressive PUE. The decision to use one or another technology depends on operational conditions and best practices as applied to specific industries. A lot of literature has been published on both, but in simple terms this is how it works:
Figure 5b – SLIC DUG immersion-cooled data hall in Houston hosting more than 40,000 servers (Picture PRBX/ Courtesy of DUG Technology)
Single-phase Liquid Immersion Cooling (SLIC)
Single-phase immersion cooling (figure 5a) servers are usually installed vertically in the container and cooled by a hydrocarbon-based dielectric fluid that’s similar to mineral oil, as was used by the gaming geek in the early days. The heat is transferred from the dissipating components to the coolant, which is then cooled via a heat exchanger in a cooling distribution unit (CDU). Single phase is very simple to operate and maintain. Beside hyperscale datacenters (figure 5b), SLIC is the preferred solution for industrial computing systems operating in harsh environments requiring a very high level of safety
Two-phase Liquid Immersion Cooling (TLIC)
In a two-phase immersion cooled system (figure 6a), servers are immersed inside a bath of specially engineered fluorocarbon-based liquid. Because the fluid has a low boiling point (often below 50 degrees C vs. 100 degrees C for water), heat from the servers easily boils the surrounding fluid. The boiling of the liquid causes a change of state from liquid to gas, thus giving two-phase immersion cooling its name. The vapor is then condensed back to the liquid form via water-cooled condenser coils that are integrated into the top of the sealed racks. The condensed liquid drips back into the bath of fluid to be recycled through the system (figure 6b).
Power to Immersion
The vast majority of datacenter equipments are powered by a front-end rectifier converting the AC voltage to 48V DC. Some are using a High Voltage DC (HVDC) distribution (e.g. 400 VDC). In the case of immersed equipment, the power supplies are often outside the tank and power supplies are off the shelf, however a number of highly integrated, high-density computing units are integrating the complete power solution.
Figure 6a – Two-phase Liquid Immersion Cooling (TLIC) basic principle (Picture: Powerbox (PRBX))
Originally used in harsh environments where safety is important and cooling is complicated, immersed electronics has been in practice for many years. With the high demand for small networks with high computing capacity, the development of a new generation of highly integrated, immersed servers began, including the AC/DC power supply (figure 7).
Although most of the power supply components are compatible with the different coolants used in SLIC and TLIC, power designers must carefully select electrolytic capacitors. Electrolytic capacitors are designed to sustain humidity, however their sealing capsule properties can be affected when permanently immersed. Because operating such a capacitor in an immersed conditions may be outside its normal specifications, it is important to simulate, test and verify capacitors’ life time when immersed for deployment as such.
Figure 6b – TLIC servers at Microsoft Quincy, Washington datacenter (Picture: PRBX / Courtesy of MICROSOFT)
Another important parameter to take into consideration is thermal conditions, which in the case of a liquid coolant is very much different to that when operating in air conditions. In both cases, by circulation in the case of SLIC, and evaporation in the case of TLIC, the calories are evacuated from dissipating components much faster than in air. In some components such positive temperature coefficients (PTC) are temperature-dependent and in the case of immersed applications the gradient between low and high temperatures is much lower. Power designers must take this into consideration.
Coolants have high dielectric properties and there is no problem in operating high voltage switching topologies in immersed power supplies, although it is important to maintain a high physical isolation e.g. the use of a conformal coating to prevent corrosion from electrolyte effect that might happen when immersed in fluid.
Most of the layout principles used in air-cooling apply in immersed applications but it is important to make sure the fluid circulation is optimized within the power supply.
Last but not least, when operated in immersed conditions temperature measurement can be a challenge. Although conventional temperature sensors are often used with specific coupling to the dissipating components, other types of monitoring by digital measurement are often used instead. Also, techniques such as ripple and noise envelope analysis help to monitor the overall performance and to apply preventative maintenance when parameters are outside critical limits.
What’s the next step?
Figure 7 – PRBX AC/DC power supply OFI600A-12 designed for immersion cooling applications (Picture: Powerbox (PRBX))
What was started by gaming geeks immersing their computers in fish-tanks, and now to hyperscale massive computing datacenters, the immersed computing world is aimed to grow fast. The outbreak of COVID-19 is boosting Datacenter service demand and also new technologies such as autonomous vehicles, and 5G in its early stages. SLIC and TLIC will continue to improve performance levels, as will power supplies manufacturers working on highly efficient topologies using Wide Band Gap semiconductors (SiC and GaN). After 10 years of experimentation and local initiatives, the world of immersed computing is opening up nicely.
Power designers love a challenge and developing multi kilowatts immersed power supplies is a challenge they cannot resist.
Author: Patrick Le Fèvre,
PRBX, Chief Marketing and Communication Officer
About the author:
Chief Marketing and Communications Officer for Powerbox, Patrick Le Fèvre is an experienced, senior marketer and degree-qualified engineer with a 35-year track record of success in power electronics. He has pioneered the marketing of new technologies such as digital power and technical initiatives to reduce energy consumption. Le Fèvre has written and presented numerous white papers and articles at the world’s leading international power electronics conferences. These have been published over 350 times in media throughout the world. He is also involved in several environmental forums, sharing his expertise and knowledge of clean energy.
Powerbox – A Cosel Group Company| https://www.prbx.com
References:
The Node Pole: https://www.nodepole.com/
LiquidStack (Allied Control): https://liquidstack.com/
3M Novec: https://www.3m.com/3M/en_US/novec-us/
DUG Technology : https://dug.com/dug-cool/
Microsoft : https://news.microsoft.com/innovation-stories/datacenter-liquid-cooling/
Powerbox (PRBX): https://www.prbx.com/2019/11/powerbox-announces-industrys-first-power-supply-for-immerged-computing-applications/