\n
<\/p>\n
<\/p>\n Follow us<\/p>\n <\/p>\n \nAdd us as a preferred source on Google<\/span>\n<\/p>\n <\/p>\n <\/span> Newsletter<\/p>\n <\/p>\n \nSubscribe to our newsletter<\/span>\n<\/p>\n <\/p>\n <\/span> Google\u2019s tensor processing units (TPUs) \u2014 its custom AI chips \u2014 now serve AI in production at Meta (creator of the widely used PyTorch AI framework) through a new PyTorch-native version of the SGLang inference engine, announced on Google\u2019s developer blog. A matching version of vLLM, another widely used inference engine, was also announced as open source.<\/p>\n Previously, both engines used Google\u2019s own JAX AI framework to run on TPUs. The new versions allow the hardware to be a native option for PyTorch, mostly without rewriting.<\/p>\n Go deeper with TH Premium: AI and data centers<\/p>\n \n (Image credit: Microsoft)<\/span><\/p>\n \u201cSGLang-TorchTPU is running in production at Meta today\u201d after \u201ca close three-way collaboration between Meta, Google, and the SGLang team,\u201d said Meta\u2019s PyTorch-TPU lead Zongwei Zhou. That team works at RadixArk, a startup founded by SGLang\u2019s creators, which built the engine with Google and Meta. The engine, he said, gives Meta \u201ca remarkably mature, PyTorch-native serving stack.\u201d<\/p>\n Latest Videos From<\/span>Tom’s Hardware<\/span><\/span><\/p>\n Jana van Greunen, director of PyTorch engineering at Meta, added in Google\u2019s TorchTPU post that Meta \u201ccannot be constrained by a single hardware architecture\u201d and that TorchTPU \u201cenables us to deploy our PyTorch workloads to Google TPUs.\u201d Google said it co-developed TorchTPU with Meta\u2019s PyTorch team.<\/p>\n \n (Image credit: Google)<\/span><\/p>\n This is not Meta\u2019s first attempt at using TPUs. Google calls PyTorch\/XLA, the older bridge from PyTorch to TPUs, a “predecessor” to the new TorchTPU, and SemiAnalysis reported last year that Meta research teams used TPUs through this bridge from 2020 to 2023 before Meta canceled those contracts. Google\u2019s PyTorch-on-TPU push was aimed mainly at Meta, which did not want to migrate to JAX.<\/p>\n Although it is unclear where Meta\u2019s production TPUs sit, talks last year reportedly included Google Cloud TPU rentals in 2026 with purchases in 2027, and Meta later reportedly signed a multi-year rental agreement.<\/p>\n BREAKING NEWS\ud83d\udea8\ud83d\udea8: TODAY META CONFIRMS THEY ARE RUNNING TPU INFERENCE IN PRODUCTION. What does this mean for the CUDA moat? The announcement came one day after Amazon said it was open-sourcing TorchNeuron, its own native PyTorch backend for its Trainium2 and Trainium3 chips, at the same PyTorch Conference. Both moves appear to be aimed at loosening Nvidia\u2019s software lock-in.<\/p>\n On performance, a tuned vLLM setup serving Moonshot AI\u2019s open-weight, 2.8-trillion-parameter Kimi K3 model reportedly tops 700 tokens per second for a single user on the seventh-generation Ironwood TPU. That speed figure comes from Inferact, the startup founded by vLLM\u2019s creators that helped Google build vLLM-TorchTPU. Its test ran 16 Ironwood chips with speculative decoding, a method of speeding up large language model inference with a small draft model, against vLLM\u2019s published setup on 16 Nvidia GB200s, which reached 452 tokens per second. Its kernels for Kimi K3 and Qwen 3.8 27B deliver roughly 1.4 to 2 times the GB200 baseline\u2019s throughput at small batch sizes without speculative decoding.<\/p>\n SGLang-TorchTPU will keep adding models, starting with DeepSeek V4.1 Flash and Reflection Beam. It’s also working toward Day 0 TPU support for new open models, meaning support on release day, by the first quarter of 2027, according to RadixArk. Google, Meta and RadixArk plan to share more at SGLang Summit 2026, Nov. 12\u201313 in San Francisco.<\/p>\n \n Follow Tom\u2019s Hardware on Google News, or add us as a preferred source, to get our latest news, analysis, & reviews in your feeds.<\/p>\n
\n<\/span>
\n<\/p>\n
\n<\/span><\/p>\n
\n You may like
\n <\/span><\/p>\n
\nFor those who read our public newsletter, this is not a surprise, as on Nov 28, 2025, we publicly wrote that Meta is a TPU customer & has lots of engineers working on TPUs. Our accelerator supply chain model has details about Meta’s TPU procurement!October 21, 2026<\/a><\/p>\n