Last week on PyTorch’s YouTube channel, the official PyTorch documentary, “Powering the AI Revolution” made its debut. What did the 35-minute video reveal about our favorite deep learning library?

“Obviously, we’re in the middle of an explosion in the whole AI space,” the documentary begins. It’s Brian Granger speaking — a senior principal technologist at Amazon Web Services (and a leader of the Python project and co-founder of Project Jupyter.) And by the end of the documentary, it’s been noted that under the hood, both ChatGPT and Stable Diffusion are using the PyTorch framework.

“The state of AI today would be very much less developed if PyTorch hadn’t have come out,” says Jeremy Howard, co-founder of the nonprofit Fast.ai…

The documentary ends with some compelling statistics…

• 60% of AI research implementations now use PyTorch.
• PyTorch is used in over 600,000 repositories on GitHub.
• PyTorch is used everywhere from NASA to top entertainment companies like Disney and Pixar, and even at Astra Zeneca for drug discovery.

In 2022, the Linux Foundation had put it succinctly. “If you peel back the cover of any AI application, there is a strong chance PyTorch is involved in some way…”

But the documentary traces the story of PyTorch all the way back to its original development by a small team at Meta, through the explosion in popularity after its initial release in 2016 — and then on to the launch of its own foundation as part of the Linux Foundation.

And somewhere along the way, it also provides some valuable perspective on the tools being used by AI researchers today. The documentary offers a glimpse at not only how far we’ve come — but also where we might be going next…

The Days of Fragmentation

The documentary has some high-powered sponsors — AWS, AMD, Google Cloud, Meta and Microsoft Azure. But it begins by looking back to a time where the number of AI tools was proliferating wildly. Remembering 2015, PyTorch co-creator Soumith Chintala says “There were like 15 or 20 tools that, at that time, were all trying to enable research.” (Choices at the time included Torch, Theano, Caffee…)

“A lot of innovation was happening, but it was pocketed…” adds Microsoft’s principal PM lead for AI Frameworks, Parinita Rahi. “The big players were doing individual model development, and a lot of it was siloed within individual organizations or research institutions.”

Yangqing Jia — who was the director of Facebook AI for three years starting in 2016 — says in the documentary that the available products at the time wavered between being useful and “being a cute tool…”

Maybe that’s why Chintala also remembers the release of TensorFlow in 2015  — along with its credibility and polish. “It was from Google, high-budget, high marketing…” In comparison, all the other tools were started by what he calls “enthusiasts… Started by researchers, as a way for them to do their work… TensorFlow was built from the ground up, with Google engineering.”

But another player was about to arrive…

2016

Facebook’s fundamental AI research lab was trying to “make progress” on computer vision, Chintala remembers in the documentary — and they needed AI applications for Facebook’s newsfeed, search results, and recommendations. So one of his first tasks when joining Facebook was building an AI framework that would be “battle-tested” — making hundreds of trillions of predictions per day. (And in a wide variety of deployment environments.)

Chintala remembered how one of PyTorch’s co-creators was a first-year undergraduate at the University of Warsaw looking for an internship. Adam Paszke had been active in the community building Lua Torch — and in the documentary shares his own recollection of 2016. “I was excited about machine learning. I just thought it was a really cool field, and I wanted to learn more about it.”

From December of 2015 to April of 2016, their small team worked on separating the C and C++ math functions from Lua’s bindings — then realized they could be linked to any other language. “All the other libraries were already in Python, and people were happy with it and they were liking it,” Paszke remembered — so trying to bind it to Python was “a fairly natural next step.”

PyTorch was released in January of 2017 — and Chintala remembered getting the same feedback from several people: “We were enabling them to do great work.” Lightning AI CEO Will Falcon — who at the time was using TensorFlow for AI research at Stanford — says in the documentary that after trying PyTorch, “It was like night and day.”

Chintala also remembered that “In 2017, I think we were working 16-plus hour days.”

Performant and Popular

After a two-year effort to make PyTorch more performant, its popularity boomed. “We started to see libraries explode on top of PyTorch,” says Joe Spisak (who at the time was AWS’s head of AI partnerships). There were libraries for NLP, computer vision, reinforcement learning… “The whole world started to build on top of it, as this stable foundation.”

AMD heard that PyTorch was the preferred machine learning framework for “a number of different customers, of different sizes, across different industries,” remembered AMD’s solutions architecture director Niles Burbank. And it was clear: “This was something that was going to be important… We started working on taking PyTorch and adapting it in a way that it could take advantage of the hardware acceleration capabilities of — particularly, our GPU products.” They tested improvements on an internal fork, and then contributed them back into the project, over the course of several years.

Chintala saw PyTorch being used by major self-driving car companies — including Tesla, Uber, and Cruise — and said the PyTorch team found that “really cool — almost intimidating to us” said it made them feel “very responsible” for making sure the code was bug-free. Microsoft had developers using PyTorch — and in top Microsoft projects like Bing and Office.

Spisak remembers that eventually, Google started thinking about supporting PyTorch on their TPUs. And this was a kind of milestone, said Dwarak Rajagopal, who led the PyTorch Core Frameworks team at Meta, and is now a senior engineering director for Google’s AI. Rajagopal describes it as proving you could take PyTorch to other hardware platforms beyond GPUs — and calls this “a pretty big moment.”

Appreciating Contributors

The documentary also includes the 2022 decision to move the open source PyTorch project into its own foundation. At the time the Linux Foundation quipped, “We are grateful for Meta’s trust in ‘passing us the torch’ (pun intended)… ”

Their announcement emphasized that “Open source communities are playing and will play a leading role in development of the tools and solutions that make AI and ML possible — and make it better over time.” And they noted that from August 2021 through August 2022, “PyTorch counted over 65,000 commits. Over 2,400 contributors participated in the effort…”

Red carpet premiere of the PyTorch Documentary in Paris was a success 🤩 Thank you to everyone who was able to make it.

We’ll be doing a virtual premiere on June 25 at 10am PT – don’t miss it!https://t.co/EelfozzKMd pic.twitter.com/H6nACO42Dv

— PyTorch (@PyTorch) June 19, 2024

Looking Ahead

Toward the end of the documentary, Spisak reflects on their legacy. “Every day, we look around and say, ‘Yeah, that’s us in there. That’s us in there.’ It’s so much fun to see.”

And Rajagopal says, “I still think we are still getting started.”

AWS’s Brian Granger says deeper layers of innovation get built on top of foundational open-source projects like PyTorch, leading to individuals and organizations “able to do things that they could never do without those layers of innovation on top of those open source projects.” And Spisak ultimately predicts PyTorch will be at the heart of an even more diverse set of platforms, “whether it’s on-device or ambient computing or wearable…”

Microsoft’s Parinita Rahi predicts more and more applications built using powerful LLMs — with people freed from the limits of their hardware to build life-enriching solutions. “The sky’s the limit.” And AMD’s Niles Burbank also predicted a bright future growing out of that better technology. “When you apply more hardware or more iterations to the problem, the same basic techniques continue to yield better and better results.”

And ultimately, he sees a world where researchers and industrial customers take LLMs and “scale them up in ways that — even by the standards of the computing industry — are sort of unprecedented.”

---

WebReduce

• New York entrepreneur‘s startup uses AI to successfully drop hats onto customers.
• Lex Fridman interviews a safety researcher about the dangers of superintelligent AI.
• Stack Overflow’s podcast hosts celebrate their 5th-anniversary edition by looking back over the last half-decade.
• Fast Company’s “1994 Week” asks what we can learn from 30-year-old websites.
• Architects of the 802 and WiFi Standards discuss its creation in the 1990s and its importance.
• At GopherCon EU, the Go Time podcast hosts a live Family Feud-style gameshow called “Gophers Say”.

The post Official PyTorch Documentary Revisits Its Past, and Its Future appeared first on The New Stack.

It has been estimated that the Pytorch is used by over 60% of AI jobs today. Now, here's the story of how the deep learning framework was created.