Determined AI makes its machine learning infrastructure free and open source

Machine learning has quickly gone from niche field to crucial component of innumerable software stacks, but that doesn’t mean it’s easy. The tools needed to create and manage it are enterprise-grade and often enterprise-only — but Determined AI aims to make them more accessible than ever by open-sourcing its entire AI infrastructure product.

The company created its Determined Training Platform for developing AI in an organized, reliable way — the kind of thing that large companies have created (and kept) for themselves, the team explained when they raised a $11 million Series A last year.

Determined AI nabs $11M Series A to democratize AI development

“Machine learning is going to be a big part of how software is developed going forward. But in order for companies like Google and Amazon to be productive, they had to build all this software infrastructure,” said CEO Evan Sparks. “One company we worked for had 70 people building their internal tools for AI. There just aren’t that many companies on the planet that can withstand an effort like that.”

At smaller companies, ML is being experimented with by small teams using tools intended for academic work and individual research. To scale that up to dozens of engineers developing a real product… there aren’t a lot of options.

“They’re using things like TensorFlow and Pytorch,” said Chief Scientist Ameet Talwalkar. “A lot of the way that work is done is just conventions: how do the models get trained? Where do I write down the data on which is best? How do I transform data to a good format? All these are bread and butter tasks. There’s tech to do it, but it’s really the wild west. And the amount of work you have to do to get it set up… there’s a reason big tech companies build out these internal infrastructures.”

Determined AI, whose founders started out at UC Berkeley’s AmpLab (home of Apache Spark), has been developing its platform for a few years, with feedback and validation from some paying customers. Now, they say, it’s ready for its open source debut — with an Apache 2.0 license, of course.

“We have confidence people can pick it up and use it on their own without a lot of hand holding,” said Sparks.

You can spin up your own self-hosted installation of the platform using local or cloud hardware, but the easiest way to go about it is probably the cloud-managed version that automatically provisions resources from AWS or wherever you prefer and tears them down when they’re no longer needed.

The hope is that the Determined AI platform becomes something of a base layer that lots of small companies can agree on, providing portability to results and standards so you’re not starting from scratch at every company or project.

With machine learning development expected to expand by orders of magnitude in the coming years, even a small piece of the pie is worth claiming, but with luck Determined AI may grow to be the new de facto standard for AI development in small and medium businesses.

You can check out the platform on GitHub or at Determined AI’s developer site.