It's Time To Build Incredible Apps On Unstructured Data With LLMs, Says AI Infrastructure Expert Diptanu Gon Choudhury

Evolution in technology is a dynamic process that shapes industries and redefines human capabilities. From the internet to artificial intelligence, every advancement is marked by failures, challenges, and innovative solutions.

Industry expert Diptanu Gon Choudhury, who is also the founder of Tensorlake, believes that failure is an inherent part of technological progress. Whether it's a bug in software, a hardware malfunction, or an unforeseen environmental factor, setbacks are inevitable. However, what distinguishes successful endeavors from mere setbacks is the ability to learn, adapt, and innovate in the face of adversity.

Mr. Choudhury's experiences are a testament to his relentless pursuit of solutions to complex technological problems. From his role as a tech lead for multiple well-known streaming and social media companies, Choudhury built a real-time speech inference engine for a big multinational technology company, machine learning platforms, and developed cluster schedulers for industry giants. Mr. Choudhury has consistently pushed the boundaries of what's possible in technology.

"I have worked on several moonshot projects in my career," says Choudhury. "I built a massively scalable cluster scheduler in 2014 to run all the cloud services for a well-known streaming platform. It was thought to be impossible to pull off just because no one else had done it at that scale." Choudhury's solution was not only successful but it paved the way for the current generation of systems that developers use to run online workloads on the cloud. "The same pattern repeated much later, when AI workloads were becoming a reality around a decade ago. I worked on making speech models work really fast so that users of a popular social media platform could see transcriptions of videos on the website in real-time, and that path was full of challenges as the tools were not mature, and the research for making AI faster was very nascent."

This tech fanatic has developed a massively scalable cloud-native cluster scheduler for efficient computing for applications of Large Language Models. Mr. Choudhury believes that internet applications require scalable software, and the market is shifting towards serverless computing. He is exploring combining serverless infrastructure with computing for large language models and addressing the need for hardware accelerators, like GPUs, with his latest endeavor.

At the forefront of Diptanu Choudhury's latest endeavor lies Indexify - a groundbreaking solution poised to revolutionize the way businesses interact with unstructured data. In an era where data is king, Indexify offers a game-changing approach to extracting valuable insights from vast troves of unstructured data embedded within videos, audio files, and documents.

"There is a paradigm shift happening right now in the field of computing," says Choudhury. "Software 1.0 was about using explicit instructions on data from structured databases. By contrast, Large Language Models enable developers to build applications from data that is more omnipresent around us and on the internet. This data is largely unstructured. The web page you are reading this article on, a video call you made this morning, or the emails coming in your inbox, that is all unstructured data. Indexify is the infrastructure through which LLMs gain access to, and make use of, that unstructured data."

This innovative structured extraction and indexing engine enables companies, developers, and individuals to unlock the latent potential of unstructured data by seamlessly extracting structured information that can be fed into large language models. By democratizing access to data infrastructure, Indexify paves the way for a new era of innovation and discovery.

As the technological landscape continues to evolve, the shift to serverless computing emerges as a defining trend reshaping the way companies interact with their data. With the advent of open-source cloud computing platforms, the traditional paradigm of local servers is giving way to a more flexible and scalable approach to resource management.

Mr. Choudhury recognizes the significance of this radical transformation and is committed to providing solutions that address the evolving needs of businesses in the serverless era. By combining serverless infrastructure with LLMs, Choudhury and his team are at the forefront of innovation in cloud-native computing.

Identifying structure in unstructured data is one of the most difficult aspects of leveraging the data's power. While structured data has long been the cornerstone of data analytics, usage of unstructured data for analytics has been neglected. Being able to use unstructured data directly presents new opportunities and challenges for businesses seeking to extract actionable insights.

Indexify's innovative approach to processing unstructured data bridges the gap between raw information and actionable insights, enabling businesses to leverage the full potential of their data assets. By providing a scalable and cost-effective solution for processing unstructured data, Indexify empowers businesses to stay ahead of the curve in an increasingly data-driven world.

Central to Mr. Choudhury's vision for his technology solutions is the principle of open-source innovation. By making Indexify fully open-source, Choudhury aims to democratize access to data infrastructure and empower developers to build upon the foundation of collaborative innovation.

Diptanu Choudhury truly aims to stand out with his company and solutions that promote transparency and openness in a market where proprietary solutions are dominant. By embracing the power of open source, Mr. Choudhury and his team are paving the way for a new era of collaborative innovation, where the collective wisdom of the community drives progress and prosperity.

As we stand on the precipice of a new technological frontier, the journey ahead is filled with both promise and uncertainty. However, one thing remains clear: with every challenge comes an opportunity for innovation. By embracing failure, harnessing the power of solutions, and embracing the principles of openness and collaboration, we can chart a course toward a future defined by endless possibilities.

In the words of Diptanu Gon Choudhury, "The path to innovation is paved with failures, challenges, and solutions. As we navigate the ever-changing landscape of technology, we need to embrace the spirit of innovation and seize the opportunities that lie ahead."

I have worked on several moonshot projects in my career, I built a massively scalable cluster scheduler in 2013 to run all of Netflix's cloud infrastructure. It was thought to be impossible to pull it off just because no one else had done it at that scale but it was not only successful but it paved the way for the current generation of systems that developers use to run online workloads on the cloud. The same pattern repeated much later when AI workloads were becoming a reality half a decade back, I worked on making speech models work really fast so that users of Facebook could see the transcription of videos on the website in real-time, and that path was full of challenges as the tools were not mature, and the research for making AI faster was very nascent."