Motivation
2020-2030 will be remembered as the decade when LLMs (Large Language Model) boomed. ChatGPT led the industry with so many changes, but other vendors and “Tech Giants” did not wait to release the competitor models for LLM (Large Language Model): Google with Gemini, Facebook with OLLama, Atrophic with Claude, Databricks with Mosaic AI, Hugging face with BLOOM, Mistral and so many more. 

Executives and software architects are trying to integrate AI capabilities within the business logics and data platform walls. And they all have the same questions:

• Which model shall I use?
• Which development environment is better?
• Which Architecture empowers the LLMs model best?
• How can I integrate vision and LLM models in multi-agent multi-model use cases?

These are all interesting technical questions but at the most important one remains unanswered:

• How much will this AI use case cost?

In such a fast-developing field it is easy to follow the technological hype and end up with a much higher cost than imagined. As often in Computer Science there are many different solutions to the same problem. This project is about coming up with a methodology to compare different LLM based solutions and analyze cost-optimization tradeoffs.

Goals
We will build on the past TUM-DI-LAB project, and we will investigate the RAG use case. The first step will be to select a technology or come up with a custom approach for the RAG evaluation (e.g. github.com/explodinggradients/ragas, https://github.com/confident-ai/deepeval). The second step will be to come up with a cost calculator for the selected solution. Finally, different aspects of the existing RAG architecture will be changed (e.g. embedding model, LLM model, embedding database, if there is time computing infrastructure) and it will
be observed how the evaluation score and the cost prediction will adapt.

Methodologies

1. Development of Evaluation Metrics:

• Define metrics to quantify the performance of LLM models, development environments, and architectures.
• Metrics could include accuracy, efficiency, scalability, and ease of integration.

2. Cost Analysis:

• Develop cost estimation models to calculate the expenses associated with implementing different AI-based use cases.
• Consider factors such deployment, maintenance, and scalability.

3. Cloud-Based Software Development:

• Design and develop a cloud-based software solution, leveraging scalable cloud infrastructure.
• Implement features for data storage, processing, and visualization to support the analysis and decision-making process.
• Design an intuitive user interface for the calculator, enabling users to easily input their requirements and preferences.

4. Simulation and Scenario Analysis:

• Conduct simulations to analyze the performance and cost implications of different AI solutions under various scenarios.
• Explore the impact of different parameters such as dataset size, complexity, and business requirements.

Requirements and opportunities
We do not need any specific requirement from the students other than a great motivation for learning and proactivity in any of the project’s phases. The more you invest in the project the more you will get out of it. Engaging in this project offers students a rare glimpse into merging state-of-the-art technologies with practical enterprise challenges. It's an academic-corporate collaboration that stands to redefine how organizations perceive and manage their knowledge repositories. We can’t wait to start!

Apply to this project here