Business value is built into our DNA, and nothing is better than hearing the success stories directly from our customers.

We’re thrilled to share that our customers have recognized DataRobot in the TrustRadius Top Rated Award for the third consecutive year in the following categories:

• Data science
• Machine learning
• Predictive analytics

We are incredibly proud of this award — based solely on customer reviews.

About TrustRadius

TrustRadius is a buyer intelligence platform for business technology and its annual Top Rated Awards are based entirely on customer feedback – they aren’t influenced by outside opinion. TrustRadius looks at the recency of reviews, relevancy of products compared to others in the same category, and overall ratings. 

With a trScore of 8.8 out of 10 and nearly 60 verified reviews from our customers, we’re proven as one of the most valuable platforms in our industry, with demonstrated impact and results.

Why our customers trust DataRobot

In their own words, our customers share the wins they’ve experienced by using the DataRobot AI Platform:

• “[DataRobot’s AI Platform] is frictionless…you don’t have to worry about sticking a bunch of services together as with the hyperscalers…it just works.” – Director of Information Technology, Real Estate 
• “A powerful tool…greatly reducing the time required to build powerful and accurate models.” – Data and Insights Lead, Healthcare
• “An excellent interface – fast enough to use for development and easy to share insights with business users.” – Information Technology, Financial Services
• “A great return on investment.” – Analyst, Insurance

When I spoke with our Chief Customer Officer, Jay Schuren, he shared his sincere appreciation for our brilliant customers and thanked them for this recognition. He said:

We continually strive to wow our customers. The Top Rated Award is only made possible by our customers’ success. When our customers win, we join them in celebrating the business transformations made possible with AI.

Learn more

Hear how customers deliver AI value at FordDirect, Freddie Mac, 84.51°, and many more.  

The post DataRobot Recognized by Customers with TrustRadius Top Rated Award for Third Consecutive Year  appeared first on DataRobot AI Platform.

AI leaders are rightfully struggling to move beyond the prototype and experimental stage, it’s our mission to change that. At DataRobot, we call this the “confidence gap”. It’s the trust, safety and accuracy and concerns surrounding GenAI that are holding teams back, and we are committed to addressing it. And, it’s the core focus of our Spring ’24 launch and its groundbreaking features.

This release focuses on the three most significant hurdles to unlocking value with GenAI. 

First, we’re bringing you enterprise-grade open-source LLM support, and a suite of evaluation and testing metrics, to help you and your teams confidently create production-grade AI applications. To help you safeguard your reputation and prevent risk from AI apps running amok, we’re bringing you real-time intervention and moderation for all your GenAI applications. And finally, to ensure your entire fleet of AI assets stay in peak performance, we’re bringing you a first-of-its-kind multi-cloud and hybrid AI Observability to help you fully govern and optimize all of your AI investments.

Confidently Create Production-Grade AI Applications 

There is a lot of talk about fine-tuning an LLM. But, we have seen that the real value lies in fine-tuning your generative AI application. It’s tricky, though. Unlike predictive AI, which has thousands of easily accessible models and common data science metrics to benchmark and assess performance against, generative AI hasn’t—until now. 

Unlike predictive AI, which has thousands of easily accessible models and common data science metrics to benchmark and assess performance against, generative AI hasn’t—until now.

In our Spring ’24 launch, get enterprise-grade support for any open-source LLM. We’ve also introduced an entire set of LLM evaluation, testing, and metrics. Now, you can fine-tune your generative AI application experience, ensuring their reliability and effectiveness.

Enterprise-Grade Open Source LLMs Hosting

Privacy, control, and flexibility remain critical for all organizations regarding LLMs.There has been no easy answer for AI Leaders who have been stuck with having to pick between vendor lock-in risks using major API-based LLMs that could become sub-optimal and expensive in the immediate future, figuring out how to stand up and host your own open source LLM, or custom-building, hosting, and maintaining your own LLM. 

With our Spring Launch, you have access to the broadest selection of LLMs, allowing you to choose the one that aligns with your security requirements and use cases. Not only do you have ready-to-use access to LLMs from leading providers like Amazon, Google, and Microsoft, but you also have the flexibility to host your own custom LLMs. Additionally, our Spring ’24 Launch offers enterprise-level access to open-source LLMs, further expanding your options.

We have made hosting and using open-source foundational models like LLaMa, Falcon, Mistral, and Hugging Face easy with DataRobot’s built-in LLM security and resources. We have eliminated the complex and labor-intensive manual DevOps integrations required and made it as easy as a drop-down selection.

LLM Evaluation, Testing and Assessment Metrics 

With DataRobot, you can freely choose and experiment across LLMs. We also give you advanced experimentation options, such as trying various chunking strategies, embedding methods, and vector databases. With our new LLM evaluation, testing, and assessment metrics, you and your teams now have a clear way of validating the quality of your GenAI application and LLM performance across these experiments. 

With our first-of-its-kind synthetic data generation for prompt-and-answer evaluation, you can quickly and effortlessly create thousands of question-and-answer pairs. This lets you easily see how well your RAG experiment performs and stays true to your vector database.  

We are also giving you an entire set of evaluation metrics. You can benchmark, compare performance, and rank your RAG experiments based on faithfulness, correctness, and other metrics to create high-quality and valuable GenAI applications. 

And with DataRobot, it’s always about choice. You can do all of this as low code or in our fully hosted notebooks, which also have a rich set of new codespace functionality that eliminates infrastructure and resource management and facilitates easy collaboration. 

Observe and Intervene in Real-Time

The biggest concern I hear from AI leaders about generative AI is reputational risk. There are already plenty of news articles about GenAI applications exposing private data and legal courts holding companies accountable for the promises their GenAI applications made. In our Spring ’24 Launch, we’ve addressed this issue head-on. 

With our rich library of customizable guards, workflows, and notifications, you can build a multi-layered defense to detect and prevent unexpected or unwanted behaviors across your entire fleet of GenAI applications in real time. 

Our library of pre-built guards can be fully customized to prevent prompt injections and toxicity, detect PII, mitigate hallucinations, and more. Our moderation guards and real-time intervention can be applied to all of your generative AI applications – even those built outside of DataRobot, giving you peace of mind that your AI assets will perform as intended.

Govern and Optimize Infrastructure Investments

Because of generative AI, the proliferation of new AI tools, projects, and teams working on them has increased exponentially. I often hear about “shadow GenAI” projects and how AI leaders and IT teams struggle to reign it all in. They find it challenging to get a comprehensive view, compounded by complex multi-cloud and hybrid environments. The lack of AI observability opens organizations up to AI misuse and security risks. 

Cross-Environment AI Observability 

We’re here to help you thrive in this new normal where AI exists in multiple environments and locations. With our Spring ’24 Launch, we’re bringing the first-of-its-kind, cross-environment AI observability –  giving you unified security, governance, and visibility across clouds and on-premise environments. 

Your teams get to work in the tools and ways they want; AI leaders get the unified governance, security, and observability they need to protect their organizations. 

Our customized alerts and notification policies integrate with the tools of your choice, from ITSM to Jira and Slack, to help you reduce time-to-detection (TTD) and time-to-resolution (TTR). 

Insights and visuals help your teams see, diagnose, and troubleshoot issues with your AI assets – Trace prompts to the response and content in your vector database with ease, See Generative AI topic drift with multi-language diagnostics, and more.  

NVIDIA and GPU integrations 

And, if you’ve made investments in NVIDIA, we’re the first and only AI platform to have deep integrations across the entire surface area of NVIDIA’s AI Infrastructure – from NIMS, to NeMoGuard models, to their new Triton inference services, all ready for you at the click of a button. No more managing separate installs or integration points, DataRobot makes accessing your GPU investments easy. 

Our Spring ’24 launch is packed with exciting features, including GenAI, predictive capabilities, and enhancements in time series forecasting, multimodal modeling, and data wrangling. 

All of these new features are available in cloud, on-premise, and hybrid environments. So, whether you’re an AI leader or part of an AI team, our Spring ’24 launch sets the foundation for your success. 

This is just the beginning of the innovations we’re bringing you. We have so much more in store for you in the months ahead. Stay tuned as we’re hard at work on the next wave of innovations. 

Get Started 

Learn more about DataRobot’s GenAI solutions and accelerate your journey today. 

• Join our Catalyst program to accelerate your AI adoption and unlock the full potential of GenAI for your organization.
• See DataRobot’s GenAI solutions in action by scheduling a demo tailored to your specific needs and use cases.
• Explore our new features, and connect with your dedicated DataRobot Applied AI Expert to get started with them. 

The post Spring Launch ‘24: Meet DataRobot’s Newest Features to Confidently Build and Deploy Production-Grade GenAI Applications appeared first on DataRobot AI Platform.

In the fast-evolving landscape of generative AI, choosing the right components for your AI solution is critical. With the wide variety of available large language models (LLMs), embedding models, and vector databases, it’s essential to navigate through the choices wisely, as your decision will have important implications downstream. 

A particular embedding model might be too slow for your specific application. Your system prompt approach might generate too many tokens, leading to higher costs. There are many similar risks involved, but the one that is often overlooked is obsolescence. 

As more capabilities and tools go online, organizations are required to prioritize interoperability as they look to leverage the latest advancements in the field and discontinue outdated tools. In this environment, designing solutions that allow for seamless integration and evaluation of new components is essential for staying competitive.

Confidence in the reliability and safety of LLMs in production is another critical concern. Implementing measures to mitigate risks such as toxicity, security vulnerabilities, and inappropriate responses is essential for ensuring user trust and compliance with regulatory requirements.

In addition to performance considerations, factors such as licensing, control, and security also influence another choice, between open source and commercial models: 

• Commercial models offer convenience and ease of use, particularly for quick deployment and integration
• Open source models provide greater control and customization options, making them preferable for sensitive data and specialized use cases

With all this in mind, it’s obvious why platforms like HuggingFace are extremely popular among AI builders. They provide access to state-of-the-art models, components, datasets, and tools for AI experimentation. 

A good example is the robust ecosystem of open source embedding models, which have gained popularity for their flexibility and performance across a wide range of languages and tasks. Leaderboards such as the Massive Text Embedding Leaderboard offer valuable insights into the performance of various embedding models, helping users identify the most suitable options for their needs. 

The same can be said about the proliferation of different open source LLMs, like Smaug and DeepSeek, and open source vector databases, like Weaviate and Qdrant.  

With such mind-boggling selection, one of the most effective approaches to choosing the right tools and LLMs for your organization is to immerse yourself in the live environment of these models, experiencing their capabilities firsthand to determine if they align with your objectives before you commit to deploying them. The combination of DataRobot and the immense library of generative AI components at HuggingFace allows you to do just that. 

Let’s dive in and see how you can easily set up endpoints for models, explore and compare LLMs, and securely deploy them, all while enabling robust model monitoring and maintenance capabilities in production.

Simplify LLM Experimentation with DataRobot and HuggingFace

Note that this is a quick overview of the important steps in the process. You can follow the whole process step-by-step in this on-demand webinar by DataRobot and HuggingFace. 

To start, we need to create the necessary model endpoints in HuggingFace and set up a new Use Case in the DataRobot Workbench. Think of Use Cases as an environment that contains all sorts of different artifacts related to that specific project. From datasets and vector databases to LLM Playgrounds for model comparison and related notebooks.

In this instance, we’ve created a use case to experiment with various model endpoints from HuggingFace. 

The use case also contains data (in this example, we used an NVIDIA earnings call transcript as the source), the vector database that we created with an embedding model called from HuggingFace, the LLM Playground where we’ll compare the models, as well as the source notebook that runs the whole solution. 

You can build the use case in a DataRobot Notebook using default code snippets available in DataRobot and HuggingFace, as well by importing and modifying existing Jupyter notebooks. 

Now that you have all of the source documents, the vector database, all of the model endpoints, it’s time to build out the pipelines to compare them in the LLM Playground. 

Traditionally, you could perform the comparison right in the notebook, with outputs showing up in the notebook. But this experience is suboptimal if you want to compare different models and their parameters. 

The LLM Playground is a UI that allows you to run multiple models in parallel, query them, and receive outputs at the same time, while also having the ability to tweak the model settings and further compare the results. Another good example for experimentation is testing out the different embedding models, as they might alter the performance of the solution, based on the language that’s used for prompting and outputs. 

This process obfuscates a lot of the steps that you’d have to perform manually in the notebook to run such complex model comparisons. The Playground also comes with several models by default (Open AI GPT-4, Titan, Bison, etc.), so you could compare your custom models and their performance against these benchmark models.

You can add each HuggingFace endpoint to your notebook with a few lines of code. 

Once the Playground is in place and you’ve added your HuggingFace endpoints, you can go back to the Playground, create a new blueprint, and add each one of your custom HuggingFace models. You can also configure the System Prompt and select the preferred vector database (NVIDIA Financial Data, in this case). 

Figures 6, 7. Adding and Configuring HuggingFace Endpoints in an LLM Playground

After you’ve done this for all of the custom models deployed in HuggingFace, you can properly start comparing them.

Go to the Comparison menu in the Playground and select the models that you want to compare. In this case, we’re comparing two custom models served via HuggingFace endpoints with a default Open AI GPT-3.5 Turbo model.

Note that we didn’t specify the vector database for one of the models to compare the model’s performance against its RAG counterpart. You can then start prompting the models and compare their outputs in real time.

There are tons of settings and iterations that you can add to any of your experiments using the Playground, including Temperature, maximum limit of completion tokens, and more. You can immediately see that the non-RAG model that doesn’t have access to the NVIDIA Financial data vector database provides a different response that is also incorrect. 

Once you’re done experimenting, you can register the selected model in the AI Console, which is the hub for all of your model deployments. 

The lineage of the model starts as soon as it’s registered, tracking when it was built, for which purpose, and who built it. Immediately, within the Console, you can also start tracking out-of-the-box metrics to monitor the performance and add custom metrics, relevant to your specific use case. 

For example, Groundedness might be an important long-term metric that allows you to understand how well the context that you provide (your source documents) fits the model (what percentage of your source documents is used to generate the answer). This allows you to understand whether you’re using actual / relevant information in your solution and update it if necessary.

With that, you’re also tracking the whole pipeline, for each question and answer, including the context retrieved and passed on as the output of the model. This also includes the source document that each specific answer came from.

How to Choose the Right LLM for Your Use Case

Overall, the process of testing LLMs and figuring out which ones are the right fit for your use case is a multifaceted endeavor that requires careful consideration of various factors. A variety of settings can be applied to each LLM to drastically change its performance. 

This underscores the importance of experimentation and continuous iteration that allows to ensure the robustness and high effectiveness of deployed solutions. Only by comprehensively testing models against real-world scenarios, users can identify potential limitations and areas for improvement before the solution is live in production.

A robust framework that combines live interactions, backend configurations, and thorough monitoring is required to maximize the effectiveness and reliability of generative AI solutions, ensuring they deliver accurate and relevant responses to user queries.

By combining the versatile library of generative AI components in HuggingFace with an integrated approach to model experimentation and deployment in DataRobot organizations can quickly iterate and deliver production-grade generative AI solutions ready for the real world.

The post How to Choose the Right LLM for Your Use Case appeared first on DataRobot AI Platform.

We developed a mission to serve our community thoughtfully.

Women@DR seeks to create, promote and expand an inclusive culture that connects, educates and advances the needs, professional goals, and aspirations of female-identifying members and allies.

As we celebrate Women’s History Month and the International Women’s Day in March, we look to our community of strong, resilient, and skillful leaders to set the tone. In our February 2024 global company kick off, 7 women were celebrated as DataRobot AI Legends. They were honored as the embodiment of DataRobot values and success.

We caught up with some of these trailblazers and asked them questions that might inspire others to become legendary:

• How have you built confidence and/or resiliency over the course of your career?
• What advice would you give a female identifying person who is just starting their career?

Here’s what they had to say …

Ina Ko, Senior Product Director, Customer Engineering

How have you built confidence and/or resiliency over the course of your career?

Building confidence and resilience over the course of my career has involved embracing humility, seeking new challenges, using a supportive network for guidance and feedback, and focusing on the impact of my work. Recognizing the fluidity of life and career paths as growth opportunities rather than setbacks has been essential. This mindset has helped me navigate uncertainties, turn failures into lessons, and embrace continuous learning and adaptation.

What advice would you give a female identifying person who is just starting their career?

For women just starting their careers, my advice is to focus on your unique journey without comparing yourself to others. Embrace change, as it brings valuable lessons that refine your path to success. Cultivate a mindset of positive intent and seek to understand the motivations of your colleagues and peers. This approach will help you foster a collaborative environment by building deep and authentic connections with those you work alongside. Embrace every opportunity to grow, lean into changes, and remember that your unique contributions and resilience will define your path.

Julia Townsend, Director, Revenue Strategy & Execution

How have you built confidence and/or resiliency over the course of your career?

When I first started out back in the day, I had a big goal I wanted to achieve and everything I did was chosen to chip away at the goal.  This really helped me see a bigger picture and not get too down when stumbling blocks came along. I also have a personal mantra that ‘this is one moment, and will pass with time’, that really helps!

Embracing new opportunities when they come up, even if you think you might not have all the skills you think are needed, has led to some interesting career experiences for me. This has made me feel confident that I can turn my hand to new projects even if I do not have all the answers right away.

What advice would you give a female identifying person who is just starting their career?

I would recommend really taking the time to invest in building relationships and champions across the business.  This is such a great career and life skill that is essential to master in order to get things done and have trust and influence.  This has a lasting impact for every new move you make.

Brook Miller, Senior Director, Demand Planning, Creative, & APAC Marketing

How have you built confidence and/or resiliency over the course of your career?

For me, confidence comes from knowing that mastery isn’t the end goal for me. Instead, I choose to embrace a mindset of continual learning and build confidence through collaborating with people who can teach me new things as well as sharing my own experience with others. 

In the times where I need to draw on resilience, I find that zooming out to the bigger picture to gain some perspective really helps. By viewing the tough times as just a chapter in the overall story of your career, you can keep moving forward without being consumed by what’s happening in the present moment. It’s also ok to ask for help, say you don’t know the answer or take some time out to reset when needed.

What advice would you give a female identifying person who is just starting their career?

Take the time to understand yourself, your strengths and where you can contribute and then put your hand up for something that scares you every year. The right mindset will take you very far while you’re building your skillset. Align yourself with the people you admire and want to learn from and don’t be afraid to ask them for guidance. One day you’ll be able to give that same gift to someone else.

Katherine Stepanova, Senior Director, Legal – Lead Americas Commercial Counsel

How have you built confidence and/or resiliency over the course of your career?

Building confidence and resilience is a process that requires a lot of patience, support from others, both professionally and personally, and lots of hard work. My confidence and resilience, first and foremost, come from my grandma, an incredibly strong woman who had gone through a myriad of challenges in her life and was able to embrace the challenges and overcome difficulties and failures. She taught me from early childhood that girls can do anything they set out to do if they get educated, work hard, practice commendable work ethic no matter what they do, and believe in themselves. She’s been my biggest cheerleader all my life, and I’m eternally grateful to her for that, and that’s something I will try to pass onto my daughter as well.    

What advice would you give a female identifying person who is just starting their career?

Build a strong support system, both professionally and personally. There will be good days and bad ones, bumps on the road, but with a strong support system you’ll never lose the desire and ability to move forward, reflect on the challenges, and overcome difficulties. Don’t forget to enjoy the journey! 

Sahana Sreenath, Lead, Database Engineer

How have you built confidence and/or resiliency over the course of your career?

For me, it has been through stepping out of my comfort zone and taking on tasks I found challenging initially. Reaching out readily when in need of guidance and not being afraid of making mistakes in the beginning all make for a solid foundation to build on.  

What advice would you give a female identifying person who is just starting their career?

I found the following very helpful when starting out: 1) Remaining curious everyday will carry you far and help you to learn the intricacies of your roles and responsibilities better, 2) Frequently soliciting and incorporating feedback will enable you to become more efficient in the workplace, and 3) Continue to stay motivated and dedicated, develop a strong work ethic, challenge yourself whenever possible and your work will speak for itself!

Teresa Gearin, Marketing Operations Project Manager

How have you built confidence and/or resiliency over the course of your career?

Embracing change! Change is inevitable and sometimes feels constant but learning to adapt and being open to learning and enhancing your skills makes change feel less cumbersome and more like an opportunity. I am always actively seeking ways to improve my environment whether it is processes, organization tactics, or just in the way I think and perceive any situation. This has helped me build the confidence needed to navigate male dominated industries/teams, tough situations, and advocate for myself. At the end of the day I am my greatest ally and worst critic, practicing balancing the two helps. 

What advice would you give a female identifying person who is just starting their career?

Your voice matters! Learn to embrace the power of advocating for yourself and others confidently and assertively. Your time, experiences, and thoughts are valuable. Always be coachable, proactive in your learning and development, and honest with yourself about what you truly value and strive for it. 

The post Belong @ DataRobot: Celebrating 2024 Women’s History Month with DataRobot AI Legends appeared first on DataRobot AI Platform.

A vector embedding model is responsible for the transformation of unstructured data (text, images, audio, video) into a vector of numbers that capture semantic similarity between data objects. Embedding models are widely used beyond RAG applications, including recommendation systems, search engines, databases, and other data processing systems. 

Understanding their purpose, internals, advantages, and disadvantages is crucial and that’s what we’ll cover today. While we’ll be discussing text embedding models only, models for other types of unstructured data work similarly.

What Is an Embedding Model?

Machine learning models don’t work with text directly, they require numbers as input. Since text is ubiquitous, over time, the ML community developed many solutions that handle the conversion from text to numbers. There are many approaches of varying complexity, but we’ll review just some of them.

A simple example is one-hot encoding: treat words of a text as categorical variables and map each word to a vector of 0s and single 1.

Unfortunately, this embedding approach is not very practical, since it leads to a large number of unique categories and results in unmanageable dimensionality of output vectors in most practical cases. Also, one-hot encoding does not put similar vectors closer to one another in a vector space.

Embedding models were invented to tackle these issues. Just like one-hot encoding, they take text as input and return vectors of numbers as output, but they are more complex as they are taught with supervised tasks, often using a neural network. A supervised task can be, for example, predicting product review sentiment score. In this case, the resulting embedding model would place reviews of similar sentiment closer to each other in a vector space. The choice of a supervised task is critical to producing relevant embeddings when building an embedding model.

On the diagram above we can see word embeddings only, but we often need more than that since human language is more complex than just many words put together. Semantics, word order, and other linguistic parameters should all be taken into account, which means we need to take it to the next level – sentence embedding models. 

Sentence embeddings associate an input sentence with a vector of numbers, and, as expected, are way more complex internally since they have to capture more complex relationships.

Thanks to progress in deep learning, all state-of-the-art embedding models are created with deep neural nets, since they better capture complex relationships inherent to a human language.

A good embedding model should: 

• Be fast since often it is just a preprocessing step in a larger application
• Return vectors of manageable dimensions
• Return vectors that capture enough information about similarity to be practical

Let’s now quickly look into how most embedding models are organized internally.

Modern Neural Networks Architecture

As we just mentioned, all well-performing state-of-the-art embedding models are deep neural networks. 

This is an actively developing field and most top performing models are associated with some novel architecture improvement. Let’s briefly cover two very important architectures: BERT and GPT.

BERT (Bidirectional Encoder Representations from Transformers) was published in 2018 by researchers at Google and described the application of the bidirectional training of “transformer”, a popular attention model, to language modeling. Standard transformers include two separate mechanisms: an encoder for reading text input and a decoder that makes a prediction. 

BERT uses an encoder that reads the entire sentence of words at once which allows the model to learn the context of a word based on all of its surroundings, left and right unlike legacy approaches that looked at a text sequence from left to right or right to left. Before feeding word sequences into BERT, some words are replaced with [MASK] tokens and then the model attempts to predict the original value of the masked words, based on the context provided by the other, non-masked words in the sequence.  

Standard BERT does not perform very well in most benchmarks and BERT models require task-specific fine-tuning. But it is open-source, has been around since 2018, and has relatively modest system requirements (can be trained on a single medium-range GPU). As a result, it became very popular for many text-related tasks. It is fast, customizable, and small. For example, a very popular all-Mini-LM model is a modified version of BERT.

GPT (Generative Pre-Trained Transformer) by OpenAI is different. Unlike BERT, It is unidirectional, i.e. text is processed in one direction and uses a decoder from a transformer architecture that is suitable for predicting the next word in a sequence. These models are slower and produce very high dimensional embeddings, but they usually have many more parameters, do not require fine-tuning, and are more applicable to many tasks out of the box. GPT is not open source and is available as a paid API.

Context Length and Training Data

Another important parameter of an embedding model is context length. Context length is the number of tokens a model can remember when working with a text. A longer context window allows the model to understand more complex relationships within a wider body of text. As a result, models can provide outputs of higher quality, e.g. capture semantic similarity better.

To leverage a longer context, training data should include longer pieces of coherent text: books, articles, and so on. However, increasing context window length increases the complexity of a model and increases compute and memory requirements for training. 

There are methods that help manage resource requirements e.g. approximate attention, but they do this at a cost to quality. That’s another trade-off that affects quality and costs: larger context lengths capture more complex relationships of a human language, but require more resources.

Also, as always, the quality of training data is very important for all models. Embedding models are no exception. 

Semantic Search and Information Retrieval

Using embedding models for semantic search is a relatively new approach. For decades, people used other technologies: boolean models, latent semantic indexing (LSI), and various probabilistic models.

Some of these approaches work reasonably well for many existing use cases and are still widely used in the industry. 

One of the most popular traditional probabilistic models is BM25 (BM is “best matching”), a search relevance ranking function. It is used to estimate the relevance of a document to a search query and ranks documents based on the query terms from each indexed document. Only recently have embedding models started consistently outperforming it, but BM25 is still used a lot since it is simpler than using embedding models, it has lower computer requirements, and the results are explainable.

Benchmarks

Not every model type has a comprehensive evaluation approach that helps to choose an existing model. 

Fortunately, text embedding models have common benchmark suites such as:

• BEIR

The article “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models” proposed a reference set of benchmarks and datasets for information retrieval tasks. The original BEIR benchmark consists of a set of 19 datasets and methods for search quality evaluation. Methods include: question-answering, fact-checking, and entity retrieval. Now anyone who releases a text embedding model for information retrieval tasks can run the benchmark and see how their model ranks against the competition.

• MTEB

Massive Text Embedding Benchmarks include BEIR and other components that cover 58 datasets and 112 languages. The public leaderboard for MTEB results can be found here.

These benchmarks have been run on a lot of existing models and their leaderboards are very useful to make an informed choice about model selection.

Using Embedding Models in a Production Environment

Benchmark scores on standard tasks are very important, but they represent only one dimension.

When we use an embedding model for search, we run it twice:

• When doing offline indexing of available data
• When embedding a user query for a search request 

There are two important consequences of this. 

The first is that we have to reindex all existing data when we change or upgrade an embedding model. All systems built using embedding models should be designed with upgradability in mind because newer and better models are released all the time and, most of the time, upgrading a model is the easiest way to improve overall system performance. An embedding model is a less stable component of the system infrastructure in this case.

The second consequence of using an embedding model for user queries is that the inference latency becomes very important when the number of users goes up. Model inference takes more time for better-performing models, especially if they require GPU to run: having latency higher than 100ms for a small query is not unheard of for models that have more than 1B parameters. It turns out that smaller, leaner models are still very important in a higher-load production scenario. 

The tradeoff between quality and latency is real and we should always remember about it when choosing an embedding model.

As we have mentioned above, embedding models help manage output vector dimensionality which affects the performance of many algorithms downstream. Generally the smaller the model, the shorter the output vector length, but, often, it is still too great for smaller models. That’s when we need to use dimensionality reduction algorithms such as PCA (principal component analysis), SNE / tSNE (stochastic neighbor embedding), and UMAP (uniform manifold approximation). 

Another place we can use dimensionality reduction is before storing embeddings in a database. Resulting vector embeddings will occupy less space and retrieval speed will be faster, but will come at a price for the quality downstream. Vector databases are often not the primary storage, so embeddings can be regenerated with better precision from the original source data. Their use helps to reduce the output vector length and, as a result, makes the system faster and leaner.

Making the Right Choice

There’s an abundance of factors and trade-offs that should be considered when choosing an embedding model for a use case. The score of a potential model in common benchmarks is important, but we should not forget that it’s the larger models that have a better score. Larger models have higher inference time which can severely limit their use in low latency scenarios as often an embedding model is a pre-processing step in a larger pipeline. Also, larger models require GPUs to run. 

If you intend to use a model in a low-latency scenario, it’s better to focus on latency first and then see which models with acceptable latency have the best-in-class performance. Also, when building a system with an embedding model you should plan for changes since better models are released all the time and often it’s the simplest way to improve the performance of your system.

The post Choosing the Right Vector Embedding Model for Your Generative AI Use Case appeared first on DataRobot AI Platform.

Our employee business resource community, BEACON, embarked on a purposeful journey to celebrate and amplify the voices of Black artists. This theme served as the guiding light for BEACON’s initiatives throughout the month, creating a unique and dynamic space for exploration, education, and celebration.

Personally, the exploration of Black Art during this month has been a journey of discovery and connection. The infusion of African, Caribbean, and Black American lived experiences in art has provided a lens through which to appreciate the depth of cultural narratives, the resilience of communities, and the celebration of identity.

It’s a reminder that art is not merely an expression; it is a testament to the collective experiences that shape our perceptions of the world.

With BEACON, the celebration of Black History Month took on a unique and dynamic form. This year, we chose to amplify the voices of Black employees at DataRobot through a dedicated podcast focused on the theme of Black Art. The episode delved into the intricate stories behind various art forms, exploring how they serve as vessels for cultural preservation, activism, and personal expression. 

In producing the podcast episode, we witnessed firsthand the power of storytelling in fostering understanding and unity. The narratives shared by our guests illuminated the often-overlooked aspects of Black art, showcasing its ability to challenge stereotypes, inspire change, and foster a sense of pride within the community.

As we bid farewell to Black History Month, the echoes of the conversations, stories, and artistic expressions linger. The theme of Black Art has left an indelible mark on our collective consciousness, catalyzing continued exploration, dialogue, and celebration of the rich tapestry that makes up the Black experience. The journey does not end here; it’s a continuous exploration of the interconnectedness of our stories and the profound impact of Black Art on shaping a more inclusive and understanding world.

The post Reflecting on the Richness of Black Art appeared first on DataRobot AI Platform.

You have to deliver value from generative AI (GenAI) to keep the board happy and stay ahead of the competition. But you also have to stay on top of the growing chaos, as new tools and ecosystems arrive on the market. 

You also have to juggle new GenAI projects, use cases, and enthusiastic users across the organization. Oh, and data security. Your leadership doesn’t want to be the next cautionary tale of good AI gone bad. 

If you’re being asked to prove ROI for GenAI but it feels more like you’re playing Whack-a-Mole, you’re not alone. 

According to Deloitte, proving AI’s business value is the top challenge for AI leaders. Companies across the globe are struggling to move past prototyping to production. So, here’s how to get it done — and what you need to watch out for.  

6 Roadblocks (and Solutions) to Realizing Business Value from GenAI

Roadblock #1. You Set Yourself Up For Vendor Lock-In 

GenAI is moving crazy fast. New innovations — LLMs, vector databases, embedding models — are being created daily. So getting locked into a specific vendor right now doesn’t just risk your ROI a year from now. It could literally hold you back next week.  

Let’s say you’re all in on one LLM provider right now. What if costs rise and you want to switch to a new provider or use different LLMs depending on your specific use cases? If you’re locked in, getting out could eat any cost savings that you’ve generated with your AI initiatives — and then some. 

Solution: Choose a Versatile, Flexible Platform 

Prevention is the best cure. To maximize your freedom and adaptability, choose solutions that make it easy for you to move your entire AI lifecycle, pipeline, data, vector databases, embedding models, and more – from one provider to another. 

For instance, DataRobot gives you full control over your AI strategy — now, and in the future. Our open AI platform lets you maintain total flexibility, so you can use any LLM, vector database, or embedding model – and swap out underlying components as your needs change or the market evolves, without breaking production. We even give our customers the access to experiment with common LLMs, too.

Roadblock #2. Off-the-Grid Generative AI Creates Chaos 

If you thought predictive AI was challenging to control, try GenAI on for size. Your data science team likely acts as a gatekeeper for predictive AI, but anyone can dabble with GenAI — and they will. Where your company might have 15 to 50 predictive models, at scale, you could well have 200+ generative AI models all over the organization at any given time. 

Worse, you might not even know about some of them. “Off-the-grid” GenAI projects tend to escape leadership purview and expose your organization to significant risk. 

While this enthusiastic use of AI can be a recipe for greater business value, in fact, the opposite is often true. Without a unifying strategy, GenAI can create soaring costs without delivering meaningful results. 

Solution: Manage All of Your AI Assets in a Unified Platform

Fight back against this AI sprawl by getting all your AI artifacts housed in a single, easy-to-manage platform, regardless of who made them or where they were built. Create a single source of truth and system of record for your AI assets — the way you do, for instance, for your customer data. 

Once you have your AI assets in the same place, then you’ll need to apply an LLMOps mentality: 

• Create standardized governance and security policies that will apply to every GenAI model. 
• Establish a process for monitoring key metrics about models and intervening when necessary.
• Build feedback loops to harness user feedback and continuously improve your GenAI applications. 

DataRobot does this all for you. With our AI Registry, you can organize, deploy, and manage all of your AI assets in the same location – generative and predictive, regardless of where they were built. Think of it as a single source of record for your entire AI landscape – what Salesforce did for your customer interactions, but for AI. 

Roadblock #3. GenAI and Predictive AI Initiatives Aren’t Under the Same Roof

If you’re not integrating your generative and predictive AI models, you’re missing out. The power of these two technologies put together is a massive value driver, and businesses that successfully unite them will be able to realize and prove ROI more efficiently.

Here are just a few examples of what you could be doing if you combined your AI artifacts in a single unified system:  

• Create a GenAI-based chatbot in Slack so that anyone in the organization can query predictive analytics models with natural language (Think, “Can you tell me how likely this customer is to churn?”). By combining the two types of AI technology, you surface your predictive analytics, bring them into the daily workflow, and make them far more valuable and accessible to the business.

• Use predictive models to control the way users interact with generative AI applications and reduce risk exposure. For instance, a predictive model could stop your GenAI tool from responding if a user gives it a prompt that has a high probability of returning an error or it could catch if someone’s using the application in a way it wasn’t intended.  

• Set up a predictive AI model to inform your GenAI responses, and create powerful predictive apps that anyone can use. For example, your non-tech employees could ask natural language queries about sales forecasts for next year’s housing prices, and have a predictive analytics model feeding in accurate data.   

• Trigger GenAI actions from predictive model results. For instance, if your predictive model predicts a customer is likely to churn, you could set it up to trigger your GenAI tool to draft an email that will go to that customer, or a call script for your sales rep to follow during their next outreach to save the account. 

However, for many companies, this level of business value from AI is impossible because they have predictive and generative AI models siloed in different platforms. 

Solution: Combine your GenAI and Predictive Models 

With a system like DataRobot, you can bring all your GenAI and predictive AI models into one central location, so you can create unique AI applications that combine both technologies. 

Not only that, but from inside the platform, you can set and track your business-critical metrics and monitor the ROI of each deployment to ensure their value, even for models running outside of the DataRobot AI Platform.

Roadblock #4. You Unknowingly Compromise on Governance

For many businesses, the primary purpose of GenAI is to save time — whether that’s reducing the hours spent on customer queries with a chatbot or creating automated summaries of team meetings. 

However, this emphasis on speed often leads to corner-cutting on governance and monitoring. That doesn’t just set you up for reputational risk or future costs (when your brand takes a major hit as the result of a data leak, for instance.) It also means that you can’t measure the cost of or optimize the value you’re getting from your AI models right now. 

Solution: Adopt a Solution to Protect Your Data and Uphold a Robust Governance Framework

To solve this issue, you’ll need to implement a proven AI governance tool ASAP to monitor and control your generative and predictive AI assets. 

A solid AI governance solution and framework should include:

• Clear roles, so every team member involved in AI production knows who is responsible for what
• Access control, to limit data access and permissions for changes to models in production at the individual or role level and protect your company’s data
• Change and audit logs, to ensure legal and regulatory compliance and avoid fines 
• Model documentation, so you can show that your models work and are fit for purpose
• A model inventory to govern, manage, and monitor your AI assets, irrespective of deployment or origin

Current best practice: Find an AI governance solution that can prevent data and information leaks by extending LLMs with company data.

The DataRobot platform includes these safeguards built-in, and the vector database builder lets you create specific vector databases for different use cases to better control employee access and make sure the responses are super relevant for each use case, all without leaking confidential information.

Roadblock #5. It’s Tough To Maintain AI Models Over Time

Lack of maintenance is one of the biggest impediments to seeing business results from GenAI, according to the same Deloitte report mentioned earlier. Without excellent upkeep, there’s no way to be confident that your models are performing as intended or delivering accurate responses that’ll help users make sound data-backed business decisions.

In short, building cool generative applications is a great starting point — but if you don’t have a centralized workflow for tracking metrics or continuously improving based on usage data or vector database quality, you’ll do one of two things:

1. Spend a ton of time managing that infrastructure.
2. Let your GenAI models decay over time. 

Neither of those options is sustainable (or secure) long-term. Failing to guard against malicious activity or misuse of GenAI solutions will limit the future value of your AI investments almost instantaneously.

Solution: Make It Easy To Monitor Your AI Models

To be valuable, GenAI needs guardrails and steady monitoring. You need the AI tools available so that you can track: 

• Employee and customer-generated prompts and queries over time to ensure your vector database is complete and up to date
• Whether your current LLM is (still) the best solution for your AI applications 
• Your GenAI costs to make sure you’re still seeing a positive ROI
• When your models need retraining to stay relevant

DataRobot can give you that level of control. It brings all your generative and predictive AI applications and models into the same secure registry, and lets you:  

• Set up custom performance metrics relevant to specific use cases
• Understand standard metrics like service health, data drift, and accuracy statistics
• Schedule monitoring jobs
• Set custom rules, notifications, and retraining settings. If you make it easy for your team to maintain your AI, you won’t start neglecting maintenance over time. 

Roadblock #6. The Costs are Too High – or Too Hard to Track 

Generative AI can come with some serious sticker shock. Naturally, business leaders feel reluctant to roll it out at a sufficient scale to see meaningful results or to spend heavily without recouping much in terms of business value. 

Keeping GenAI costs under control is a huge challenge, especially if you don’t have real oversight over who is using your AI applications and why they’re using them. 

Solution: Track Your GenAI Costs and Optimize for ROI

You need technology that lets you monitor costs and usage for each AI deployment. With DataRobot, you can track everything from the cost of an error to toxicity scores for your LLMs to your overall LLM costs. You can choose between LLMs depending on your application and optimize for cost-effectiveness. 

That way, you’re never left wondering if you’re wasting money with GenAI — you can prove exactly what you’re using AI for and the business value you’re getting from each application. 

Deliver Measurable AI Value with DataRobot 

Proving business value from GenAI is not an impossible task with the right technology in place. A recent economic analysis by the Enterprise Strategy Group found that DataRobot can provide cost savings of 75% to 80% compared to using existing resources, giving you a 3.5x to 4.6x expected return on investment and accelerating time to initial value from AI by up to 83%. 

DataRobot can help you maximize the ROI from your GenAI assets and: 

• Mitigate the risk of GenAI data leaks and security breaches 
• Keep costs under control
• Bring every single AI project across the organization into the same place
• Empower you to stay flexible and avoid vendor lock-in 
• Make it easy to manage and maintain your AI models, regardless of origin or deployment 

If you’re ready for GenAI that’s all value, not all talk, start your free trial today. 

The post 6 Reasons Why Generative AI Initiatives Fail and How to Overcome Them appeared first on DataRobot AI Platform.

In this blog, we’ll explore the power of our DEIB communities as one transformative force, delve into the importance of creating spaces that embrace everyone’s unique narratives, and share practical resolutions to help us all weave the principles of diversity, equity, inclusion, and belonging into the fabric of our daily lives. As we step into the unknown of this new year, let’s embark on a journey together—one that champions unity, celebrates differences, and shapes a brighter, more inclusive tomorrow for all.

In 2023, the Belong Community Leaders came together to create a space for meaningful discussions on relevant topics called Beyond Differences. This series of internal events is hosted by the Belong communities and open to ALL. The goal is to be Better Together by driving conversation, initiating action, accelerating progress, and facilitating impact.

We had two roundtables discussing cultural differences and diverse networking, and one guest speaker presenting on the topic of cultural competency. As we look forward into 2024, we plan to continue strengthening our communities, fostering real connection, and facilitating empathy and understanding by looking beyond our differences. 

Like last year, we asked our communities what their goals and resolutions are for the coming year.  

ACTnow! advocates for the diverse needs of all Asian, Asian American, and Pacific Islander teammates through educational, cultural and social activities. Leader Alex Shoop (he/him).

This year, we celebrate the year of the dragon which symbolizes fortune, resilience, and strength. On behalf of the ACTnow! community, we wish you a happy 2024! We will continue cultivating a space to celebrate Asian traditions, share cultural learnings, and spotlight our diverse employees.

ADAPT provides education and allyship to advocate for and empower our teammates with disabilities to ensure an inclusive work environment. Leader Chad Harness (he/him).

To enhance the inclusivity and engagement within our ADAPT community, we will initiate weekly discussions centered around “AI and Machine Learning for All.” Our goal is to stimulate active participation, with the aim of increasing the weekly engagement and maintaining active involvement throughout the year. We will collaborate with DataRobot’s AI experts to curate relevant content and insights tailored to inclusivity in AI, fostering a more vibrant and interactive support group.

In order to strengthen advocacy and support within our ADAPT community, we will establish a dedicated space where members can confidentially share concerns or insights related to accessibility and inclusivity. Our goal is to provide timely response or acknowledgment to these insights and concerns.

To further enhance mental health support within our ADAPT community, we will expand the existing resources, offering tech-specific insights and coping strategies.

BEACON aims to advance a diverse, inclusive, and equitable community that fosters a culture of belonging for Black teammates both current and future. Leader Lelia Colley (she/her).

In 2023, BEACON ended the year with a focus on community and connection by kicking off our Pages of Inclusion Book Club. These sessions allowed for intentional thought collaboration and to become familiar with Black authors and refreshing concepts. 

When it comes to our focus of 2024, we aim to create a more inclusive, supportive, and empowering environment for black employees and allies at DataRobot by establishing initiatives to support the mental health and well-being of black employees and allies, including mental health resources and open discussions: 

• Fostering a supportive network among black employees and allies through regular networking events, cross-functional collaborations, and knowledge-sharing forums. 
• Implementing programs to recognize and celebrate the achievements and contributions of black employees and allies within the organization. Establishing safe spaces for open dialogues where BEACON members can share experiences, concerns, and ideas to foster a more supportive community.
• Promoting allyship and inclusivity by providing resources and training to DataRobot employees, encouraging active support of marginalized communities.
• Actively participating in recruitment efforts by collaborating with the People team to identify strategies for attracting and retaining black talent. 

LATTITUD is dedicated to connecting the Latin/Hispanic community in a supportive and uplifting environment while creating space to share ideas, struggles, resources, and celebrate our diverse cultures and accomplishments. Leader Lisa Aguilar (she/her).

PrideBots provides an open, safe, inclusive community where members can connect on common interests or backgrounds and celebrate all sexes, gender identities, gender expressions, and orientations. Leader Em Radkowski (she/her).

Pridebots resolve to lean further into affirming the identities of LGBTQIA+ and questioning team members. We commit to acknowledging and raising awareness about the diverse lived experiences of LGBTQIA+ individuals, recognizing the impact of factors such as race, religion, ethnicity, age, ability status, social class, and other social characteristics.

As part of this commitment, our goal is to advise the Executive Leadership Team (ELT) on opportunities and challenges related to LGBTQIA+ team members. We will advocate for inclusive policies and practices within the organization. Additionally, our community aims to review our health benefits and advise our organization to ensure that all individuals, regardless of their identity, can benefit from comprehensive healthcare services. This goal aligns with our broader commitment to creating a workplace that values diversity and prioritizes the well-being of all employees.

DataRobot Veterans brings together those who have served in all branches of the military for ongoing resources, support and networking. Leader Robert Newsom (he/him).

2024 will be the year when the DataRobot Veterans community launches, starting with an effort to increase channel membership and a poll to identify which components our veterans belong to. We want to give presentations on topics of interest to the community, such as the PACT Act and any company policies regarding mandatory service and recall to active duty.

Women @ DR seeks to create, promote and expand an inclusive culture that connects, educates and advances the needs, professional goals and aspirations of our community of female-identifying members and allies. Leader Teresa Gearin (she/her).

Women@DR is committed to laying a solid foundation for a future mentorship program in 2024, while simultaneously enhancing the overall experience of women within the organization. Our focus is on connecting the community, building allyship, and shining a light on gaps in equity and inclusion. This groundwork will contribute to a more supportive and inclusive workplace for women at every stage of their careers. 

Promoting Diversity, Equity, Inclusion, and Belonging (DEIB)

Promoting Diversity, Equity, Inclusion, and Belonging (DEIB) is a collective effort to foster a culture that values and respects differences, for all individuals, at all levels. Here are a few ways you can join this effort in small everyday actions. 

Educate Yourself: Take the initiative to educate yourself on issues related to diversity, equity, and inclusion. Read books, articles, and attend workshops to broaden your understanding.

Listen Actively: Listen to the experiences and perspectives of people from different backgrounds without judgment. Actively seek to understand the challenges faced by others and be empathetic.

Challenge Stereotypes: Speak up when you encounter stereotypes or biased statements. Challenge misconceptions and promote a more accurate understanding of diverse groups.

Use Inclusive Language: Be mindful of the language you use and strive to use inclusive terminology. Avoid making assumptions about people based on stereotypes or preconceived notions.

Amplify Others: Amplify the voices of those who may be marginalized or underrepresented. Give credit where it is due and acknowledge the contributions of others.

Advocate for Inclusive Policies: Support and advocate for workplace policies that promote diversity, equity, and inclusion. Encourage your organization to adopt practices that create a more inclusive environment.

Call Out Microaggressions: Address microaggressions when you witness them, even if they are subtle. Help create an environment where people feel safe and respected.

Engage in Allyship: Be an ally to individuals from marginalized groups by actively supporting and standing up for them. Use your privilege to advocate for equal opportunities.

Embrace Lifelong Learning: Recognize that promoting DEIB is an ongoing process of learning and unlearning.Stay open to new ideas and be willing to adapt your perspectives based on new information.

Belong @ DataRobot is committed to continuing our journey in Diversity, Equity, Inclusion, and Belonging in 2024 by presenting opportunities for the people of DataRobot to actively participate in community, events, education, conversations, and self reflection. We wish everyone a gentle and prosperous 2024. 

The post Beyond Differences, Embracing the Journey: A New Year’s Resolution for a Better Tomorrow appeared first on DataRobot AI Platform.

Many organizations are now exploring the power of generative AI to improve their efficiency and gain new capabilities. In most cases, to fully unlock these powers, AI must have access to the relevant enterprise data. Large Language Models (LLMs) are trained on publicly available data (e.g. Wikipedia articles, books, web index, etc.), which is enough for many general-purpose applications, but there are plenty of others that are highly dependent on private data, especially in enterprise environments.

There are three main ways to provide new data to a model:

1. Pre-training a model from scratch. This rarely makes sense for most companies because it is very expensive and requires a lot of resources and technical expertise.
2. Fine-tuning an existing general-purpose LLM. This can reduce the resource requirements compared to pre-training, but still requires significant resources and expertise. Fine-tuning produces specialized models that have better performance in a domain for which it is finetuned for but may have worse performance in others. 
3. Retrieval augmented generation (RAG). The idea is to fetch data relevant to a query and include it in the LLM context so that it could “ground” its own outputs in that information. Such relevant data in this context is referred to as “grounding data”. RAG complements generic LLM models, but the amount of information that can be provided is limited by the LLM context window size (amount of text the LLM can process at once, when the information is generated).

Currently, RAG is the most accessible way to provide new information to an LLM, so let’s focus on this method and dive a little deeper.

Retrieval Augmented Generation 

In general, RAG means using a search or retrieval engine to fetch a relevant set of documents for a specified query. 

For this purpose, we can use many existing systems: a full-text search engine (like Elasticsearch + traditional information retrieval techniques), a general-purpose database with a vector search extension (Postgres with pgvector, Elasticsearch with vector search plugin), or a specialized database that was created specifically for vector search.

In two latter cases, RAG is similar to semantic search. For a long time, semantic search was a highly specialized and complex domain with exotic query languages and niche databases. Indexing data required extensive preparation and building knowledge graphs, but recent progress in deep learning has dramatically changed the landscape. Modern semantic search applications now depend on embedding models that successfully learn semantic patterns in presented data. These models take unstructured data (text, audio, or even video) as input and transform them into vectors of numbers of a fixed length, thus turning unstructured data into a numeric form that could be used for calculations Then it becomes  possible to calculate the distance between vectors using a chosen distance metric, and the resulting distance will reflect the semantic similarity between vectors and, in turn, between pieces of original data.

These vectors are indexed by a vector database and, when querying, our query is also transformed into a vector. The database searches for the N closest vectors (according to a chosen distance metric like cosine similarity) to a query vector and returns them.

A vector database is responsible for these 3 things:

1. Indexing. The database builds an index of vectors using some built-in algorithm (e.g. locality-sensitive hashing (LSH) or hierarchical navigable small world (HNSW)) to precompute data to speed up querying.
2. Querying. The database uses a query vector and an index to find the most relevant vectors in a database.
3. Post-processing. After the result set is formed, sometimes we might want to run an additional step like metadata filtering or re-ranking within the result set to improve the outcome.

The purpose of a vector database is to provide a fast, reliable, and efficient way to store and query data. Retrieval speed and search quality can be influenced by the selection of index type. In addition to the already mentioned LSH and HNSW there are others, each with its own set of strengths and weaknesses. Most databases make the choice for us, but in some, you can choose an index type manually to control the tradeoff between speed and accuracy.

At DataRobot, we believe the technique is here to stay. Fine-tuning can require very sophisticated data preparation to turn raw text into training-ready data, and it’s more of an art than a science to coax LLMs into “learning” new facts through fine-tuning while maintaining their general knowledge and instruction-following behavior. 

LLMs are typically very good at applying knowledge supplied in-context, especially when only the most relevant material is provided, so a good retrieval system is crucial.

Note that the choice of the embedding model used for RAG is essential. It is not a part of the database and choosing the correct embedding model for your application is critical for achieving good performance. Additionally, while new and improved models are constantly being released, changing to a new model requires reindexing your entire database.

Evaluating Your Options 

Choosing a database in an enterprise environment is not an easy task. A database is often the heart of your software infrastructure that manages a very important business asset: data.

Generally, when we choose a database we want:

• Reliable storage
• Efficient querying 
• Ability to insert, update, and delete data granularly (CRUD)
• Set up multiple users with various levels of access for them (RBAC)
• Data consistency (predictable behavior when modifying data)
• Ability to recover from failures
• Scalability to the size of our data

This list is not exhaustive and might be a bit obvious, but not all new vector databases have these features. Often, it is the availability of enterprise features that determine the final choice between a well-known mature database that provides vector search via extensions and a newer vector-only database. 

Vector-only databases have native support for vector search and can execute queries very fast, but often lack enterprise features and are relatively immature. Keep in mind that it takes years to build complex features and battle-test them, so it’s no surprise that early adopters face outages and data losses. On the other hand, in existing databases that provide vector search through extensions, a vector is not a first-class citizen and query performance can be much worse. 

We will categorize all current databases that provide vector search into the following groups and then discuss them in more detail:

• Vector search libraries
• Vector-only databases
• NoSQL databases with vector search 
• SQL databases with vector search 
• Vector search solutions from cloud vendors

Vector search libraries

Vector search libraries like FAISS and ANNOY are not databases – rather, they provide in-memory vector indices, and only limited data persistence options. While these features are not ideal for users requiring a full enterprise database, they have very fast nearest neighbor search and are open source. They offer good support for high-dimensional data and are highly configurable (you can choose the index type and other parameters). 

Overall, they are good for prototyping and integration in simple applications, but they are inappropriate for long-term, multi-user data storage. 

Vector-only databases 

This group includes diverse products like Milvus, Chroma, Pinecone, Weaviate, and others. There are notable differences among them, but all of them are specifically designed to store and retrieve vectors. They are optimized for efficient similarity search with indexing and support high-dimensional data and vector operations natively. 

Most of them are newer and might not have the enterprise features we mentioned above, e.g. some of them don’t have CRUD, no proven failure recovery, RBAC, and so on. For the most part, they can store the raw data, the embedding vector, and a small amount of metadata, but they can’t store other index types or relational data, which means you will have to use another, secondary database and maintain consistency between them. 

Their performance is often unmatched and they are a good option when having multimodal data (images, audio or video).

NoSQL databases with vector search 

Many so-called NoSQL databases recently added vector search to their products, including MongoDB, Redis, neo4j, and ElasticSearch. They offer good enterprise features, are mature, and have a strong community, but they provide vector search functionality via extensions which might lead to less than ideal performance and lack of first-class support for vector search. Elasticsearch stands out here as it is designed for full-text search and already has many traditional information retrieval features that can be used in conjunction with vector search.

NoSQL databases with vector search are a good choice when you are already invested in them and need vector search as an additional, but not very demanding feature.

SQL databases with vector search 

This group is somewhat similar to the previous group, but here we have established players like PostgreSQL and ClickHouse. They offer a wide array of enterprise features, are well-documented, and have strong communities. As for their disadvantages, they are designed for structured data, and scaling them requires specific expertise. 

Their use case is also similar: good choice when you already have them and the expertise to run them in place.

Vector search solutions from cloud vendors

Hyperscalers also offer vector search services. They usually have basic features for vector search (you can choose an embedding model, index type, and other parameters), good interoperability within the rest of the cloud platform, and more flexibility when it comes to cost, especially if you use other services on their platform. However, they have different maturity and different feature sets: Google Cloud vector search uses a fast proprietary index search algorithm called ScaNN and metadata filtering, but is not very user-friendly; Azure Vector search offers structured search capabilities, but is in preview phase and so on. 

Vector search entities can be managed using enterprise features of their platform like IAM (Identity and Access Management), but they are not that simple to use and suited for general cloud usage. 

Making the Right Choice 

The main use case of vector databases in this context is to provide relevant information to a model. For your next LLM project, you can choose a database from an existing array of databases that offer vector search capabilities via extensions or from new vector-only databases that offer native vector support and fast querying. 

The choice depends on whether you need enterprise features, or high-scale performance, as well as your deployment architecture and desired maturity (research, prototyping, or production). One should also consider which databases are already present in your infrastructure and whether you have multimodal data. In any case, whatever choice you will make it is good to hedge it: treat a new database as an auxiliary storage cache, rather than a central point of operations, and abstract your database operations in code to make it easy to adjust to the next iteration of the vector RAG landscape.

How DataRobot Can Help

There are already so many vector database options to choose from. They each have their pros and cons – no one vector database will be right for all of your organization’s generative AI use cases. That is why it’s important to retain optionality and leverage a solution that allows you to customize your generative AI solutions to specific use cases, and adapt as your needs change or the market evolves. 

The DataRobot AI Platform lets you bring your own vector database – whichever is right for the solution you’re building. If you require changes in the future, you can swap out your vector database without breaking your production environment and workflows. 

The post Choosing the Right Database for Your Generative AI Use Case appeared first on DataRobot AI Platform.

This recommendation aims to outline the risks and proposed recommendations for how to assess and manage off-frontier AI models – typically referring to open source models.  In summary, the recommendation from the NAIAC provides a roadmap for responsibly navigating the complexities of generative AI. This blog post aims to shed light on this recommendation and delineate how DataRobot customers can proactively leverage the platform to align their AI adaption with this recommendation.

Frontier vs Off-Frontier Models

In the recommendation, the distinction between frontier and off-frontier models of generative AI is based on their accessibility and level of advancement. Frontier models represent the latest and most advanced developments in AI technology. These are complex, high-capability systems typically developed and accessed by leading tech companies, research institutions, or specialized AI labs (such as current state-of-the-art models like GPT-4 and Google Gemini). Due to their complexity and cutting-edge nature, frontier models typically have constrained access – they are not widely available or accessible to the general public.

On the other hand, off-frontier models typically have unconstrained access – they are more widely available and accessible AI systems, often available as open source. They might not achieve the most advanced AI capabilities but are significant due to their broader usage. These models include both proprietary systems and open source AI systems and are used by a wider range of stakeholders, including smaller companies, individual developers, and educational institutions.

This distinction is important for understanding the different levels of risks, governance needs, and regulatory approaches required for various AI systems. While frontier models may need specialized oversight due to their advanced nature, off-frontier models pose a different set of challenges and risks because of their widespread use and accessibility.

What the NAIAC Recommendation Covers

The recommendation on ‘Generative AI Away from the Frontier,’ issued by NAIAC in October 2023, focuses on the governance and risk assessment of generative AI systems. The document provides two key recommendations for the assessment of risks associated with generative AI systems:

For Proprietary Off-Frontier Models: It advises the Biden-Harris administration to encourage companies to extend voluntary commitments³ to include risk-based assessments of off-frontier generative AI systems. This includes independent testing, risk identification, and information sharing about potential risks. This recommendation is particularly aimed at emphasizing the importance of understanding and sharing the information on risks associated with off-frontier models.

For Open Source Off-Frontier Models: For generative AI systems with unconstrained access, such as open-source systems, the National Institute of Standards and Technology (NIST) is charged to collaborate with a diverse range of stakeholders to define appropriate frameworks to mitigate AI risks. This group includes academia, civil society, advocacy organizations, and the industry (where legal and technical feasibility allows). The goal is to develop testing and analysis environments, measurement systems, and tools for testing these AI systems. This collaboration aims to establish appropriate methodologies for identifying critical potential risks associated with these more openly accessible systems.

NAIAC underlines the need to understand the risks posed by widely available, off-frontier generative AI systems, which include both proprietary and open-source systems. These risks range from the acquisition of harmful information to privacy breaches and the generation of harmful content. The recommendation acknowledges the unique challenges in assessing risks in open-source AI systems due to the lack of a fixed target for assessment and limitations on who can test and evaluate the system.

Moreover, it highlights that investigations into these risks require a multi-disciplinary approach, incorporating insights from social sciences, behavioral sciences, and ethics, to support decisions about regulation or governance. While recognizing the challenges, the document also notes the benefits of open-source systems in democratizing access, spurring innovation, and enhancing creative expression.

For proprietary AI systems, the recommendation points out that while companies may understand the risks, this information is often not shared with external stakeholders, including policymakers. This calls for more transparency in the field.

Regulation of Generative AI Models

Recently, discussion on the catastrophic risks of AI has dominated the conversations on AI risk, especially with regards to generative AI. This has led to calls to regulate AI in an attempt to promote responsible development and deployment of AI tools. It is worth exploring the regulatory option with regards to generative AI. There are two main areas where policy makers can regulate AI: regulation at model level and regulation at use case level.

In predictive AI, generally, the two levels significantly overlap as narrow AI is built for a specific use case and cannot be generalized to many other use cases. For example, a model that was developed to identify patients with high likelihood of readmission, can only be used for this particular use case and will require input information similar to what it was trained on. However, a single large language model (LLM), a form of generative AI models, can be used in multiple ways to summarize patient charts, generate potential treatment plans, and improve the communication between the physicians and patients. 

As highlighted in the examples above, unlike predictive AI, the same LLM can be used in a variety of use cases. This distinction is particularly important when considering AI regulation. 

Penalizing AI models at the development level, especially for generative AI models, could hinder innovation and limit the beneficial capabilities of the technology. Nonetheless, it is paramount that the builders of generative AI models, both frontier and off-frontier, adhere to responsible AI development guidelines. 

Instead, the focus should be on the harms of such technology at the use case level, especially at governing the use more effectively. DataRobot can simplify governance by providing capabilities that enable users to evaluate their AI use cases for risks associated with bias and discrimination, toxicity and harm, performance, and cost. These features and tools can help organizations ensure that AI systems are used responsibly and aligned with their existing risk management processes without stifling innovation.

Governance and Risks of Open vs Closed Source Models

Another area that was mentioned in the recommendation and later included in the recently signed executive order signed by President Biden⁴, is lack of transparency in the model development process. In the closed-source systems, the developing organization may investigate and evaluate the risks associated with the developed generative AI models. However, information on potential risks, findings around outcome of red teaming, and evaluations done internally has not generally been shared publicly. 

On the other hand, open-source models are inherently more transparent due to their openly available design, facilitating the easier identification and correction of potential concerns pre-deployment. But extensive research on potential risks and evaluation of these models has not been conducted.

The distinct and differing characteristics of these systems imply that the governance approaches for open-source models should differ from those applied to closed-source models. 

Avoid Reinventing Trust Across Organizations

Given the challenges of adapting AI, there’s a clear need for standardizing the governance process in AI to prevent every organization from having to reinvent these measures. Various organizations including DataRobot have come up with their framework for Trustworthy AI⁵. The government can help lead the collaborative effort between the private sector, academia, and civil society to develop standardized approaches to address the concerns and provide robust evaluation processes to ensure development and deployment of trustworthy AI systems. The recent executive order on the safe, secure, and trustworthy development and use of AI directs NIST to lead this joint collaborative effort to develop guidelines and evaluation measures to understand and test generative AI models. The White House AI Bill of Rights and the NIST AI Risk Management Framework (RMF) can serve as foundational principles and frameworks for responsible development and deployment of AI. Capabilities of the DataRobot AI Platform, aligned with the NIST AI RMF, can assist organizations in adopting standardized trust and governance practices. Organizations can leverage these DataRobot tools for more efficient and standardized compliance and risk management for generative and predictive AI.

¹ National AI Advisory Committee – AI.gov 

² RECOMMENDATIONS: Generative AI Away from the Frontier

³ Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence | The White House

⁴ https://www.datarobot.com/trusted-ai-101/

The post Open Source AI Models – What the U.S. National AI Advisory Committee Wants You to Know appeared first on DataRobot AI Platform.