We live in a time in which the digitization of all our business processes is crucial if we want to have control over them and know which parts we can improve more effectively, being able to measure them through monitoring and implementing artificial intelligence solutions that would otherwise be impossible. To do this, we implement platforms and web applications that serve as a way for both members of the company and customers to interact with each other. But of course, if these tools fail, it is crucial that we know as soon as possible, since every minute that they are not operational or work erratically, impacts both working time and possible lost sales.
In order to reduce this time to a minimum, synthetic probes come into play, tools that interact with these platforms by mimicking human behavior. And of course, at Datadope, as good monitoring and data addicts, we have developed our own tool: Synthetix SaaS, a product of which we are very proud and of which we like to boast. But as we like to detect problems and anticipate them, we know for a fact that it is useless to have the tallest, most handsome, bluest-eyed and best value for money child if potential customers already have a solution implemented and do not want to risk the problems associated with a migration, such as time (most crucial in the case of a solution responsible for monitoring availability) and the costs of the same. So to solve this problem, we have created a Super Scholar capable of reducing migration times by 80% and costs to an anecdotal fact thanks to the power of LLMs.
But first of all, let’s give a bit of context and explain, for those who are a bit lost, what each thing is and how it works.
What Is a Probe?
Probes are automated programs that simulate the behavior of real users interacting with systems or applications. They can be configured to follow steps, fill in fields, click on areas of a webpage, log in, log out, and essentially perform all the actions a typical user would. These tools ensure that services are functioning optimally from the user’s perspective, delivering value to the business and peace of mind to the leadership.
At Datadope, we distinguish between two types of synthetic probes: API and BROWSER.
Types of Probes
API Probes
Sondas BROWSER
API Probes
API probes simulate interactions between an application and an API. They send requests to the API, measure response times, and evaluate performance, ensuring availability. These probes can collect data from responses in earlier steps (e.g., tokens, users) for use in subsequent steps. Additionally, they verify that responses meet expectations using assertions.
For example, suppose we are using an API that receives pictures of cats and dogs and determines whether the image depicts a cat or a dog. Before sending a photo, we must obtain a token by sending a username and password to the appropriate endpoint. An API probe verifying this API’s functionality would follow these steps:
- Send username and password to the login endpoint.
- Receive the token and pass it to the next step.
- Send a dog photo to the API, including the token obtained earlier, and await a response.
- If the response is “dog,” everything is working correctly. If the response differs or an error occurs, an alert is triggered.
This approach ensures not only the API’s availability but also that it functions as expected.
BROWSER Probes
BROWSER probes simulate user behavior in a web application (e.g., making a purchase, filling out a form, logging in). These probes help detect issues in the application across different browsers and devices, ensuring a consistent and satisfactory user experience. To pinpoint errors more effectively, a video of the execution process is recorded, allowing identification of weaknesses or failures.
What Is an LLM?
LLMs (Large Language Models) are generative AI models that take an input and produce a text-based output. To solve our problem, we leverage the capabilities of a powerful model like OpenAI’s GPT-4. This model is based on Transformer technology, which falls within the domain of Deep Learning.
That sounds great... But how does it work?
At this point, it would be very easy for me to bombard you with terminology that only die-hard enthusiasts in this field would be willing to endure. Instead, I’ll try to make it more digestible for those who might not have a highly technical background. To do this, I’ll compare how GPT-4 works to the protagonist of one of the most original movies ever made: Memento. (If you haven’t seen it yet, you’re welcome for the weekend plan! 😉
For those of you who haven’t seen this masterpiece by Christopher Nolan, here’s a link to Filmaffinity so you can read the synopsis and understand why I’m comparing the protagonist to the model: Memento.
Broadly speaking, GPT-4o receives an input, which can be a text or an image, and generates an output text. But for this to happen, it must have been previously trained. But with what exactly?
Training Data
LLMs learn to generate coherent and relevant text based on a massive corpus of training data. This data comprises collections of text that the model uses to identify patterns, linguistic structures, and meanings.
Characteristics of the Training Data
Just as humans internalize the expression “we are what we eat” (even though most of us ignore it completely), LLMs behave according to the data on which they are trained. That’s why, just as a balanced diet is essential for good health, creating a good model requires training data that meets these four key characteristics:
Diversity
The data should include a wide variety of texts (books, articles, websites) to cover multiple domains of knowledge.
Volume
The dataset must be extremely massive, containing trillions of words (or tokens), to ensure the model can generalize effectively.
Representativeness
The data must reflect the complexity of human language in various contexts and styles.
Curation
The data should be filtered and processed to ensure relevance and quality.
If we ensure that the training data meets these criteria, there’s a good chance of developing a model that performs well in general terms. And I know at this point you might be wondering what data GPT-4o has been trained on. Unfortunately, as with the latest previous models, OpenAI has not released detailed information about the training process. However, we can confirm that it has used public data such as:
- Public websites:
- Wikipedia, blogs, forums like Reddit.
- Technical reference sites and tutorials.
- Books and literature:
- Works in the public domain.
- Books available under open licenses.
- Technical documentation:
- Manuals, programming guides, and software documentation.
- Academic articles:
- Scientific and research texts accessible on the open web.
- Simulated dialogues:
- Synthetic collections designed to represent human conversations.
- News and current articles:
- Reports and analyses publicly available.
As expected, this data is finite. Once the model has been trained with this data, it can no longer continue learning. This means that after a specific cut-off date, the model knows absolutely nothing beyond that point (just like the protagonist in the movie). The training data for GPT-4o has a cut-off date of October 2023.
And now I know you’ll say, “Wait a second—yesterday I asked it for a list of the top 10 movies of this year… how can it know that if it only has data up to October 2023?” Well, this is where the concept of the “context window” comes into play.
Context Window
The context window is the number of tokens the model can process and consider simultaneously to generate its output. A token can be a word, part of a word, or a special character, depending on the tokenization method used (tokenization = converting input text into tokens). When the model generates a response or performs a task, it analyzes the tokens within this window to identify patterns, context, and relationships that shape its output.
What Capabilities Does the Context Window Provide?
Analytical Capacity: The larger the context window, the more prior text the model can consider to better understand the conversation or task. In GPT-4, the context window is 128,000 tokens—approximately the length of a 350-page book.
Short-Term Memory: The model uses the context window to “remember” information from previous interactions within a single block of text. For instance, when we have a conversation with ChatGPT, the entire conversation history is sent to the model with each message, allowing it to know what has been said previously.
Similarly, when you ask it about something that occurred after its training cutoff date, about current events, or for updated information, the application performs an internet search and provides the updated information as part of the context for the model to analyze. This works much like that one friend we all have who, when they hear something they don’t know, immediately Googles it to either agree with you or prove you wrong.
Practical Limitations
Unfortunately, there are practical limits to this. If you exceed the context window, the oldest tokens (the beginning of the text) are discarded, which can lead to the loss of relevant information.
Just like GPT, our protagonist has a context window of only 5 to 15 minutes. Any information beyond that time threshold is lost and cannot be remembered due to his anterograde amnesia. For this reason, the information he receives in those initial minutes is crucial and determines his future behavior. In GPT’s case, this initial information is called the prompt. But what exactly is a prompt?
Prompt
A prompt is the initial input provided to the model to generate a response. Essentially, it is the instruction or context that guides the model in the task it needs to perform. The prompt occupies part of the model’s context window. This means that:
- The more detailed the prompt, the greater the number of tokens consumed in the context window.
- A well-designed prompt can maximize the efficiency of the context window by providing all relevant information without wasting space.
A Good Prompt Should Have These 3 Characteristics:
- Precision: Reduces ambiguity, improving the quality of the responses.
- Efficiency: Minimizes unnecessary token usage within the context window.
- Control: Allows the user to guide the model’s tone, style, and focus.
Prompt Engineering Techniques
There are many techniques to improve prompts, and the goal of this article isn’t to cover all of them. Instead, I’ll give you a few examples to show how making certain changes to our prompts can lead to better results from our models.
For instance, we won’t get the same result by saying this:
That by telling them in this way, where we give them a role, a context and a clear and specific task:
Thinking Step by Step
This is a Prompt Engineering technique that alters the way the model reasons. Instead of giving a direct answer, it applies a step-by-step reasoning of the task, which can lead to a more accurate answer.
Let’s see how the model’s response changes just by incorporating this sentence:
And with the magic phrase:
Few-shot
This technique consists of giving a few examples of what we expect as a response based on what is requested.
One-shot
This technique consists of giving a single example of what we expect as a response based on what is requested.
There are more prompt engineering techniques, but I hope that with these examples you will understand the importance of the prompt in the results obtained. In the case of GPT-4o we can pass both images and text as a prompt.
But this is not the only way to modify the behavior of a model. There is a way to permanently modify it to fit what we are looking for: Fine-Tuning.
Fine-Tuning
Fine-Tuning consists of taking a previously trained model and re-training it to fit a new data set. In this way, we alter the behavior of the model permanently. Thanks to this, there would be no need to use prompting techniques, since we would have adjusted the model to our use case, but this technique requires a huge amount of data, increasing both the costs of generating a rich enough dataset for the model to fit the case we want and the costs involved in retraining some of these models (which can reach millions of dollars due to the hardware and energy required).
I’m not going to go into more depth on this part because OpenAI does not give the option to retrain GPT-4o. Let’s say that just like the tattoos that the main character in Memento gets on his body make him change his attitude permanently, the same thing happens with Fine-Tuning in LLMs.
All of This Sounds Great, But How Do I Interact with an LLM?
There are libraries that allow you to interact with various LLMs, some of which are official ones from the companies themselves. In this case, however, I used LangChain for the following reasons:
- Integration with multiple tools and models: LangChain makes it easy to integrate language models with a variety of services and tools, such as databases, third-party APIs, or even other AI models. This enables the creation of more complex and customized workflows without the need to manually manage every component.
- Ease of managing conversation state: One of LangChain’s standout features is its ability to maintain long contexts and manage the state of a conversation across multiple interactions—something that is challenging to handle with just the official library of a language model.
- Ease of creating agents and automated workflows: LangChain provides abstractions for building conversational agents that can interact with other APIs or databases, make decisions based on received data, and adapt to different environments. This simplifies the development of more robust applications without requiring you to implement all the logic yourself.
Alright, But What Did You Do with All This?
We’ve used all the technology I just described to reduce both the time and cost involved in migrating synthetic probes from various origins to one of our flagship products: Synthetix SaaS.
At Datadope, we understand how critical it is to ensure that there are no failures during a migration and that the process lasts only as long as absolutely necessary. That’s why we’ve used these tools to minimize both the impact and the costs of migration to the lowest possible levels.
In addition, we know how important it is to avoid wasting talent on repetitive tasks, and we want our skilled people to be where they can really add value. We believe that no one spends years training and building a career only to end up as Rick and Morty’s butter-passing robot.
I hope this short blog post has piqued your curiosity about LLMs and that you will start researching how you can apply this technology to your use case. Thank you very much for your time and don’t forget to check our blog from time to time, my colleagues have a lot of interesting things to tell you about.
