Introduction

When the first technical solutions based on Artificial Intelligence began to be created, Python was the language and runtime platform of choice. It was not a total surprise as Python was the choice of the great majority of data scientists to analyse data, perform experiments and create AI models, so it was kind of natural that AI solutions were also based on Python and its rich ecosystem for data-intensive applications.

However, with the rise of Generative AI, organisations worldwide reconsider that approach given that the vast majority of GenAI-based solutions are just leveraging existing Large Language Models consumed “as a service” via web APIs. For doing that, Python does not have the same lead over the others. We must balance other key aspects of enterprise-grade solutions, such as resilience, scalability, observability, as well as the leverage of existing skills in the organisation. That means that other key languages and platforms can play a leading role in delivering and running GenAI-based solutions. Platforms such as Node.js, Go, and, of course, Java.

In Java we already have multiple valid approaches to be considered.In this article I will cover Langchain4j, one of the most popular choices in the Java ecosystem for building and running AI solutions at scale.

Why Langchain4j? Key outsdanding aspects are:

1. Framework-agnostic: As a library it imposes no constraints about how you design and build your solutions, so it can be easily integrated with any existing solution, either Spring-based, Jakarta-based, Quarkus-based, Micronaut-based, or with no framework at all.
2. Simple yet powerful API: Based on well-known patterns such as the Builder pattern, and with simple API constructs so its learning curve is simple and rewarding:
3. It works with both cloud-based “as a service” models such as OpenAI or Google Vertex, and with local/owned models via Ollama.

NOTE: The following examples are based on Langchain4j 0.36.2.

The first Langchain4j program

To demonstrate these concepts, let’s look at a “hello world” Langchain4j program.

The main interface that we need to learn about is ChatLanguageModel (from package dev.langchain4j.model.chat). This interface has the simple API that we need to send messages to an LLM and get its response. To instantiate specific models to interact with them we need the specific implementation depending on who provides them:

• For OpenAI, we leverage OpenAiChatModel from package dev.langchain4j.model.openai.OpenAiChatModel.
• For Vertex, we leverage VertexAiGeminiChatModel from package dev.langchain4j.model.vertexai.
• For Ollama, we leverage OllamaChatModel from package dev.langchain4j.model.ollama.OllamaChatModel.

As well as others. Every implementation of ChatLanguageModel has its own builder pattern to be able to add any specific configuration setting that is needed. Let’s see three brief examples of how this looks once we put it together:

OpenAI Hello World

import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.openai.OpenAiChatModel;

class OpenAIHelloWorld {

    void main() {
        // OpenAI model
        ChatLanguageModel model = OpenAiChatModel.builder()
            .apiKey(System.getenv("OPENAI_API_KEY"))
            .modelName("gpt-4o")
            .build();

        // the first prompt
        String message = "Hello world!";
        System.out.println("\n>>> " + message);

        String answer = model.generate(message);
        System.out.println(answer);
    }
}

As can be seen above, to connect with OpenAI services you must provide your own API key. You can also add explicitly the model that you want to be used.

Vertex Hello World

import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.vertexai.VertexAiGeminiChatModel;

class VertexAIHelloWorld {
    void main() {
        // Vertex AI model
        ChatLanguageModel model = VertexAiGeminiChatModel.builder()
            .project(System.getenv("VERTEXAI_PROJECT_ID"))
            .location("us-central1")
            .modelName("gemini-2.5-flash")
            .build();

        // the first prompt
        String message = "Hello world!";
        System.out.println("\n>>> " + message);

        String answer = model.generate(message);
        System.out.println(answer);
    }
}

The pattern is similar to the previous but the settings that must be provided are different: the project id in Vertex, the cloud region and the model name.

Ollama Hello World

import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.ollama.OllamaChatModel;

class OllamaGptOssHelloWorld {
    void main() {
        // gpt-oss:20b model running locally with Ollama
        ChatLanguageModel model = OllamaChatModel.builder()
            .baseUrl("http://localhost:11434")
            .modelName("gpt-oss:20b")
            .build();

        // the first prompt
        String message = "Hello world!";
        System.out.println("\n>>> " + message);

        String answer = model.generate(message);
        System.out.println(answer);

    }
}

Again, the pattern is similar. In this case we are running Ollama in the local computer and using the gpt-oss:20b model, quite competent and runnable in many personal computers.

Managing the context (a.k.a. short-term chat memory)

While the previous examples work, they lack a critical feature that any GenAI solution would need. The first important concept that we need to understand is how to manage the context of the conversation with the LLMs, also known as the short-term memory.

In essence, what we must do is to track the whole conversation with the LLM (a.k.a. “the chat”). After every interaction, the pair question and answer are saved to be sent with the next request payload, typically models expect that under a specific history entry in the request body in Json format. Fortunately, Langchain4j deals with those details and we just focus on keeping track of the conversation. The simplest way to do that is to use a in-memory store, as we can see in the following example:

ChatLanguageModelmodel = OllamaChatModel.builder()
    .baseUrl(baseUrl)
    .modelName(modelName)
    .timeout(Duration.ofSeconds(300))
    .temperature(0.0)
    .build();

// define context window
ChatMemory chatMemory = MessageWindowChatMemory.withMaxMessages(10);

// initial prompt with name and what I'm doing
String message = "Hello world! My name is Jorge and I'm writing this for Java Advent 2025.";
chatMemory.add(userMessage(message));

AiMessage answer = model.generate(chatMemory.messages()).content();
System.out.println(answer.text());
chatMemory.add(answer);

// ask for the name
message = "What is my name?";
chatMemory.add(userMessage(message));

answer = model.generate(chatMemory.messages()).content();
System.out.println(answer.text());
chatMemory.add(answer);

As can be seen in the example, the whole chat memory is passed to the model. As we keep adding every message and answer into the memory, the LLM will leverage the whole conversation (up to the limits of the memory or its own internal context window, whatever comes first) to come up with the best possible answer.

The static function userMessage from class dev.langchain4j.data.message.UserMessage helps to simplify keeping the history up to date by putting the user prompt in the right place.

Enriching answers with Retrieval Augmented Generation (RAG)

No matter how big is the LLM we use for our solutions it lacks something key: the business-specific data. Facts and figures, business processes, knowledge bases… Every bit and piece of internal information of the organisation that is therefore not part of the public data sets used to train LLMs.

If we want to create really useful GenAI-based solutions we need them to be aware of the potential user context: what do they need, what they must know.

// an embedding model good for simple documents
EmbeddingModel embModel = new AllMiniLmL6V2EmbeddingModel();

// an in-memory embedding store
EmbeddingStore<TextSegment> embStore = new InMemoryEmbeddingStore<>();

// load a PDF file from the classpath
Path path = Path.of(ClassLoader.getSystemResource("acme-know-how.pdf").toURI());

Document document = FileSystemDocumentLoader.loadDocument(path, new ApacheTikaDocumentParser());

DocumentSplitter splitter = DocumentSplitters.recursive(256, 0);

// ingest the document into the embedding store
EmbeddingStoreIngestor ingestor = EmbeddingStoreIngestor.builder()
    .documentSplitter(splitter)
    .embeddingModel(embModel)
    .embeddingStore(embStore)
    .build();

ingestor.ingest(document);

Augmenting the Responses

No matter if the knowledge base is created at runtime or it is a persistence enterprise-grade vector search database, the abstraction needed to augment the response during retrieval are the same:

• ContentRetriever from
  package dev.langchain4j.rag.content.retriever: This is responsible to abstract the embedding store and model that will be used to look for the relevant data in the KB.
• AiServices from package dev.langchain4j.service.AiServices: This is a very convenient abstraction to create AI agents combining a given chat model and chat memory (as seen in the previous examples) with the content retriever.

The retrieval example with Langchain4j is quite straightforward:

// define the content retriever connecting everything together

ContentRetriever retriever = EmbeddingStoreContentRetriever.builder()
    .embeddingModel(embModel)
    .embeddingStore(embStore)
    .maxResults(1)
    .minScore(0.8)
    .build();
       
// llama3:8b model running locally with Ollama
ChatLanguageModel chatModel = OllamaChatModel.builder()
    .baseUrl("http://localhost:11434")
    .modelName("llama3:8b")
    .build();

// define context window
ChatMemory chatMemory = MessageWindowChatMemory.withMaxMessages(100);

Agent agent = AiServices.builder(Agent.class)
    .chatLanguageModel(chatModel)
    .chatMemory(chatMemory)
    .contentRetriever(retriever)
    .build();     

String message1 = "Could you summarize in 50 words the main concepts about the Java Platform?";

String answer1 = agent.answer(message1);

interface Agent {
    @SystemMessage("""
        You are an expert in information technologies
        and software engineering.
        """)
    String answer(String inputMessage);
}

Author: deors