Unlocking the Power of Large Language Models in Real Estate: How They Work and Their Impact on Our Industry

Large language models have revolutionized the field of natural language processing (NLP) becoming an integral part of many applications, from chatbots and virtual assistants to content generation and translation. These models, such as OpenAI’s GPT-4, have proven remarkably skillful at understanding and generating human-like text. But how do they work, and what makes them so powerful? Why did they suddenly burst onto the scene in recent months even though processor design has been stagnating for years? Once we answer these questions, we can turn our attention to their application in our favorite industry, real estate.

The Basics of Large Language Models

Large language models are built upon deep learning techniques, specifically a type of neural network called a Transformer architecture. Transformer architectures are designed to handle sequential data, particularly text, in an exceptionally efficient and effective manner. What sets transformers apart is their attention mechanism (also called attention scores), which allows them to process input data in parallel, rather than sequentially like traditional recurrent neural networks (RNNs). This parallel processing capability, thanks to self-attention layers, enables transformers to capture long-range dependencies between words in a sentence or context, making them particularly adept at understanding and generating coherent text. Transformers have been the driving force behind the development of state-of-the-art LLMs, such as OpenAI’s GPT series, and have paved the way for significant advancements in a wide range of NLP applications, including machine translation, sentiment analysis, and text summarization. Their architecture’s scalability and ability to capture context and semantics efficiently have made transformers a foundational building block in the field of deep learning and natural language understanding.

Let’s look at how these LLMs are built. Here’s a breakdown of the key components and processes involved:

1. Data Collection:

The process begins with the collection of vast amounts of text data from the internet. This data is typically obtained from websites, books, articles, forums, and other publicly available sources.

The data collected is diverse and covers a wide range of topics and writing styles to make the model more versatile.

2. Data Cleaning and Preprocessing:

Raw data from the internet often contains noise, irrelevant information, and formatting issues. Data cleaning involves removing or correcting these issues.

Text is usually tokenized, breaking it into smaller units such as words, subwords, or characters. Tokenization depends on the chosen vocabulary and language.

Tokens are converted into numerical representations using word embeddings. These numerical vectors allow the model to process text as input.

Why word embeddings? There are other methods for encoding language numerically: The most common being one-hot encoding where a Boolean variable is set to one if a word or series of words are found in a text and zero if they are not. Word embeddings outperform one-hot encoding, however, as word representations in large language models (LLMs) due to their ability to capture semantic meaning and contextual information efficiently. Unlike one-hot encoding, which represents words as isolated symbols, word embeddings map words into continuous vector spaces, allowing LLMs to understand and leverage the nuanced relationships between words: Words with related meanings often find themselves near each other in the “phase” space into which they are mapped. This semantic richness empowers LLMs to generalize from training data, recognize synonyms, and grasp the contextual nuances of language. Moreover, word embeddings offer memory-efficient representations by reducing dimensionality, a critical factor in enabling LLMs to handle vast vocabularies effectively. These advantages make word embeddings the preferred choice for LLMs, enabling them to achieve remarkable performance in various natural language processing tasks.

Let’s look at the advantages word embeddings provide in more detail:

Word embeddings maintain context by capturing semantic and syntactic relationships between words in a given text corpus. They represent words as dense numerical vectors in a continuous vector space, where similar words are located closer to each other.

Word embeddings are typically trained using unsupervised learning techniques, such as Word2Vec or GloVe, on large amounts of text data. During training, these models learn to predict the surrounding words of a target word based on its context in the text. By doing so, they capture the contextual information of words.

Read more: AI in real estate, Part I

Here’s how word embeddings maintain context:

Distributional Hypothesis: Word embeddings are based on the distributional hypothesis, which states that words occurring in similar contexts tend to have similar meanings. This hypothesis forms the basis for training word embeddings.

Context Window:

During the training process, a “context window” is defined around each target word. The context window determines the neighboring words that will be used to predict the target word. By considering the words in the context window, the model learns to associate words with their surrounding context.

Learning Word Representations:

The word embedding model predicts the context words given the target word or vice versa. It updates the vector representations of words in such a way that words appearing in similar contexts have similar numerical representations. This allows the embeddings to capture the underlying semantic and syntactic relationships between words.

Vector Space Structure:

The trained word embeddings are organized in a vector space, where the positions of the word vectors encode the relationships between words. Words with similar meanings or appearing in similar contexts are located closer to each other in the vector space. For example, in a well-trained embedding space, the vectors for “dog” and “cat” would be closer together than the vectors for “dog” and “car.”

Transfer of Contextual Information:

When using word embeddings in downstream natural language processing tasks, the contextual information captured during training is preserved. The embeddings allow models to leverage the semantic and syntactic relationships between words in the vector space. This helps in tasks such as text classification, sentiment analysis, machine translation, and more, where understanding context is crucial.

By training word embeddings on large text corpora and taking into account the distributional properties of words, these representations can capture the context in which words appear and store that information in their vector representations. This allows models to benefit from the contextual knowledge embedded in the word embeddings during various text processing tasks.

3. Implementing and Leveraging Attention Scores:

Attention scores are a pivotal component in the architecture of Large Language Models (LLMs). These scores are essentially weightings assigned to different parts of input data, signifying their relevance to a given context or query. By effectively capturing the relationships and dependencies between words in a sequence, attention scores allow LLMs to focus on crucial information while ignoring irrelevant details during the learning process. This mechanism significantly enhances the model’s ability to generate coherent and contextually relevant text, making LLMs feasible for a wide range of natural language processing tasks, from translation to text generation, by harnessing the power of attention to distill intricate linguistic patterns and associations.

Here’s an overview of how attention scores work:

Input Sequence and Query:

In many applications, the input is represented as a sequence of tokens (e.g., words in a sentence). To compute attention scores, you also have a query, which is typically a single token or a set of tokens for which you want to compute attention. For example, in machine translation, the query might be a token in the target language, and you want to know how much attention should be paid to each token in the source language when translating that target token.

Key-Value Pairs:

Along with the input sequence, you have associated key-value pairs for each token in the sequence. These key-value pairs are used to compute the attention scores. In most cases, the key and value are the linear transformations of input token (generally the input tokens that are multiplied by a series of initially random weights that are also themselves subject to refinement and training), but they can also be learned representations of the input token.

Scoring Mechanism:

To compute attention scores, a scoring mechanism is applied to the query and the keys. A common scoring mechanism is the dot product or scaled dot product, but other mechanisms like additive attention are also used. The scoring mechanism measures the similarity or compatibility between the query and each key.

Attention Weights:

The scores obtained in the previous step are normalized using a softmax function to convert them into probabilities. These probabilities are called attention weights and represent how much attention should be given to each key-value pair in the input sequence. High attention weights indicate that a particular token in the input sequence is highly relevant to the query.

Weighted Sum:

Finally, the attention weights are used to compute a weighted sum of the corresponding values in the input sequence. This weighted sum is the output of the attention mechanism and can be used in various ways depending on the task. For example, in sequence-to-sequence tasks like machine translation, this weighted sum can be used as part of the decoder’s input.

In summary, attention scores are used to determine how much importance or relevance each token in an input sequence has concerning a given query. These scores are computed by comparing the query with keys associated with each token in the input sequence. The resulting attention weights guide the model’s decision on which parts of the input to focus on when generating output, making them a crucial component in many deep learning architectures, especially transformers.

4. Creating Training Samples:

To train the model, the dataset is divided into training samples, where each sample consists of a sequence of tokens (words or subwords).

Samples can be of varying lengths, but they are often grouped into batches of fixed-length sequences for efficient processing during training.

The dataset is divided into a training set, a validation set, and a test set. The training set is used to teach the model, the validation set helps tune hyperparameters and monitor progress, and the test set evaluates the model’s performance.

5. The Role of Transformers in LLM Training:

Transformers represent a groundbreaking architecture that plays a pivotal role in the training and effectiveness of Large Language Models (LLMs). They have revolutionized natural language processing by offering efficient and powerful mechanisms for modeling sequential data, particularly text. The core innovation of transformers lies in their ability to process input data in parallel, a significant departure from the sequential processing approach of traditional recurrent neural networks (RNNs). This parallelism, powered by self-attention mechanisms, has ushered in a new era of language understanding and generation.

Here’s a comprehensive overview of transformer architectures and their significance in LLM training:

Self-Attention Mechanism:

At the heart of transformers is the self-attention mechanism, which allows the model to capture dependencies between words or tokens across varying distances within a sequence. This mechanism enables transformers to understand context and relationships by assigning different levels of attention to different parts of the input data.

Parallel Processing:

Unlike sequential models, which process input data step by step, transformers process all tokens in the input sequence simultaneously. This parallelism is achieved through self-attention, which allows each token to attend to all other tokens. As a result, transformers can capture long-range dependencies efficiently, making them exceptionally suited for understanding the nuances of natural language.

Multi-Head Attention:

Transformers often incorporate multi-head attention mechanisms, where multiple sets of attention scores are computed in parallel. Each attention head focuses on different aspects of the input data, allowing the model to capture diverse types of information and dependencies. This multi-head approach enhances the model’s capacity to learn complex patterns.

Positional Encoding:

Since transformers do not inherently encode the order or position of tokens in a sequence, positional encoding is added to the input embeddings. Positional encoding provides the model with information about the token’s position in the sequence, ensuring that the model understands the sequential nature of the data.

Feedforward Neural Networks:

Transformers also include feedforward neural networks for processing the output of the attention mechanism. These networks allow the model to apply non-linear transformations to the input data, further enhancing its ability to capture complex patterns and relationships.

Encoder-Decoder Architecture:

In tasks like machine translation, transformers use an encoder-decoder architecture, where one set of transformers (the encoder) processes the source language, and another set (the decoder) generates the target language. This architecture has proven highly effective in sequence-to-sequence tasks.

Scalability:

Transformers are highly scalable, making them suitable for training on vast datasets and handling large vocabularies. This scalability has been a key factor in the success of LLMs, enabling them to capture a wide range of language nuances.

6. Masking and Next-Word Prediction:

During training, the model learns to predict the next word or token in a sequence given the context of previous tokens.

For each training sample, a portion of the tokens is masked, and the model’s objective is to predict these masked tokens based on the unmasked context. This process is known as a masked language modeling (MLM) task.

Read more: Using social media sentiment to make predictions

7. Data Augmentation and Fine-Tuning:

After the initial pretraining on the large, diverse dataset, the model can undergo further training, often called fine-tuning, on a more specific dataset related to a particular task or domain.

Fine-tuning helps adapt the model to specific applications, such as medical text generation or customer service chatbots.

Classification models are used to determine which of these domain specific models should be utilized for a given question—with the “first model” serving as a catchall backup.

Large language models represent a remarkable leap in NLP technology, enabling applications that were once considered science fiction. However, their development and deployment come with ethical, environmental, and practical challenges. As the field continues to evolve, addressing these challenges will be essential to harness the full potential of large language models while ensuring their responsible and fair use.

Uses for Large Language Models in the Real Estate Industry

Large language models like GPT-3 can be utilized in several ways within the real estate industry to streamline processes, enhance user experiences, and provide valuable insights. Here are a few potential applications:

Property Search and Recommendations:

Language models can be employed to develop intelligent property search platforms. Users can describe their preferences, requirements, and budget in natural language, and the model can generate personalized property recommendations based on the input. This can assist buyers, renters, or investors in finding suitable properties more efficiently.

Virtual Assistants and Chatbots:

Language models can power virtual assistants or chatbots that provide instant responses to customer inquiries. These AI-powered assistants can handle common queries about property listings, pricing, availability, and provide guidance on real estate processes. They can enhance customer service, provide 24/7 support, and free up human agents’ time for more complex tasks.

Market Analysis and Pricing:

Large language models can process vast amounts of real estate data, including property listings, historical sales data, and market trends. By analyzing this information, the models can generate insights on property valuations, price trends, and identify emerging market opportunities. Such analysis can aid real estate professionals, investors, and developers in making informed decisions.

Document Analysis and Contract Generation:

Real estate transactions involve numerous documents like contracts, agreements, and legal paperwork. Language models can assist in automating the analysis of these documents, extracting relevant information, identifying potential issues, and generating standardized contracts. This can improve efficiency, reduce errors, and streamline the negotiation and closing processes.

Natural Language Interfaces for Property Management:

Language models can be leveraged to develop intuitive natural language interfaces for property management systems. Property owners, managers, or tenants can interact with these systems using everyday language to perform tasks such as rental applications, maintenance requests, lease renewals, or payment processing. This simplifies user interactions and enhances user experiences.

Market Research and Customer Insights:

Language models can be utilized to analyze online reviews, social media conversations, and customer feedback related to real estate properties, agents, or development projects. This can provide valuable insights into customer sentiment, preferences, and help real estate professionals understand market trends, identify areas for improvement, and tailor their offerings accordingly.

Read more: Agent based models

Alex Kisselev

Alex Kisselev, Director of Housing Analytics at Kukun, is an expert modeler with a focus on credit, operational risk, automated valuation, and cash flow. With a strong background in hard science and advanced degrees in physics and mathematics, including a Ph.D. in Computational Physics, he excels in various programming languages, including R, Python, SAS, Java, and C/C++. Alex has shared his knowledge at George Mason University and contributed his expertise to diverse organizations, from Fannie Mae to cutting-edge hedge funds.

Published October 17, 2023