9 GenAI Concepts You Should Know Before Calling Yourself an LLM Engineer
Using ChatGPT, Claude, Gemini, or other LLM-powered tools is one thing.
Understanding how these systems actually work is another.
If you're learning Generative AI, preparing for an LLM/GenAI interview, or planning to build AI-powered applications, there are several concepts you should understand beyond simply knowing how to write prompts.
Here are 9 foundational concepts worth knowing well.
1. Tokenization
Before an LLM can process text, the text needs to be converted into tokens.
For example:
"Generative AI is powerful."
may be broken into several tokens, which are then converted into numerical token IDs that the model can process.
Common tokenization approaches include:
BPE (Byte Pair Encoding)
WordPiece
SentencePiece
Tokenization matters because it affects:
Know this: LLMs don't directly "read words." They process sequences of tokens.
2. Embeddings
Embeddings represent information as dense numerical vectors.
The idea is simple:
Similar meanings → similar vectors.
For example, the concepts:
"car"
"automobile"
"vehicle"
can have mathematically similar representations in an embedding space.
Embeddings are fundamental to:
It's important to distinguish between the token embedding layer inside an LLM and embedding models commonly used for semantic search and RAG. They serve related but different purposes.
Know this: embeddings turn semantic relationships into something machines can compare mathematically.
3. Transformers
The Transformer architecture is the foundation behind most modern large language models.
A Transformer processes sequences using components such as:
Transformers made it practical for models to understand relationships between different parts of a sequence and process large amounts of data efficiently.
Modern LLMs such as GPT-style models are built from Transformer-based architectures.
Know this: the Transformer is the architecture; attention is one of its most important mechanisms.
4. Attention
Attention allows a model to determine which parts of the input are important when processing a particular token.
At its core, attention uses:
Query + Key + Value
to calculate how information from different tokens should influence one another.
You should understand the difference between:
Self-attention — tokens attend to other tokens in the same sequence
Causal attention — commonly used for autoregressive generation, where a token cannot look ahead at future tokens
Cross-attention — one sequence attends to another sequence
You should also understand why attention can become computationally expensive as sequence length increases.
Know this: attention is a mechanism that helps the model determine relationships and dependencies within the available context.
5. Pre-training
Pre-training is where a foundation model learns general patterns from enormous datasets.
A simplified version looks like this:
Large dataset → Tokenization → Model training → Learned weights
For autoregressive language models, a common objective is predicting the next token.
For example:
"The coffee is..."
The model learns to assign probabilities to possible next tokens such as:
"hot", "ready", "good", etc.
After seeing enormous amounts of data, the model develops capabilities related to language, patterns, reasoning, and other learned representations.
Know this: pre-training creates the general-purpose foundation model. It is not the same as fine-tuning.
6. Fine-tuning
Fine-tuning takes a pre-trained model and adapts it for a particular task, behavior, or domain.
Examples include:
There are several approaches, including:
The important question isn't simply:
"Can I fine-tune a model?"
It's:
"Should I fine-tune the model, or would prompting/RAG solve the problem more efficiently?"
For example, if your problem is that the model doesn't know today's inventory data, fine-tuning usually isn't the right solution.
That's where RAG can help.
7. RLHF and DPO
A capable model isn't automatically a useful assistant.
Models often need additional training to make their responses more aligned with desired human behavior and preferences.
RLHF — Reinforcement Learning from Human Feedback
A simplified RLHF pipeline can involve:
Human preferences → Reward model → Reinforcement learning → Improved model
Humans compare or rate model responses, and this preference information is used during alignment.
DPO — Direct Preference Optimization
DPO provides another approach to preference optimization.
Instead of separately training a reward model and then performing reinforcement learning, DPO directly optimizes the model using preferred and rejected responses.
Know this: RLHF and DPO are approaches for aligning model behavior with preference data; they are different from pre-training and ordinary supervised fine-tuning.
8. RAG — Retrieval-Augmented Generation
One of the most important concepts for building practical enterprise AI systems is RAG.
Suppose your company has:
Policies
Product catalogs
ERP data
Reports
Manuals
Internal documents
You don't necessarily want to retrain the LLM every time one of those documents changes.
Instead, RAG can retrieve relevant information at runtime.
A simplified RAG pipeline looks like:
Documents → Chunking → Embeddings → Vector/Hybrid Search → Retrieval → Context → LLM → Answer
The model receives relevant external information along with the user's question.
Good RAG systems require more than simply putting documents into a vector database.
You need to think about:
Chunking strategy
Embedding quality
Metadata filtering
Hybrid search
Reranking
Retrieval quality
Context limits
Hallucination
Evaluation
Know this: RAG primarily solves a knowledge retrieval problem at inference time. Fine-tuning primarily changes the model's learned behavior or capabilities.
9. LoRA and PEFT
Full fine-tuning can require updating a huge number of model parameters.
That's expensive.
Parameter-Efficient Fine-Tuning (PEFT) provides approaches for adapting models while training only a small portion of additional parameters.
One of the most popular techniques is LoRA — Low-Rank Adaptation.
The basic idea is:
Freeze the original model → Train small adapter parameters → Use the adapter with the base model
This can significantly reduce:
Trainable parameters
GPU memory requirements
Training cost
Storage requirements
LoRA is therefore particularly useful when you need to adapt a model without performing expensive full-model fine-tuning.
Know this: LoRA doesn't replace the base model. It provides an efficient way to adapt it.
How These Concepts Actually Connect
These concepts are related, but they aren't simply one linear pipeline.
A better mental model is:
┌── Pre-training ──→ Foundation Model
│
Tokens → Transformer + Attention
│
└── Fine-tuning / Alignment
│
┌────┴────┐
│ │
Full FT PEFT/LoRA
External Knowledge
↓
Documents
↓
Chunking
↓
Embeddings
↓
Retrieval
↓
Relevant Context
↓
LLM
↓
Generated Answer
In other words, RAG is not simply the final step after fine-tuning. It is an application architecture that can be used alongside a foundation model, fine-tuned model, or adapted model.
The Trade-offs You Should Understand
For interviews and real-world projects, memorizing definitions isn't enough.
You should be able to explain why you would choose one approach over another.
RAG vs Fine-tuning
Ask:
Do I need to give the model new information, or change how the model behaves?
Changing external knowledge → RAG
Changing behavior/task specialization → Fine-tuning
Sometimes the right architecture uses both.
Tokens vs Parameters vs Context Window
These are frequently confused.
Tokens
The units of text processed by the model.
Parameters
The learned numerical values that make up the model.
Context window
The amount of tokenized information the model can consider within a single request/context.
Training vs Inference
Training
The model learns by updating parameters or adapter weights.
Inference
The trained model generates an output from an input.
A production AI system may spend far more time and money on inference than on training, depending on its usage pattern.
Latency vs Throughput
Latency:
How long it takes to process a request.
Throughput:
How much work the system can process over a period of time.
A system can have low latency for one request but still struggle to handle thousands of simultaneous users.
Quality vs Cost
Larger models may provide better capabilities for some tasks, but they can also require more compute and cost more to operate.
Real-world GenAI engineering is often about finding the right balance between:
Quality + Cost + Latency + Reliability
Full Fine-tuning vs LoRA
Full fine-tuning:
Update a large portion or all of the model's parameters.
LoRA/PEFT:
Keep the base model frozen and train a much smaller set of adapter parameters.
The right choice depends on the task, model, dataset, infrastructure, and desired outcome.
What You Should Be Able to Explain in an Interview
Don't just memorize:
"RAG stands for Retrieval-Augmented Generation."
Be able to answer:
Why would you use RAG?
When would you choose fine-tuning instead?
How does vector search work?
What is an embedding?
Why does chunk size matter?
What causes poor retrieval?
What is a context window?
Why does tokenization affect cost?
What is attention?
Why is causal attention needed for autoregressive generation?
What is the difference between training and inference?
What problem does LoRA solve?
What is the difference between RLHF and DPO?
How would you evaluate a RAG system?
That's where understanding starts to matter more than memorization.
The Bigger Picture
You don't need to become a researcher to build useful GenAI systems.
But if you want to move beyond simply using AI tools and start engineering AI systems, these concepts form a strong foundation.
The progression is roughly:
Tokens → Representations → Transformers → Attention → Pre-training → Fine-tuning → Alignment → Retrieval → Efficient Adaptation
And the most valuable skill is not knowing every definition.
It's knowing:
What problem does this technology solve, what are its limitations, and when should I use it?
That is the difference between simply using an LLM and understanding how to build systems around one.
