AI-Powered E-commerce: Building Recommendation Systems and Personalization

Note: This guide is based on established recommendation system algorithms documented in RecSys research papers, scikit-learn documentation, and production patterns from e-commerce platforms like Amazon, Netflix, and Shopify. All code examples use documented machine learning libraries and follow industry best practices for recommendation systems.

AI has transformed e-commerce from generic shopping experiences to hyper-personalized customer journeys. Recommendation systems—the technology behind “Customers who bought this also bought” and personalized homepages—drive 35% of Amazon’s revenue and 75% of Netflix viewing.

This guide demonstrates building production-ready recommendation systems using collaborative filtering, content-based filtering, and leveraging large language models (LLMs) for automated product description generation and personalized marketing content.

Prerequisites

Required Knowledge:

Python 3.8+ programming
Basic understanding of machine learning concepts (supervised learning, features, training)
SQL or pandas for data manipulation
Understanding of e-commerce metrics (conversion rate, AOV, CTR)

Required Tools:

# Install core ML libraries
pip install pandas==2.1.0 numpy==1.24.3 scikit-learn==1.3.0

# Install recommendation system libraries
pip install surprise==1.1.3  # Collaborative filtering
pip install implicit==0.7.2  # Matrix factorization (ALS)

# Install LLM libraries for content generation
pip install openai==1.3.0  # OpenAI GPT-4
pip install anthropic==0.7.0  # Claude API

# Install API framework
pip install fastapi==0.104.0 uvicorn==0.24.0
pip install redis==5.0.0  # Caching for production

# Install data visualization (optional)
pip install matplotlib==3.8.0 seaborn==0.12.2

Data Requirements:

User interaction data (views, clicks, purchases, ratings)
Product catalog (IDs, categories, attributes, descriptions)
Minimum 1000 interactions for meaningful recommendations

Understanding Recommendation Systems

Types of Recommendation Algorithms

Algorithm Type	How It Works	Strengths	Limitations
Collaborative Filtering	Finds similar users/items based on interactions	Discovers unexpected connections	Cold start problem (new users/items)
Content-Based	Recommends similar items based on features	Works for new users	Limited serendipity, needs good metadata
Hybrid	Combines multiple approaches	Best of both worlds	More complex to implement
Deep Learning	Neural networks learn patterns	Can capture complex relationships	Needs large datasets, expensive training

Real-World E-commerce AI Applications

1. Product Recommendations:

Collaborative filtering: “Customers like you bought…”
Content-based: “Similar products…”
Session-based: “Frequently bought together”

2. Personalized Search:

Re-rank search results based on user preferences
Query understanding with NLP

3. Dynamic Content Generation:

Product descriptions with LLMs
Personalized email subject lines
SEO-optimized category pages

4. Pricing Optimization:

Dynamic pricing based on demand/competition
Personalized discounts

Building a Collaborative Filtering Recommender

Data Preparation

# ecommerce_data.py - Sample data structure and loading
import pandas as pd
import numpy as np
from typing import Tuple

def load_ecommerce_data() -> Tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]:
    """
    Load e-commerce interaction data

    Returns:
        interactions_df: User-item interactions (user_id, item_id, rating, timestamp)
        users_df: User metadata (user_id, age, gender, location)
        items_df: Product catalog (item_id, name, category, price, description)
    """
    # Example schema - replace with your actual data source
    interactions_df = pd.DataFrame({
        'user_id': [1, 1, 1, 2, 2, 3, 3, 3, 4, 5],
        'item_id': [101, 102, 105, 101, 103, 102, 104, 105, 101, 102],
        'rating': [5, 4, 3, 5, 4, 5, 3, 4, 5, 4],  # Explicit ratings or implicit (1=view, 5=purchase)
        'timestamp': pd.date_range('2024-01-01', periods=10, freq='D')
    })

    items_df = pd.DataFrame({
        'item_id': [101, 102, 103, 104, 105],
        'name': ['Wireless Mouse', 'USB Keyboard', 'Monitor Stand', 'Laptop Bag', 'Webcam'],
        'category': ['Electronics', 'Electronics', 'Accessories', 'Accessories', 'Electronics'],
        'price': [29.99, 49.99, 39.99, 34.99, 79.99],
        'description': ['Ergonomic wireless mouse', 'Mechanical keyboard', 'Adjustable stand', 'Laptop carry bag', 'HD webcam']
    })

    users_df = pd.DataFrame({
        'user_id': [1, 2, 3, 4, 5],
        'age': [25, 35, 28, 42, 31],
        'location': ['NY', 'CA', 'TX', 'FL', 'WA']
    })

    return interactions_df, users_df, items_df


def create_user_item_matrix(interactions_df: pd.DataFrame) -> pd.DataFrame:
    """
    Create user-item interaction matrix

    Args:
        interactions_df: DataFrame with columns [user_id, item_id, rating]

    Returns:
        Pivot table with users as rows, items as columns, ratings as values
    """
    matrix = interactions_df.pivot_table(
        index='user_id',
        columns='item_id',
        values='rating',
        fill_value=0  # 0 means no interaction
    )

    return matrix


def train_test_split_temporal(
    interactions_df: pd.DataFrame,
    test_size: float = 0.2
) -> Tuple[pd.DataFrame, pd.DataFrame]:
    """
    Split data temporally (last 20% of interactions as test)
    Important: Never split randomly for time-series e-commerce data!
    """
    interactions_df = interactions_df.sort_values('timestamp')
    split_idx = int(len(interactions_df) * (1 - test_size))

    train_df = interactions_df.iloc[:split_idx]
    test_df = interactions_df.iloc[split_idx:]

    return train_df, test_df

User-Based Collaborative Filtering

# collaborative_filtering.py - User-based and item-based CF
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
import pandas as pd
from typing import List, Tuple

class CollaborativeFilteringRecommender:
    """
    User-based collaborative filtering recommender
    "Users like you also liked..."
    """

    def __init__(self, min_overlap: int = 3):
        """
        Args:
            min_overlap: Minimum number of common items to consider users similar
        """
        self.min_overlap = min_overlap
        self.user_item_matrix = None
        self.user_similarity_matrix = None

    def fit(self, interactions_df: pd.DataFrame) -> None:
        """
        Build user similarity matrix

        Args:
            interactions_df: DataFrame with columns [user_id, item_id, rating]
        """
        # Create user-item matrix
        self.user_item_matrix = create_user_item_matrix(interactions_df)

        # Calculate user similarity using cosine similarity
        # Cosine similarity: measures angle between user vectors
        # Range: -1 (opposite) to 1 (identical)
        self.user_similarity_matrix = pd.DataFrame(
            cosine_similarity(self.user_item_matrix),
            index=self.user_item_matrix.index,
            columns=self.user_item_matrix.index
        )

    def predict_rating(self, user_id: int, item_id: int, k: int = 10) -> float:
        """
        Predict rating for user-item pair using k nearest neighbors

        Args:
            user_id: Target user
            item_id: Target item
            k: Number of similar users to consider

        Returns:
            Predicted rating (0 if cannot predict)
        """
        if user_id not in self.user_item_matrix.index:
            return 0.0

        if item_id not in self.user_item_matrix.columns:
            return 0.0

        # Get users who rated this item
        item_ratings = self.user_item_matrix[item_id]
        rated_users = item_ratings[item_ratings > 0].index

        if len(rated_users) == 0:
            return 0.0

        # Get similarity scores for users who rated this item
        similarities = self.user_similarity_matrix.loc[user_id, rated_users]

        # Get top k similar users
        top_k_users = similarities.nlargest(k).index

        if len(top_k_users) == 0:
            return 0.0

        # Weighted average of ratings from similar users
        numerator = sum(
            self.user_similarity_matrix.loc[user_id, sim_user] *
            self.user_item_matrix.loc[sim_user, item_id]
            for sim_user in top_k_users
        )

        denominator = sum(
            abs(self.user_similarity_matrix.loc[user_id, sim_user])
            for sim_user in top_k_users
        )

        if denominator == 0:
            return 0.0

        return numerator / denominator

    def recommend(
        self,
        user_id: int,
        n_recommendations: int = 10,
        exclude_purchased: bool = True
    ) -> List[Tuple[int, float]]:
        """
        Generate top N recommendations for user

        Args:
            user_id: Target user
            n_recommendations: Number of items to recommend
            exclude_purchased: Whether to exclude already purchased items

        Returns:
            List of (item_id, predicted_rating) tuples
        """
        if user_id not in self.user_item_matrix.index:
            return []

        # Get all items
        all_items = self.user_item_matrix.columns

        # Optionally exclude already interacted items
        if exclude_purchased:
            purchased_items = self.user_item_matrix.loc[user_id]
            purchased_items = purchased_items[purchased_items > 0].index
            candidate_items = all_items.difference(purchased_items)
        else:
            candidate_items = all_items

        # Predict ratings for all candidate items
        predictions = []
        for item_id in candidate_items:
            predicted_rating = self.predict_rating(user_id, item_id)
            if predicted_rating > 0:
                predictions.append((item_id, predicted_rating))

        # Sort by predicted rating
        predictions.sort(key=lambda x: x[1], reverse=True)

        return predictions[:n_recommendations]

Matrix Factorization with ALS (Production-Grade)

# matrix_factorization.py - ALS recommender (faster, more scalable)
from implicit.als import AlternatingLeastSquares
from scipy.sparse import csr_matrix
import numpy as np
import pandas as pd
from typing import List, Tuple

class ALSRecommender:
    """
    Alternating Least Squares (ALS) matrix factorization
    Used by Spotify, Netflix - scales to millions of users/items
    """

    def __init__(
        self,
        factors: int = 50,
        regularization: float = 0.01,
        iterations: int = 15,
        alpha: float = 40.0
    ):
        """
        Args:
            factors: Number of latent factors (embedding dimensions)
            regularization: L2 regularization strength
            iterations: Training iterations
            alpha: Confidence scaling for implicit feedback
        """
        self.model = AlternatingLeastSquares(
            factors=factors,
            regularization=regularization,
            iterations=iterations,
            calculate_training_loss=True,
            random_state=42
        )
        self.alpha = alpha
        self.user_id_map = {}
        self.item_id_map = {}
        self.reverse_user_map = {}
        self.reverse_item_map = {}
        self.user_item_sparse = None

    def fit(self, interactions_df: pd.DataFrame) -> None:
        """
        Train ALS model on interaction data

        Args:
            interactions_df: DataFrame with columns [user_id, item_id, rating]
        """
        # Create user and item ID mappings (ALS requires 0-indexed IDs)
        unique_users = interactions_df['user_id'].unique()
        unique_items = interactions_df['item_id'].unique()

        self.user_id_map = {uid: idx for idx, uid in enumerate(unique_users)}
        self.item_id_map = {iid: idx for idx, iid in enumerate(unique_items)}
        self.reverse_user_map = {idx: uid for uid, idx in self.user_id_map.items()}
        self.reverse_item_map = {idx: iid for iid, idx in self.item_id_map.items()}

        # Map external IDs to internal indices
        interactions_df = interactions_df.copy()
        interactions_df['user_idx'] = interactions_df['user_id'].map(self.user_id_map)
        interactions_df['item_idx'] = interactions_df['item_id'].map(self.item_id_map)

        # Create sparse matrix (memory-efficient for large datasets)
        # Confidence = 1 + alpha * rating
        confidence = 1 + self.alpha * interactions_df['rating'].values

        self.user_item_sparse = csr_matrix(
            (confidence, (interactions_df['user_idx'], interactions_df['item_idx'])),
            shape=(len(self.user_id_map), len(self.item_id_map))
        )

        # Train model
        self.model.fit(self.user_item_sparse, show_progress=False)

    def recommend(
        self,
        user_id: int,
        n_recommendations: int = 10,
        filter_already_liked: bool = True
    ) -> List[Tuple[int, float]]:
        """
        Generate recommendations using trained model

        Args:
            user_id: External user ID
            n_recommendations: Number of recommendations
            filter_already_liked: Exclude items user already interacted with

        Returns:
            List of (item_id, score) tuples
        """
        if user_id not in self.user_id_map:
            return []

        user_idx = self.user_id_map[user_id]

        # Get recommendations (returns internal indices)
        item_indices, scores = self.model.recommend(
            userid=user_idx,
            user_items=self.user_item_sparse[user_idx],
            N=n_recommendations,
            filter_already_liked_items=filter_already_liked
        )

        # Map back to external item IDs
        recommendations = [
            (self.reverse_item_map[idx], float(score))
            for idx, score in zip(item_indices, scores)
        ]

        return recommendations

    def similar_items(self, item_id: int, n_similar: int = 10) -> List[Tuple[int, float]]:
        """
        Find similar items (for "Customers also viewed" recommendations)

        Args:
            item_id: External item ID
            n_similar: Number of similar items

        Returns:
            List of (item_id, similarity_score) tuples
        """
        if item_id not in self.item_id_map:
            return []

        item_idx = self.item_id_map[item_id]

        # Get similar items
        similar_indices, scores = self.model.similar_items(
            itemid=item_idx,
            N=n_similar + 1  # +1 because item is most similar to itself
        )

        # Exclude the item itself and map to external IDs
        similar_items = [
            (self.reverse_item_map[idx], float(score))
            for idx, score in zip(similar_indices[1:], scores[1:])
        ]

        return similar_items

Content-Based Recommendations

TF-IDF Product Similarity

# content_based.py - Content-based recommendations using product features
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import pandas as pd
from typing import List, Tuple

class ContentBasedRecommender:
    """
    Content-based recommendations using product descriptions and categories
    Works well for new users (cold start) since it doesn't need interaction data
    """

    def __init__(self):
        self.vectorizer = TfidfVectorizer(
            max_features=1000,
            stop_words='english',
            ngram_range=(1, 2)  # Unigrams and bigrams
        )
        self.item_features = None
        self.similarity_matrix = None
        self.items_df = None

    def fit(self, items_df: pd.DataFrame) -> None:
        """
        Build item similarity matrix from product metadata

        Args:
            items_df: DataFrame with columns [item_id, name, category, description]
        """
        self.items_df = items_df.set_index('item_id')

        # Combine text features
        items_df['combined_features'] = (
            items_df['name'] + ' ' +
            items_df['category'] + ' ' +
            items_df['description']
        )

        # Create TF-IDF features
        self.item_features = self.vectorizer.fit_transform(
            items_df['combined_features']
        )

        # Calculate item similarity
        self.similarity_matrix = cosine_similarity(self.item_features)

    def similar_items(self, item_id: int, n_similar: int = 10) -> List[Tuple[int, float]]:
        """
        Find similar items based on content

        Args:
            item_id: Target item
            n_similar: Number of similar items

        Returns:
            List of (item_id, similarity_score) tuples
        """
        if item_id not in self.items_df.index:
            return []

        item_idx = self.items_df.index.get_loc(item_id)
        similarity_scores = self.similarity_matrix[item_idx]

        # Get top N similar items (excluding item itself)
        similar_indices = similarity_scores.argsort()[::-1][1:n_similar+1]

        similar_items = [
            (self.items_df.index[idx], float(similarity_scores[idx]))
            for idx in similar_indices
        ]

        return similar_items

    def recommend_for_user(
        self,
        user_interactions: List[int],
        n_recommendations: int = 10
    ) -> List[Tuple[int, float]]:
        """
        Recommend items based on user's interaction history

        Args:
            user_interactions: List of item IDs user has interacted with
            n_recommendations: Number of recommendations

        Returns:
            List of (item_id, score) tuples
        """
        if not user_interactions:
            return []

        # Get indices of user's items
        user_item_indices = [
            self.items_df.index.get_loc(item_id)
            for item_id in user_interactions
            if item_id in self.items_df.index
        ]

        if not user_item_indices:
            return []

        # Average similarity across all user's items
        avg_similarity = self.similarity_matrix[user_item_indices].mean(axis=0)

        # Exclude already interacted items
        for idx in user_item_indices:
            avg_similarity[idx] = -1

        # Get top N recommendations
        top_indices = avg_similarity.argsort()[::-1][:n_recommendations]

        recommendations = [
            (self.items_df.index[idx], float(avg_similarity[idx]))
            for idx in top_indices
        ]

        return recommendations

LLM-Powered Product Description Generation

Using OpenAI GPT-4 for Content Creation

# llm_content_generation.py - Product descriptions with LLMs
import openai
from anthropic import Anthropic
from typing import Dict, Optional
import os

class ProductDescriptionGenerator:
    """
    Generate SEO-optimized product descriptions using LLMs
    """

    def __init__(self, provider: str = "openai", api_key: Optional[str] = None):
        """
        Args:
            provider: "openai" or "anthropic"
            api_key: API key (or set OPENAI_API_KEY/ANTHROPIC_API_KEY env var)
        """
        self.provider = provider

        if provider == "openai":
            self.client = openai.OpenAI(api_key=api_key or os.getenv("OPENAI_API_KEY"))
            self.model = "gpt-4-turbo-preview"
        elif provider == "anthropic":
            self.client = Anthropic(api_key=api_key or os.getenv("ANTHROPIC_API_KEY"))
            self.model = "claude-3-sonnet-20240229"
        else:
            raise ValueError(f"Unknown provider: {provider}")

    def generate_description(
        self,
        product_name: str,
        category: str,
        features: Dict[str, str],
        tone: str = "professional",
        max_words: int = 150
    ) -> str:
        """
        Generate product description

        Args:
            product_name: Name of product
            category: Product category
            features: Dict of product features {"color": "black", "material": "leather"}
            tone: "professional", "casual", "luxury", "technical"
            max_words: Maximum description length

        Returns:
            Generated product description
        """
        features_text = "\n".join([f"- {k}: {v}" for k, v in features.items()])

        prompt = f"""Generate a compelling product description for an e-commerce store.

Product: {product_name}
Category: {category}
Features:
{features_text}

Requirements:
- Tone: {tone}
- Length: {max_words} words maximum
- Include benefits (not just features)
- SEO-friendly (include relevant keywords naturally)
- No made-up specifications or claims

Description:"""

        if self.provider == "openai":
            response = self.client.chat.completions.create(
                model=self.model,
                messages=[
                    {"role": "system", "content": "You are an expert e-commerce copywriter. Write compelling, accurate product descriptions."},
                    {"role": "user", "content": prompt}
                ],
                max_tokens=300,
                temperature=0.7
            )
            return response.choices[0].message.content.strip()

        elif self.provider == "anthropic":
            response = self.client.messages.create(
                model=self.model,
                max_tokens=300,
                messages=[
                    {"role": "user", "content": prompt}
                ]
            )
            return response.content[0].text.strip()

    def generate_email_subject(
        self,
        user_name: str,
        recommended_products: List[str],
        occasion: Optional[str] = None
    ) -> str:
        """
        Generate personalized email subject line

        Args:
            user_name: Customer's first name
            recommended_products: List of product names
            occasion: Optional occasion (e.g., "Black Friday", "Birthday")

        Returns:
            Email subject line
        """
        products_text = ", ".join(recommended_products[:3])
        occasion_text = f" for {occasion}" if occasion else ""

        prompt = f"""Generate an engaging email subject line for a personalized product recommendation.

Customer: {user_name}
Recommended Products: {products_text}
Occasion: {occasion or "Regular recommendation"}

Requirements:
- Personal (use customer name if appropriate)
- Creates urgency or curiosity
- Under 60 characters
- No clickbait or spam triggers

Subject line:"""

        if self.provider == "openai":
            response = self.client.chat.completions.create(
                model=self.model,
                messages=[{"role": "user", "content": prompt}],
                max_tokens=50,
                temperature=0.8
            )
            return response.choices[0].message.content.strip()

        elif self.provider == "anthropic":
            response = self.client.messages.create(
                model=self.model,
                max_tokens=50,
                messages=[{"role": "user", "content": prompt}]
            )
            return response.content[0].text.strip()

Production API Deployment

FastAPI Recommendation Service

# api.py - Production recommendation API
from fastapi import FastAPI, HTTPException, Depends
from pydantic import BaseModel
from typing import List, Optional
import redis
import json
import hashlib

app = FastAPI(title="E-commerce Recommendation API")

# Initialize Redis for caching
redis_client = redis.Redis(host='localhost', port=6379, decode_responses=True)

# Load trained models (in production, load from model registry)
als_recommender = ALSRecommender()
content_recommender = ContentBasedRecommender()

class RecommendationRequest(BaseModel):
    user_id: int
    n_recommendations: int = 10
    include_reasons: bool = False

class RecommendationResponse(BaseModel):
    user_id: int
    recommendations: List[Dict[str, any]]

def get_cache_key(request: RecommendationRequest) -> str:
    """Generate cache key from request"""
    key_str = f"{request.user_id}:{request.n_recommendations}"
    return hashlib.md5(key_str.encode()).hexdigest()

@app.post("/recommendations/collaborative", response_model=RecommendationResponse)
async def get_collaborative_recommendations(request: RecommendationRequest):
    """
    Get collaborative filtering recommendations
    Cache recommendations for 1 hour
    """
    # Check cache
    cache_key = get_cache_key(request)
    cached = redis_client.get(f"collab:{cache_key}")

    if cached:
        return json.loads(cached)

    # Generate recommendations
    try:
        recs = als_recommender.recommend(
            user_id=request.user_id,
            n_recommendations=request.n_recommendations
        )

        response = {
            "user_id": request.user_id,
            "recommendations": [
                {"item_id": item_id, "score": score}
                for item_id, score in recs
            ]
        }

        # Cache for 1 hour
        redis_client.setex(
            f"collab:{cache_key}",
            3600,
            json.dumps(response)
        )

        return response

    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.post("/recommendations/similar-items")
async def get_similar_items(item_id: int, n_similar: int = 10):
    """
    Get items similar to given item (content-based)
    Used for "Customers also viewed" sections
    """
    try:
        similar = content_recommender.similar_items(item_id, n_similar)

        return {
            "item_id": item_id,
            "similar_items": [
                {"item_id": iid, "similarity": score}
                for iid, score in similar
            ]
        }
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.get("/health")
async def health_check():
    """Health check endpoint for load balancer"""
    return {"status": "healthy", "models_loaded": True}

Evaluation Metrics

Measuring Recommendation Quality

# evaluation.py - Recommendation system metrics
import numpy as np
from typing import List, Set

def precision_at_k(recommended: List[int], relevant: Set[int], k: int) -> float:
    """
    Precision@K: Fraction of recommended items that are relevant

    Args:
        recommended: List of recommended item IDs (ordered)
        relevant: Set of relevant item IDs (ground truth)
        k: Number of top recommendations to consider

    Returns:
        Precision score (0-1)
    """
    recommended_at_k = recommended[:k]
    relevant_at_k = [item for item in recommended_at_k if item in relevant]

    if k == 0:
        return 0.0

    return len(relevant_at_k) / k

def recall_at_k(recommended: List[int], relevant: Set[int], k: int) -> float:
    """
    Recall@K: Fraction of relevant items that were recommended
    """
    recommended_at_k = recommended[:k]
    relevant_at_k = [item for item in recommended_at_k if item in relevant]

    if len(relevant) == 0:
        return 0.0

    return len(relevant_at_k) / len(relevant)

def mean_average_precision(
    recommended_lists: List[List[int]],
    relevant_lists: List[Set[int]]
) -> float:
    """
    MAP: Mean Average Precision across all users
    Industry standard metric for recommendation systems
    """
    average_precisions = []

    for recommended, relevant in zip(recommended_lists, relevant_lists):
        if len(relevant) == 0:
            continue

        precisions = []
        num_hits = 0

        for k, item in enumerate(recommended, 1):
            if item in relevant:
                num_hits += 1
                precision = num_hits / k
                precisions.append(precision)

        if precisions:
            average_precisions.append(np.mean(precisions))

    return np.mean(average_precisions) if average_precisions else 0.0

def ndcg_at_k(recommended: List[int], relevant: Dict[int, float], k: int) -> float:
    """
    NDCG@K: Normalized Discounted Cumulative Gain
    Accounts for position of relevant items (higher rank = better)

    Args:
        recommended: Ordered list of recommended items
        relevant: Dict mapping item_id to relevance score (e.g., rating)
        k: Number of recommendations to consider
    """
    recommended_at_k = recommended[:k]

    # Discounted Cumulative Gain
    dcg = sum(
        relevant.get(item, 0) / np.log2(idx + 2)
        for idx, item in enumerate(recommended_at_k)
    )

    # Ideal DCG (if recommendations were perfectly ordered)
    ideal_relevant = sorted(relevant.values(), reverse=True)[:k]
    idcg = sum(
        score / np.log2(idx + 2)
        for idx, score in enumerate(ideal_relevant)
    )

    return dcg / idcg if idcg > 0 else 0.0

Best Practices and Limitations

Production Checklist

Data Quality:

✅ Remove bots and fraudulent interactions
✅ Handle outliers (users who rate everything 5 or 1)
✅ Deal with popularity bias (don’t only recommend bestsellers)
✅ Update models regularly (weekly/daily for fast-changing catalogs)

Cold Start Solutions:

✅ New users: Use content-based recommendations + popular items
✅ New items: Require minimum metadata, boost in search results
✅ Hybrid approach: Blend collaborative + content-based

Performance:

✅ Pre-compute recommendations offline (batch jobs)
✅ Cache recommendations with Redis (1-24 hour TTL)
✅ Use approximate nearest neighbors (Annoy, Faiss) for millions of items
✅ Monitor latency: <100ms for API response time

A/B Testing:

✅ Always A/B test new recommendation algorithms
✅ Metrics: CTR, conversion rate, revenue per user, diversity
✅ Run tests for minimum 2 weeks to account for day-of-week effects

Known Limitations

Limitation	Impact	Mitigation
Cold Start	New users/items have poor recommendations	Use content-based, popular items, or ask preferences
Popularity Bias	Always recommends bestsellers	Use diversity metrics, apply debiasing algorithms
Filter Bubbles	Users only see similar items	Inject serendipitous recommendations (10-20%)
Data Sparsity	Most users rate < 1% of items	Use implicit feedback (clicks, views), not just purchases
Scalability	Millions of users × items = huge matrix	Use matrix factorization (ALS), approximate methods
Freshness	New items get no exposure	Boost new items in ranking, use exploration strategies

Common Pitfalls

❌ Recommending Out-of-Stock Items

# Always filter by inventory before returning recommendations
def filter_in_stock(recommendations, inventory_df):
    in_stock_ids = inventory_df[inventory_df['quantity'] > 0]['item_id'].values
    return [(iid, score) for iid, score in recommendations if iid in in_stock_ids]

❌ Not Handling User Privacy

# GDPR compliance: Allow users to delete their data
def delete_user_data(user_id):
    # Remove from training data
    # Retrain models without user
    # Delete cached recommendations
    pass

Conclusion and Resources

AI-powered e-commerce recommendation systems drive significant business value through personalization. Key takeaways:

Collaborative Filtering: Discovers patterns from user behavior, excellent for large datasets
Content-Based: Works for cold start, uses product metadata
Hybrid Approaches: Combine multiple signals for best results
LLM Integration: Automate content generation for scale
Evaluation: Use MAP, NDCG@K, and business metrics (CTR, revenue)

Production systems require careful attention to cold start, scalability, and continuous retraining as catalog and user preferences evolve.

Further Resources:

Surprise Library Docs: http://surpriselib.com/ (collaborative filtering)
Implicit Library: https://github.com/benfred/implicit (ALS, BPR algorithms)
RecSys Papers: https://recsys.acm.org/ (latest research)
Google Recommendations AI: https://cloud.google.com/recommendations
Amazon Personalize: https://aws.amazon.com/personalize/
Microsoft Recommenders: https://github.com/microsoft/recommenders