LEANN元数据筛选使用指南

LEANN 元数据筛选使用指南

概述

Leann 具备元数据筛选功能，可让您根据分块时设置的任意元数据字段对搜索结果进行筛选。此功能支持多种用例，例如无剧透的图书搜索、按日期/类型筛选文档、按文件类型搜索代码，以及更多潜在应用场景。

基本用法

向您的文档添加元数据

在构建索引时，为每个文本分块添加元数据：

from leann.api import LeannBuilder

builder = LeannBuilder("hnsw")

# Add text with metadata
builder.add_text(
    text="Chapter 1: Alice falls down the rabbit hole",
    metadata={
        "chapter": 1,
        "character": "Alice",
        "themes": ["adventure", "curiosity"],
        "word_count": 150
    }
)

builder.build_index("alice_in_wonderland_index")
​

使用元数据筛选器进行搜索

在搜索调用中使用 metadata_filters 参数：

from leann.api import LeannSearcher

searcher = LeannSearcher("alice_in_wonderland_index")

# Search with filters
results = searcher.search(
    query="What happens to Alice?",
    top_k=10,
    metadata_filters={
        "chapter": {"<=": 5},           # Only chapters 1-5
        "spoiler_level": {"!=": "high"} # No high spoilers
    }
)
​

筛选器语法

基本结构

metadata_filters = {
    "field_name": {"operator": value},
    "another_field": {"operator": value}
}
​

支持的操作符

比较操作符

• "==": Equal to
• "!=": Not equal to
• "<": Less than
• "<=": Less than or equal
• ">": Greater than
• ">=": 大于或等于

# Examples
{"chapter": {"==": 1}}           # Exactly chapter 1
{"page": {">": 100}}            # Pages after 100
{"rating": {">=": 4.0}}         # Rating 4.0 or higher
{"word_count": {"<": 500}}      # Short passages
​

成员操作符

• "in": Value is in list
• "not_in": 值不在列表中

# Examples
{"character": {"in": ["Alice", "Bob"]}}      # Alice OR Bob
{"genre": {"not_in": ["horror", "thriller"]}} # Exclude genres
{"tags": {"in": ["fiction", "adventure"]}}   # Any of these tags
​

字符串操作符

• "contains": String contains substring
• "starts_with": String starts with prefix
• "ends_with": 字符串以后缀结尾

# Examples
{"title": {"contains": "alice"}}        # Title contains "alice"
{"filename": {"ends_with": ".py"}}      # Python files
{"author": {"starts_with": "Dr."}}      # Authors with "Dr." prefix
​

布尔操作符

• "is_true": Field is truthy
• "is_false": 字段为假值

# Examples
{"is_published": {"is_true": True}}     # Published content
{"is_draft": {"is_false": False}}       # Not drafts
​

同一字段上的多个操作符

您可以对同一字段应用多个操作符（AND 逻辑）：

metadata_filters = {
    "word_count": {
        ">=": 100,    # At least 100 words
        "<=": 500     # At most 500 words
    }
}
​

复合筛选器

多个字段通过 AND 逻辑组合：

metadata_filters = {
    "chapter": {"<=": 10},              # Up to chapter 10
    "character": {"==": "Alice"},       # About Alice
    "spoiler_level": {"!=": "high"}     # No major spoilers
}
​

用例示例

1. 无剧透的图书搜索

# Reader has only read up to chapter 5
def search_spoiler_free(query, max_chapter):
    return searcher.search(
        query=query,
        metadata_filters={
            "chapter": {"<=": max_chapter},
            "spoiler_level": {"in": ["none", "low"]}
        }
    )

results = search_spoiler_free("What happens to Alice?", max_chapter=5)
​

2. 按日期管理文档

# Find recent documents
recent_docs = searcher.search(
    query="project updates",
    metadata_filters={
        "date": {">=": "2024-01-01"},
        "document_type": {"==": "report"}
    }
)
​

3. 按文件类型搜索代码

# Search only Python files
python_code = searcher.search(
    query="authentication function",
    metadata_filters={
        "file_extension": {"==": ".py"},
        "lines_of_code": {"<": 100}
    }
)
​

4. 按受众筛选内容

# Age-appropriate content
family_content = searcher.search(
    query="adventure stories",
    metadata_filters={
        "age_rating": {"in": ["G", "PG"]},
        "content_warnings": {"not_in": ["violence", "adult_themes"]}
    }
)
​

5. 多卷系列管理

# Search across first 3 books only
early_series = searcher.search(
    query="character development",
    metadata_filters={
        "series": {"==": "Harry Potter"},
        "book_number": {"<=": 3}
    }
)
​

运行示例

您可以通过我们的无剧透图书 RAG 示例直观地看到元数据筛选的实际效果：

# Don't forget to set up the environment
uv venv
source .venv/bin/activate

# Set your OpenAI API key (required for embeddings, but you can update the example locally and use ollama instead)
export OPENAI_API_KEY="your-api-key-here"

# Run the spoiler-free book RAG example
uv run examples/spoiler_free_book_rag.py
​

此示例演示了：

• 使用元数据构建索引（章节号、角色、主题、地点）
• 使用筛选器进行搜索以避免剧透（例如，仅显示前 5 章的结果）
• 针对处于不同阅读阶段的读者的不同场景

该示例以《爱丽丝梦游仙境》作为样本数据，展示了如何在不泄露后续章节情节的情况下搜索相关信息。

高级模式

自定义分块与元数据

def chunk_book_with_metadata(book_text, book_info):
    chunks = []

    for chapter_num, chapter_text in parse_chapters(book_text):
        # Extract entities, themes, etc.
        characters = extract_characters(chapter_text)
        themes = classify_themes(chapter_text)
        spoiler_level = assess_spoiler_level(chapter_text, chapter_num)

        # Create chunks with rich metadata
        for paragraph in split_paragraphs(chapter_text):
            chunks.append({
                "text": paragraph,
                "metadata": {
                    "book_title": book_info["title"],
                    "chapter": chapter_num,
                    "characters": characters,
                    "themes": themes,
                    "spoiler_level": spoiler_level,
                    "word_count": len(paragraph.split()),
                    "reading_level": calculate_reading_level(paragraph)
                }
            })

    return chunks
​

性能考虑

高效筛选策略

1. 搜索后筛选：在向量搜索之后应用筛选器，对于典型结果集（10-100 条结果）应具有高效性。

2. 元数据设计：保持元数据字段简单，避免深度嵌套结构。

最佳实践

1. 一致的元数据模式：在所有文档中使用一致的字段名和值类型。

2. 合理的元数据大小：保持元数据大小合理，避免存储开销过大。

3. 类型一致性：对相同字段使用一致的数据类型（例如，章节号始终为整数）。

4. 索引多个粒度：考虑在不同级别（段落、小节、章节）进行分块，并配备适当的元数据。

向现有索引添加元数据

要为现有索引添加元数据筛选功能，您需要重新构建索引并加入元数据：

# Read existing passages and add metadata
def add_metadata_to_existing_chunks(chunks):
    for chunk in chunks:
        # Extract or assign metadata based on content
        chunk["metadata"] = extract_metadata(chunk["text"])
    return chunks

# Rebuild index with metadata
enhanced_chunks = add_metadata_to_existing_chunks(existing_chunks)
builder = LeannBuilder("hnsw")
for chunk in enhanced_chunks:
    builder.add_text(chunk["text"], chunk["metadata"])
builder.build_index("enhanced_index")
​

免责声明：
本文档采用基于机器的 AI 翻译服务进行翻译。尽管我们力求准确，但请注意，自动翻译可能存在错误或不准确之处。应以原文语言版本的文档作为权威依据。如需获取关键信息，建议使用专业的人工翻译。对于因使用本翻译而产生的任何误解或误读，我们概不负责。