Search the docs, videos, blog, apps, teams, skills and release notes.
At a glance
What it is
Vector database engineering covering Pinecone, Weaviate, Qdrant, pgvector, Milvus, and Chroma selection and configuration, indexing strategies (HNSW, IVF, PQ), similarity search optimization, hybrid search, filtering, multi-tenancy, scaling patterns, and production operations.
A skill is a written procedure a bot loads when a job calls for it. This one is a single file, SKILL.md, and the whole file is on this page.
The whole file, exactly as it downloads. Read it before you rely on it.
SKILL.md
377 lines · 1,733 words
Vector databases are purpose-built systems for storing, indexing, and querying high-dimensional embedding vectors at scale. They underpin semantic search, RAG systems, recommendation engines, and similarity-based applications. This skill covers database selection, index configuration, query optimization, hybrid search, scaling, and production operations across major platforms.
Vector database engineering covering Pinecone, Weaviate, Qdrant, pgvector, Milvus, and Chroma selection and configuration, indexing strategies (HNSW, IVF, PQ), similarity search optimization, hybrid search, filtering, multi-tenancy, scaling patterns, and production operations.
Nothing. It ships with Murage, which is open source. Your bots run on the AI plan you already have.
Yes. Open it in Settings → Skills to read, edit or duplicate it. Skill Guard scans it again after every edit.
Download the app, connect the AI you already pay for, and tell your Chief of Staff what needs doing. Plan on about ten minutes from install to a working team.
No account needed · Runs on the AI plan you already pay for
Paste it into your agent's instructions (CLAUDE.md, AGENTS.md or a custom GPT), or add it to a bot in Murage.
Ask Ember for the bot, then switch the skill on in its settings.
Step 1
Send your Chief of Staff this message. It proposes the bot and its role first, and creates it when you say go.
Send this to Ember
“Create a bot that uses the Vector DB Engineer skill, and show me the plan first.”
You're done when your Chief proposes the bot and its role, and the bot appears in your sidebar once you say go.
Step 2
You're done when you see the bot’s skills and the library.
Step 3
You're done when the skill shows as on for that bot.
Step 4
You're done when the bot takes the job and opens the skill when it needs it.
Prefer to do it yourself? Choose + in the sidebar, then New Bot.Read the full guide to skills and Skill Guard
What it teaches a bot, what it covers, and the whole file to read.
What it covers
Overview
Database Comparison
Indexing Strategies
Platform Setup
Hybrid Search
Multi-Tenancy
Production Operations
Checklist
When to Use
Output Format
Example
Edge Cases
| Feature | Pinecone | Weaviate | Qdrant | pgvector | Milvus | Chroma |
|---|
| Type | Managed | Both | Both | Extension | Both | OSS |
| Language | Proprietary | Go | Rust | C (PG) | Go/C++ | Python |
| Max Vectors | 100B+ | Billions | Billions | ~10M | Billions | Millions |
| Hybrid Search | Yes | Yes (BM25) | Yes (sparse) | With tsvector | Yes | No |
| Multi-tenancy | Namespaces | Native | Collections | Schemas/RLS | Partitions | Collections |
| Quantization | Auto | PQ, BQ | Scalar, PQ | Half-prec | PQ, SQ | No |
| GPU Accel | Server-side | No | No | No | Yes | No |
| Database | Free Tier | 10M vectors (1536d) |
|---|---|---|
| Pinecone | 100K vectors | ~$70/mo |
| Weaviate Cloud | 50K vectors | ~$150/mo |
| Qdrant Cloud | 1M vectors | ~$100/mo |
| pgvector | Self-hosted | DB hosting cost |
| Zilliz (Milvus) | 200K vectors | ~$200/mo |
| Chroma | Unlimited local | Server cost |
Already using PostgreSQL?
YES -> pgvector (zero new infra)
Over 5-10M vectors? -> Add dedicated vector DB
Prototyping?
YES -> Chroma (install the package via pip, zero config)
Need fully managed, minimal ops?
YES -> Pinecone serverless or Qdrant Cloud
Need hybrid search (vector + keyword)?
YES -> Weaviate (BM25) or Qdrant (sparse vectors)
Need GPU-accelerated search?
YES -> Milvus / Zilliz
Performance critical, low-level control?
YES -> Qdrant (Rust) or Milvus
Cost sensitive at scale?
YES -> Self-hosted Qdrant or Milvus
Default -> Pinecone serverless (simplest operations)| Index | Query Speed | Memory | Recall@10 | Best For |
|---|---|---|---|---|
| Flat (Brute Force) | Slow O(n) | Low | 100% | <100K, ground truth |
| HNSW | Very Fast | High | 95-99% | General purpose, <50M |
| IVF-Flat | Fast | Medium | 90-98% | Large datasets, GPU |
| IVF-PQ | Very Fast | Low | 85-95% | Billions of vectors |
| DiskANN | Fast | Very Low | 95-99% | SSD-based, cost-sensitive |
HNSW_PROFILES = {
"speed": {"M": 8, "ef_construction": 64, "ef_search": 32}, # ~95% recall
"balanced":{"M": 16, "ef_construction": 128, "ef_search": 64}, # ~97% recall
"quality": {"M": 32, "ef_construction": 256, "ef_search": 128}, # ~99% recall
}
def estimate_hnsw_memory(n_vectors: int, dimensions: int, M: int = 16) -> dict:
"""Estimate HNSW index memory requirements."""
vector_bytes = n_vectors * dimensions * 4
graph_bytes = n_vectors * M * 2 * 8
overhead = n_vectors * 64
total = vector_bytes + graph_bytes + overhead
return {
"vectors_gb": round(vector_bytes / 1e9, 2),
"graph_gb": round(graph_bytes / 1e9, 2),
"total_gb": round(total / 1e9, 2),
"recommended_ram_gb": round(total / 1e9 * 1.3, 2),
}
# Example: 10M vectors, 1536 dims, M=16 -> ~62 GB totalfrom pinecone import Pinecone, ServerlessSpec
pc = Pinecone(api_key="YOUR_KEY")
pc.create_index(
name="production-search", dimension=1536, metric="cosine",
spec=ServerlessSpec(cloud="aws", region="us-east-1"),
)
index = pc.Index("production-search")
# Upsert with metadata
index.upsert(vectors=[{
"id": "doc_001", "values": embedding,
"metadata": {"source": "manual.pdf", "department": "engineering"},
}], namespace="production")
# Query with filter
results = index.query(
vector=query_embedding, top_k=10,
filter={"department": {"$eq": "engineering"}},
include_metadata=True, namespace="production",
)import weaviate
from weaviate.classes.config import Configure, Property, DataType
client = weaviate.connect_to_local()
collection = client.collections.create(
name="Document",
vectorizer_config=Configure.Vectorizer.text2vec_openai(model="text-embedding-3-small"),
properties=[
Property(name="content", data_type=DataType.TEXT),
Property(name="source", data_type=DataType.TEXT),
],
)
# Hybrid search (vector + BM25)
results = collection.query.hybrid(
query="How do vector databases work?",
alpha=0.7, # 0=pure BM25, 1=pure vector
limit=10,
)from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance, PointStruct, Filter, FieldCondition, MatchValue
client = QdrantClient(url="[reference URL]")
client.create_collection(
collection_name="documents",
vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
hnsw_config={"m": 16, "ef_construct": 128},
)
client.create_payload_index(collection_name="documents", field_name="dept", field_schema="keyword")
# Search with filter
results = client.query_points(
collection_name="documents", query=query_embedding, limit=10,
query_filter=Filter(must=[FieldCondition(key="dept", match=MatchValue(value="eng"))]),
)CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE documents (
id SERIAL PRIMARY KEY,
content TEXT NOT NULL,
metadata JSONB DEFAULT '{}',
embedding vector(1536)
);
CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 128);
-- Similarity search with filter
SET hnsw.ef_search = 100;
SELECT id, content, 1 - (embedding <=> $1::vector) AS similarity
FROM documents
WHERE metadata->>'source' = 'manual.pdf'
ORDER BY embedding <=> $1::vector LIMIT 10;
-- Hybrid: vector + full-text
SELECT id, content,
0.7 * (1 - (embedding <=> $1::vector)) +
0.3 * ts_rank(to_tsvector('english', content), plainto_tsquery($2)) AS score
FROM documents
WHERE to_tsvector('english', content) @@ plainto_tsquery($2)
ORDER BY score DESC LIMIT 10;def reciprocal_rank_fusion(result_lists: list[list[dict]], k: int = 60, weights: list[float] = None) -> list[dict]:
"""Combine ranked lists (vector + keyword) using RRF."""
weights = weights or [1.0] * len(result_lists)
scores = {}
for weight, results in zip(weights, result_lists):
for rank, doc in enumerate(results, 1):
if doc["id"] not in scores:
scores[doc["id"]] = {"doc": doc, "score": 0.0}
scores[doc["id"]]["score"] += weight / (k + rank)
return [s["doc"] | {"rrf_score": s["score"]}
for s in sorted(scores.values(), key=lambda x: x["score"], reverse=True)]Pre-filtering (filter before vector search):
Selective filter (<10% matches) -> Pre-filter
Efficient but may miss semantically relevant results
Post-filtering (filter after vector search):
Broad filter (>50% matches) -> Post-filter or skip filter
Full semantic quality but may return <top_k results
In between -> Pre-filter with over-get (search top_k*3, then filter)| Pattern | Isolation | Cost | Best For |
|---|---|---|---|
| Metadata filtering | None | Lowest | Many small tenants |
| Namespace per tenant | Logical | Low | Small-medium tenants |
| Collection per tenant | Strong | High | Strict isolation |
| Cluster per tenant | Physical | Highest | Enterprise compliance |
class MultiTenantVectorStore:
def __init__(self, client, collection: str):
self.client = client
self.collection = collection
def upsert(self, tenant_id: str, vectors: list[dict]):
for v in vectors:
v.setdefault("metadata", {})["tenant_id"] = tenant_id
self.client.upsert(collection=self.collection, points=vectors)
def search(self, tenant_id: str, query_vector: list[float], top_k: int = 10):
return self.client.search(
collection=self.collection, query_vector=query_vector,
limit=top_k, filter={"tenant_id": {"$eq": tenant_id}},
)| Metric | Warning | Critical | Action |
|---|---|---|---|
| Query latency p99 | >200ms | >500ms | Scale up, tune ef_search |
| Memory usage | >80% | >90% | Scale or enable disk index |
| Recall@10 | <95% | <90% | Increase M or ef_search |
| Error rate | >0.1% | >1% | Check connectivity |
1. Measure baseline (p50/p95/p99 latency, recall, memory)
2. Identify bottleneck:
Latency high -> Increase ef_search, enable quantization, index filters
Recall low -> Increase M and ef_construction, increase ef_search
Memory high -> Enable quantization, use disk index, reduce dimensions
3. Validate changes (re-measure all metrics, load test)Vertical: More RAM (larger index in memory), faster SSD, more CPU cores
Limit: Single machine capacity
Horizontal: Hash/range sharding, read replicas
Native support: Milvus, Qdrant distributed, Pinecone
Tiered Storage:
Hot: In-memory HNSW (frequent access)
Warm: Disk-based index (recent data)
Cold: Object storage (archive)Select vector database based on scale, hosting, and feature needs
Choose index type (HNSW for most, IVF-PQ for billions)
Tune HNSW parameters (M, ef_construction, ef_search)
Implement hybrid search if keyword matching adds value
Set up payload/metadata indexes for filtered search
Design multi-tenancy strategy
Enable quantization if memory constrained
Implement backup and recovery procedures
Set up monitoring for latency, memory, recall, errors
Load test at expected QPS with realistic patterns
Plan scaling strategy (vertical first, horizontal when needed)
Document index parameters, schemas, and runbooks
Use this skill when:
Designing or implementing vector db engineer solutions
Reviewing or improving existing vector db engineer approaches
Making architectural or implementation decisions about vector db engineer
Learning vector db engineer patterns and best practices
Troubleshooting vector db engineer-related issues
Do NOT use this skill when:
The question is about a fundamentally different technology domain
A more specific sibling skill covers the exact topic needed
The user needs a complete hands-on tutorial rather than expert guidance
# Vector Db Engineer Analysis
## Context Assessment
[Situation summary and constraints]
## Recommended Approach
[Primary recommendation with rationale]
## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]
## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]
## Next Steps
- [Immediate action item]
- [Follow-up action item]Input: "Help me implement vector db engineer for a medium-scale production application"
Output: A structured analysis covering current state assessment, recommended vector db engineer approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.
Legacy system integration: When vector db engineer must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities
SKILL.md
377 lines · 1,733 words
The text of SKILL.md is shown as it downloads, under the Apache-2.0 license, by Ferrox Labs. The license is the one in the file's own frontmatter.
Your new bot
Vector DB Engineer bot
Waiting in your sidebar once you say go.
No account needed · Runs on the AI plan you already pay for