Fix Dockerfile build issue
This commit is contained in:
kyy committed
1 parent
6814230bfb
commit
9323aa254a
228 files changed
+467
-3488
No files matched your search
@@ -1,2 +1 @@
|
||||
OPENAI_API_KEY=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
|
||||
LLAMA_CLOUD_API_KEY=llx-MkHkuDxnSxXEHvsIPAtjEZl4iSB8pHS1mgYDVZQlA690LUub
|
||||
+18
-7
@@ -7,8 +7,9 @@ ENV PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
PIP_NO_CACHE_DIR=1
|
||||
|
||||
# Copy only requirements files first to leverage Docker cache
|
||||
COPY pyproject.toml ./
|
||||
# Copy only necessary files
|
||||
COPY pyproject.toml ./
|
||||
COPY requirements.txt ./
|
||||
|
||||
# Install system and Python dependencies in a single layer
|
||||
RUN apt-get update && \
|
||||
@@ -19,10 +20,20 @@ RUN apt-get update && \
|
||||
pip install --no-cache-dir --upgrade pip setuptools setuptools-scm && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy project files
|
||||
COPY . .
|
||||
|
||||
# Install base project
|
||||
RUN pip install --no-cache-dir -e .
|
||||
RUN pip install -r requirements.txt
|
||||
|
||||
CMD ["bash"]
|
||||
# Copy project files
|
||||
COPY autorag /usr/src/app/autorag
|
||||
COPY main.py /usr/src/app/main.py
|
||||
COPY making.sh /usr/src/app/making.sh
|
||||
COPY entrypoint.sh /usr/src/app/entrypoint.sh
|
||||
|
||||
# Set permissions for entrypoint
|
||||
RUN chmod +x /usr/src/app/entrypoint.sh
|
||||
|
||||
# Use entrypoint.sh as the container entrypoint
|
||||
ENTRYPOINT ["/usr/src/app/entrypoint.sh"]
|
||||
|
||||
# Set a default command (optional, can be overridden)
|
||||
CMD ["bash"]
|
||||
@@ -1,51 +1,68 @@
|
||||
# AutoRAG Evaluation
|
||||
# 📊 AutoRAG Evaluation
|
||||
|
||||
이 문서는 AutoRAG을 활용하여 RAG 파이프라인을 설정하고 평가하는 과정에 대한 안내입니다.
|
||||
AutoRAG Evaluation은 RAG(Relevance-Augmented Generation) 파이프라인을 평가하기 위한 도구입니다.
|
||||
이 문서는 데이터 생성부터 RAG 평가 및 대시보드 활용까지의 전체 과정에 대한 안내를 제공합니다.
|
||||
|
||||
---
|
||||
|
||||
## 📌 환경 세팅
|
||||
## **🚀 1. 환경 설정**
|
||||
|
||||
1. `.env` 파일을 설정합니다. (`.env.sample` 참고)
|
||||
2. Docker 이미지를 빌드합니다.
|
||||
### **1️⃣ `.env` 파일 설정**
|
||||
|
||||
`.env.sample` 파일을 참고하여 `.env` 파일을 생성하고 필요한 환경 변수를 설정합니다.
|
||||
|
||||
### **2️⃣ Docker 이미지 빌드**
|
||||
|
||||
```bash
|
||||
docker build -t autorag-base .
|
||||
docker build -t autorag-base .
|
||||
```
|
||||
|
||||
3. Docker Compose를 실행하여 서비스를 시작합니다.
|
||||
### **3️⃣ Docker Compose 백그라운드 실행**
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
4. Hugginface embedding, Ollama LLM을 위한 추가 모듈 설치를 시작합니다.
|
||||
### **4️⃣ 실행 모드 (Docker)**
|
||||
|
||||
AutoRAG Evaluation은 실행 모드에 따라 다르게 동작합니다.
|
||||
Docker 실행 시 아래 명령어를 사용하여 모드를 선택하세요.
|
||||
|
||||
```bash
|
||||
pip install -r requirements_custom.txt
|
||||
docker run autorag-base <mode>
|
||||
```
|
||||
|
||||
| 실행 모드 | 설명 |
|
||||
| ---------- | -------------------------------- |
|
||||
| `data-gen` | 데이터 생성 (`making.sh` 실행) |
|
||||
| `evaluate` | RAG 평가 (`main.py` 실행) |
|
||||
| `bash` | 컨테이너 내부 접근 (`/bin/bash`) |
|
||||
|
||||
예시:
|
||||
|
||||
```bash
|
||||
docker run autorag-base data-gen
|
||||
docker run autorag-base evaluate
|
||||
docker run autorag-base bash
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📂 데이터 생성
|
||||
## **📂 2. 데이터 생성**
|
||||
|
||||
RAG 평가를 위해 **QA 데이터 세트**와 **코퍼스 데이터 세트**가 필요합니다.
|
||||
|
||||
### 1️⃣ 프로젝트 폴더 생성
|
||||
### **1️⃣ 프로젝트 폴더 생성**
|
||||
|
||||
```bash
|
||||
cd projects
|
||||
mkdir -p "project_name"
|
||||
cd "project_name"
|
||||
mkdir -p raw_data config
|
||||
mkdir -p example_project/{raw_data,config}
|
||||
```
|
||||
|
||||
- **`raw_data/`**: 분석할 원본 데이터를 저장 (`.pdf` 등)
|
||||
- **`config/`**: 파싱(`parse.yaml`), 청킹(`chunk.yaml`) 설정 파일을 저장
|
||||
- **`raw_data/`**: 원본 데이터 저장 (`.pdf`, `.txt` 등)
|
||||
- **`config/`**: 설정 파일 저장 (`parse.yaml`, `chunk.yaml` 등)
|
||||
|
||||
### 2️⃣ 파싱 설정 (`parse.yaml`)
|
||||
|
||||
파싱 모듈을 설정합니다.
|
||||
### **2️⃣ 파싱 설정 (`parse.yaml`)**
|
||||
|
||||
```yaml
|
||||
modules:
|
||||
@@ -53,11 +70,7 @@ modules:
|
||||
parse_method: pdfminer
|
||||
```
|
||||
|
||||
여러 개의 파싱 모듈을 동시에 사용할 수도 있습니다.
|
||||
|
||||
### 3️⃣ 청킹 설정 (`chunk.yaml`)
|
||||
|
||||
청킹 모듈을 설정합니다.
|
||||
### **3️⃣ 청킹 설정 (`chunk.yaml`)**
|
||||
|
||||
```yaml
|
||||
modules:
|
||||
@@ -68,25 +81,21 @@ modules:
|
||||
add_file_name: en
|
||||
```
|
||||
|
||||
여러 개의 청킹 모듈을 사용할 경우, QA 데이터와 매핑해야 합니다.
|
||||
|
||||
### 4️⃣ QA 데이터 생성
|
||||
|
||||
`raw_data/`에 저장된 파일을 바탕으로 **파싱 → 청킹 → QA 데이터 생성**을 진행합니다.
|
||||
QA 데이터는 `GPT-4o-mini` 모델을 사용하여 **20건**을 생성합니다.
|
||||
### **4️⃣ QA 데이터 생성**
|
||||
|
||||
```bash
|
||||
cd autorag-workspace
|
||||
sh making.sh
|
||||
bash making.sh
|
||||
```
|
||||
|
||||
> **참고:** `GPT-4o-mini` 모델을 사용하여 자동으로 QA 데이터를 생성합니다. (기본 20건)
|
||||
|
||||
---
|
||||
|
||||
## 🔍 RAG Pipeline 평가
|
||||
## **🔍 3. RAG Pipeline 평가**
|
||||
|
||||
### 1️⃣ Ollama 모델 다운로드
|
||||
### **1️⃣ Ollama 모델 다운로드**
|
||||
|
||||
WSL(Windows Subsystem for Linux)에서 실행합니다.
|
||||
WSL(Windows Subsystem for Linux)에서 실행해야 합니다.
|
||||
|
||||
```bash
|
||||
docker exec -it autorag-ollama bash
|
||||
@@ -94,61 +103,32 @@ ollama pull phi4
|
||||
ollama list
|
||||
```
|
||||
|
||||
### 2️⃣ AutoRAG 평가 실행
|
||||
### **2️⃣ 평가 실행**
|
||||
|
||||
```bash
|
||||
cd Autorag-workspace
|
||||
python main.py
|
||||
```
|
||||
|
||||
### 3️⃣ 평가 결과 확인
|
||||
### **3️⃣ 평가 결과 확인**
|
||||
|
||||
평가 결과는 프로젝트 폴더 내 `benchmark_*` 경로에 저장됩니다.
|
||||
|
||||
#### ✅ 전체 파이프라인 평가 결과
|
||||
#### **✅ 전체 평가 결과**
|
||||
|
||||
```bash
|
||||
cd projects/프로젝트이름/benchmark_{*}/*/
|
||||
summary.csv
|
||||
cd projects/example_project/benchmark_*/summary.csv
|
||||
```
|
||||
|
||||
#### ✅ 세부 평가 결과
|
||||
#### **✅ 세부 평가 결과**
|
||||
|
||||
- **검색기 노드 라인 평가 결과**
|
||||
```bash
|
||||
cd ./retrieve_node_line
|
||||
summary.csv
|
||||
```
|
||||
- **검색 노드 평가 결과**
|
||||
```bash
|
||||
cd ./retrieve_node_line/retrival
|
||||
summary.csv
|
||||
```
|
||||
- **리랭커 노드 평가 결과**
|
||||
```bash
|
||||
cd ./retrieve_node_line/passage_reranker
|
||||
summary.csv
|
||||
```
|
||||
- **생성기 노드 라인 평가 결과**
|
||||
```bash
|
||||
cd ./post_retrieve_node_line
|
||||
summary.csv
|
||||
```
|
||||
- **생성 노드 평가 결과**
|
||||
```bash
|
||||
cd ./post_retrieve_node_line/generator
|
||||
summary.csv
|
||||
```
|
||||
| 평가 항목 | 파일 경로 |
|
||||
| ------------------------- | --------------------------------------------------- |
|
||||
| **검색기 노드 라인 평가** | `./retrieve_node_line/summary.csv` |
|
||||
| **검색 노드 평가** | `./retrieve_node_line/retrieval/summary.csv` |
|
||||
| **리랭커 노드 평가** | `./retrieve_node_line/passage_reranker/summary.csv` |
|
||||
| **생성기 노드 라인 평가** | `./post_retrieve_node_line/summary.csv` |
|
||||
| **생성 노드 평가** | `./post_retrieve_node_line/generator/summary.csv` |
|
||||
|
||||
> 📌 **참고:** `./projects/example_01` 폴더는 데이터 생성부터 평가까지 진행된 예제입니다.
|
||||
> 📌 `./projects/example_01` 폴더에는 데이터 생성부터 평가까지 진행된 예제 파일이 포함되어 있습니다.
|
||||
|
||||
---
|
||||
|
||||
## 📊 평가 대시보드 실행
|
||||
|
||||
```bash
|
||||
cd Autorag-workspace
|
||||
sh dashboard.sh
|
||||
```
|
||||
|
||||
AutoRAG 평가 결과를 자세히 확인할 수 있습니다.
|
||||
@@ -1,209 +0,0 @@
|
||||
import importlib.resources
|
||||
import logging
|
||||
import os
|
||||
import pathlib
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import click
|
||||
import nest_asyncio
|
||||
|
||||
from autorag import dashboard
|
||||
from autorag.deploy import extract_best_config as original_extract_best_config
|
||||
from autorag.deploy.api import ApiRunner
|
||||
from autorag.evaluator import Evaluator
|
||||
from autorag.validator import Validator
|
||||
|
||||
logger = logging.getLogger("AutoRAG")
|
||||
|
||||
autorag_dir = os.path.dirname(os.path.realpath(__file__))
|
||||
version_file = os.path.join(autorag_dir, "VERSION")
|
||||
with open(version_file, "r") as f:
|
||||
__version__ = f.read().strip()
|
||||
|
||||
|
||||
@click.group()
|
||||
@click.version_option(__version__)
|
||||
def cli():
|
||||
pass
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--config",
|
||||
"-c",
|
||||
help="Path to config yaml file. Must be yaml or yml file.",
|
||||
type=str,
|
||||
)
|
||||
@click.option(
|
||||
"--qa_data_path", help="Path to QA dataset. Must be parquet file.", type=str
|
||||
)
|
||||
@click.option(
|
||||
"--corpus_data_path", help="Path to corpus dataset. Must be parquet file.", type=str
|
||||
)
|
||||
@click.option(
|
||||
"--project_dir", help="Path to project directory.", type=str, default=None
|
||||
)
|
||||
@click.option(
|
||||
"--skip_validation",
|
||||
help="Skip validation or not. Default is False.",
|
||||
type=bool,
|
||||
default=False,
|
||||
)
|
||||
def evaluate(config, qa_data_path, corpus_data_path, project_dir, skip_validation):
|
||||
if not config.endswith(".yaml") and not config.endswith(".yml"):
|
||||
raise ValueError(f"Config file {config} is not a yaml or yml file.")
|
||||
if not os.path.exists(config):
|
||||
raise ValueError(f"Config file {config} does not exist.")
|
||||
evaluator = Evaluator(qa_data_path, corpus_data_path, project_dir=project_dir)
|
||||
evaluator.start_trial(config, skip_validation=skip_validation)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--config_path", type=str, help="Path to extracted config yaml file.", default=None
|
||||
)
|
||||
@click.option("--host", type=str, default="0.0.0.0", help="Host address")
|
||||
@click.option("--port", type=int, default=8000, help="Port number")
|
||||
@click.option(
|
||||
"--trial_dir",
|
||||
type=click.Path(file_okay=False, dir_okay=True, exists=True),
|
||||
default=None,
|
||||
help="Path to trial directory.",
|
||||
)
|
||||
@click.option(
|
||||
"--project_dir", help="Path to project directory.", type=str, default=None
|
||||
)
|
||||
@click.option(
|
||||
"--remote", help="Run the API server in remote mode.", type=bool, default=False
|
||||
)
|
||||
def run_api(config_path, host, port, trial_dir, project_dir, remote: bool):
|
||||
if trial_dir is None:
|
||||
runner = ApiRunner.from_yaml(config_path, project_dir=project_dir)
|
||||
else:
|
||||
runner = ApiRunner.from_trial_folder(trial_dir)
|
||||
logger.info(f"Running API server at {host}:{port}...")
|
||||
nest_asyncio.apply()
|
||||
runner.run_api_server(host, port, remote=remote)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--yaml_path", type=click.Path(path_type=Path), help="Path to the YAML file."
|
||||
)
|
||||
@click.option(
|
||||
"--project_dir",
|
||||
type=click.Path(path_type=Path),
|
||||
help="Path to the project directory.",
|
||||
)
|
||||
@click.option(
|
||||
"--trial_path", type=click.Path(path_type=Path), help="Path to the trial directory."
|
||||
)
|
||||
def run_web(
|
||||
yaml_path: Optional[str], project_dir: Optional[str], trial_path: Optional[str]
|
||||
):
|
||||
try:
|
||||
with importlib.resources.path("autorag", "web.py") as web_path:
|
||||
web_py_path = str(web_path)
|
||||
except ImportError:
|
||||
raise ImportError(
|
||||
"Could not locate the web.py file within the autorag package."
|
||||
" Please ensure that autorag is correctly installed."
|
||||
)
|
||||
|
||||
if not yaml_path and not trial_path:
|
||||
raise ValueError("yaml_path or trial_path must be given.")
|
||||
elif yaml_path and trial_path:
|
||||
raise ValueError("yaml_path and trial_path cannot be given at the same time.")
|
||||
elif yaml_path and not project_dir:
|
||||
subprocess.run(
|
||||
["streamlit", "run", web_py_path, "--", "--yaml_path", yaml_path]
|
||||
)
|
||||
elif yaml_path and project_dir:
|
||||
subprocess.run(
|
||||
[
|
||||
"streamlit",
|
||||
"run",
|
||||
web_py_path,
|
||||
"--",
|
||||
"--yaml_path",
|
||||
yaml_path,
|
||||
"--project_dir",
|
||||
project_dir,
|
||||
]
|
||||
)
|
||||
elif trial_path:
|
||||
subprocess.run(
|
||||
["streamlit", "run", web_py_path, "--", "--trial_path", trial_path]
|
||||
)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--trial_dir",
|
||||
type=click.Path(dir_okay=True, file_okay=False, exists=True),
|
||||
required=True,
|
||||
)
|
||||
@click.option(
|
||||
"--port", type=int, default=7690, help="Port number. The default is 7690."
|
||||
)
|
||||
def run_dashboard(trial_dir: str, port: int):
|
||||
dashboard.run(trial_dir, port=port)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option("--trial_path", type=click.Path(), help="Path to the trial directory.")
|
||||
@click.option(
|
||||
"--output_path",
|
||||
type=click.Path(),
|
||||
help="Path to the output directory." " Must be .yaml or .yml file.",
|
||||
)
|
||||
def extract_best_config(trial_path: str, output_path: str):
|
||||
original_extract_best_config(trial_path, output_path)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option("--trial_path", help="Path to trial directory.", type=str)
|
||||
def restart_evaluate(trial_path):
|
||||
if not os.path.exists(trial_path):
|
||||
raise ValueError(f"trial_path {trial_path} does not exist.")
|
||||
project_dir = str(pathlib.PurePath(trial_path).parent)
|
||||
qa_data_path = os.path.join(project_dir, "data", "qa.parquet")
|
||||
corpus_data_path = os.path.join(project_dir, "data", "corpus.parquet")
|
||||
evaluator = Evaluator(qa_data_path, corpus_data_path, project_dir)
|
||||
evaluator.restart_trial(trial_path)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--config",
|
||||
"-c",
|
||||
help="Path to config yaml file. Must be yaml or yml file.",
|
||||
type=str,
|
||||
)
|
||||
@click.option(
|
||||
"--qa_data_path", help="Path to QA dataset. Must be parquet file.", type=str
|
||||
)
|
||||
@click.option(
|
||||
"--corpus_data_path", help="Path to corpus dataset. Must be parquet file.", type=str
|
||||
)
|
||||
def validate(config, qa_data_path, corpus_data_path):
|
||||
if not config.endswith(".yaml") and not config.endswith(".yml"):
|
||||
raise ValueError(f"Config file {config} is not a parquet file.")
|
||||
if not os.path.exists(config):
|
||||
raise ValueError(f"Config file {config} does not exist.")
|
||||
validator = Validator(qa_data_path=qa_data_path, corpus_data_path=corpus_data_path)
|
||||
validator.validate(config)
|
||||
|
||||
|
||||
cli.add_command(evaluate, "evaluate")
|
||||
cli.add_command(run_api, "run_api")
|
||||
cli.add_command(run_web, "run_web")
|
||||
cli.add_command(run_dashboard, "dashboard")
|
||||
cli.add_command(extract_best_config, "extract_best_config")
|
||||
cli.add_command(restart_evaluate, "restart_evaluate")
|
||||
cli.add_command(validate, "validate")
|
||||
|
||||
if __name__ == "__main__":
|
||||
cli()
|
||||
@@ -1,199 +0,0 @@
|
||||
import ast
|
||||
import logging
|
||||
import os
|
||||
from typing import Dict, List
|
||||
|
||||
import matplotlib.pyplot as plt
|
||||
import pandas as pd
|
||||
import panel as pn
|
||||
import seaborn as sns
|
||||
import yaml
|
||||
from bokeh.models import NumberFormatter, BooleanFormatter
|
||||
|
||||
from autorag.utils.util import dict_to_markdown, dict_to_markdown_table
|
||||
|
||||
pn.extension(
|
||||
"terminal",
|
||||
"tabulator",
|
||||
"mathjax",
|
||||
"ipywidgets",
|
||||
console_output="disable",
|
||||
sizing_mode="stretch_width",
|
||||
css_files=[
|
||||
"https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.2/css/all.min.css"
|
||||
],
|
||||
)
|
||||
logger = logging.getLogger("AutoRAG")
|
||||
|
||||
|
||||
def find_node_dir(trial_dir: str) -> List[str]:
|
||||
trial_summary_df = pd.read_csv(os.path.join(trial_dir, "summary.csv"))
|
||||
result_paths = []
|
||||
for idx, row in trial_summary_df.iterrows():
|
||||
node_line_name = row["node_line_name"]
|
||||
node_type = row["node_type"]
|
||||
result_paths.append(os.path.join(trial_dir, node_line_name, node_type))
|
||||
return result_paths
|
||||
|
||||
|
||||
def get_metric_values(node_summary_df: pd.DataFrame) -> Dict:
|
||||
non_metric_column_names = [
|
||||
"filename",
|
||||
"module_name",
|
||||
"module_params",
|
||||
"execution_time",
|
||||
"average_output_token",
|
||||
"is_best",
|
||||
]
|
||||
best_row = node_summary_df.loc[node_summary_df["is_best"]].drop(
|
||||
columns=non_metric_column_names, errors="ignore"
|
||||
)
|
||||
assert len(best_row) == 1, "The best module must be only one."
|
||||
return best_row.iloc[0].to_dict()
|
||||
|
||||
|
||||
def make_trial_summary_md(trial_dir):
|
||||
markdown_text = f"""# Trial Result Summary
|
||||
- Trial Directory : {trial_dir}
|
||||
|
||||
"""
|
||||
node_dirs = find_node_dir(trial_dir)
|
||||
for node_dir in node_dirs:
|
||||
node_summary_filepath = os.path.join(node_dir, "summary.csv")
|
||||
node_type = os.path.basename(node_dir)
|
||||
node_summary_df = pd.read_csv(node_summary_filepath)
|
||||
best_row = node_summary_df.loc[node_summary_df["is_best"]].iloc[0]
|
||||
metric_dict = get_metric_values(node_summary_df)
|
||||
markdown_text += f"""---
|
||||
|
||||
## {node_type} best module
|
||||
|
||||
### Module Name
|
||||
|
||||
{best_row['module_name']}
|
||||
|
||||
### Module Params
|
||||
|
||||
{dict_to_markdown(ast.literal_eval(best_row['module_params']), level=3)}
|
||||
|
||||
### Metric Values
|
||||
|
||||
{dict_to_markdown_table(metric_dict, key_column_name='metric_name', value_column_name='metric_value')}
|
||||
|
||||
"""
|
||||
|
||||
return markdown_text
|
||||
|
||||
|
||||
def node_view(node_dir: str):
|
||||
non_metric_column_names = [
|
||||
"filename",
|
||||
"module_name",
|
||||
"module_params",
|
||||
"execution_time",
|
||||
"average_output_token",
|
||||
"is_best",
|
||||
]
|
||||
summary_df = pd.read_csv(os.path.join(node_dir, "summary.csv"))
|
||||
bokeh_formatters = {
|
||||
"float": NumberFormatter(format="0.000"),
|
||||
"bool": BooleanFormatter(),
|
||||
}
|
||||
first_df = pd.read_parquet(os.path.join(node_dir, "0.parquet"), engine="pyarrow")
|
||||
|
||||
each_module_df_widget = pn.widgets.Tabulator(
|
||||
pd.DataFrame(columns=first_df.columns),
|
||||
name="Module DataFrame",
|
||||
formatters=bokeh_formatters,
|
||||
pagination="local",
|
||||
page_size=20,
|
||||
widths=150,
|
||||
)
|
||||
|
||||
def change_module_widget(event):
|
||||
if event.column == "detail":
|
||||
filename = summary_df["filename"].iloc[event.row]
|
||||
filepath = os.path.join(node_dir, filename)
|
||||
each_module_df = pd.read_parquet(filepath, engine="pyarrow")
|
||||
each_module_df_widget.value = each_module_df
|
||||
|
||||
df_widget = pn.widgets.Tabulator(
|
||||
summary_df,
|
||||
name="Summary DataFrame",
|
||||
formatters=bokeh_formatters,
|
||||
buttons={"detail": '<i class="fa fa-eye"></i>'},
|
||||
widths=150,
|
||||
)
|
||||
df_widget.on_click(change_module_widget)
|
||||
|
||||
try:
|
||||
fig, ax = plt.subplots(figsize=(10, 5))
|
||||
metric_df = summary_df.drop(columns=non_metric_column_names, errors="ignore")
|
||||
sns.stripplot(data=metric_df, ax=ax)
|
||||
strip_plot_pane = pn.pane.Matplotlib(fig, tight=True)
|
||||
|
||||
fig2, ax2 = plt.subplots(figsize=(10, 5))
|
||||
sns.boxplot(data=metric_df, ax=ax2)
|
||||
box_plot_pane = pn.pane.Matplotlib(fig2, tight=True)
|
||||
plot_pane = pn.Row(strip_plot_pane, box_plot_pane)
|
||||
|
||||
layout = pn.Column(
|
||||
"## Summary distribution plot",
|
||||
plot_pane,
|
||||
"## Summary DataFrame",
|
||||
df_widget,
|
||||
"## Module Result DataFrame",
|
||||
each_module_df_widget,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Skipping make boxplot and stripplot with error {e}")
|
||||
layout = pn.Column("## Summary DataFrame", df_widget)
|
||||
layout.servable()
|
||||
return layout
|
||||
|
||||
|
||||
CSS = """
|
||||
div.card-margin:nth-child(1) {
|
||||
max-height: 300px;
|
||||
}
|
||||
div.card-margin:nth-child(2) {
|
||||
max-height: 400px;
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def yaml_to_markdown(yaml_filepath):
|
||||
markdown_content = ""
|
||||
with open(yaml_filepath, "r", encoding="utf-8") as file:
|
||||
try:
|
||||
content = yaml.safe_load(file)
|
||||
markdown_content += f"## {os.path.basename(yaml_filepath)}\n```yaml\n{yaml.safe_dump(content, allow_unicode=True)}\n```\n\n"
|
||||
except yaml.YAMLError as exc:
|
||||
print(f"Error in {yaml_filepath}: {exc}")
|
||||
return markdown_content
|
||||
|
||||
|
||||
def run(trial_dir: str, port: int = 7690):
|
||||
trial_summary_md = make_trial_summary_md(trial_dir=trial_dir)
|
||||
trial_summary_tab = pn.pane.Markdown(trial_summary_md, sizing_mode="stretch_width")
|
||||
|
||||
node_views = [
|
||||
(str(os.path.basename(node_dir)), node_view(node_dir))
|
||||
for node_dir in find_node_dir(trial_dir)
|
||||
]
|
||||
|
||||
yaml_file_markdown = yaml_to_markdown(os.path.join(trial_dir, "config.yaml"))
|
||||
|
||||
yaml_file = pn.pane.Markdown(yaml_file_markdown, sizing_mode="stretch_width")
|
||||
|
||||
tabs = pn.Tabs(
|
||||
("Summary", trial_summary_tab),
|
||||
*node_views,
|
||||
("Used YAML file", yaml_file),
|
||||
dynamic=True,
|
||||
)
|
||||
|
||||
template = pn.template.FastListTemplate(
|
||||
site="AutoRAG", title="Dashboard", main=[tabs], raw_css=[CSS]
|
||||
).servable()
|
||||
template.show(port=port)
|
||||
@@ -1,8 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
export BOKEH_ALLOW_WS_ORIGIN="localhost:7690,172.16.9.208:7690"
|
||||
|
||||
python -m autorag.cli dashboard \
|
||||
--trial_dir ../projects/daesan-dangjin_01/benchmark/1
|
||||
|
||||
echo "📊 AutoRAG 대시보드 http://localhost:7690/..."
|
||||
@@ -1,32 +0,0 @@
|
||||
modules:
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ Token, Sentence ]
|
||||
chunk_size: [ 1024, 512 ]
|
||||
chunk_overlap: 24
|
||||
add_file_name: en
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ SentenceWindow ]
|
||||
window_size: 3
|
||||
add_file_name: en
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ Semantic_llama_index ]
|
||||
embed_model: openai
|
||||
buffer_size: 1
|
||||
breakpoint_percentile_threshold: 95
|
||||
add_file_name: en
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ SemanticDoubleMerging ]
|
||||
add_file_name: en
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ SimpleFile ]
|
||||
add_file_name: en
|
||||
- module_type: langchain_chunk
|
||||
chunk_method: sentencetransformerstoken
|
||||
- module_type: langchain_chunk
|
||||
chunk_method: recursivecharacter
|
||||
separators: [ " ", "\n" ]
|
||||
- module_type: langchain_chunk
|
||||
chunk_method: character
|
||||
separator: ". "
|
||||
- module_type: langchain_chunk
|
||||
chunk_method: Konlpy
|
||||
@@ -1,19 +0,0 @@
|
||||
modules:
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ Token, Sentence ]
|
||||
chunk_size: [ 1024, 512 ]
|
||||
chunk_overlap: 24
|
||||
add_file_name: ko
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ SentenceWindow ]
|
||||
sentence_splitter: kiwi
|
||||
add_file_name: ko
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ Semantic_llama_index ]
|
||||
embed_model: openai
|
||||
add_file_name: ko
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ SimpleFile ]
|
||||
add_file_name: ko
|
||||
- module_type: langchain_chunk
|
||||
chunk_method: KonlpyTextSplitter
|
||||
@@ -1,3 +0,0 @@
|
||||
modules:
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: Token
|
||||
@@ -1,25 +0,0 @@
|
||||
# You can use only one of the following modules at a time.
|
||||
modules:
|
||||
# Use Directory Parse
|
||||
- module_type: langchain_parse
|
||||
file_type: all_files
|
||||
parse_method: directory
|
||||
# Use Unstructured
|
||||
- module_type: langchain_parse
|
||||
file_type: all_files
|
||||
parse_method: unstructured
|
||||
# Use Upsatge Document Parse
|
||||
- module_type: langchain_parse
|
||||
file_type: all_files
|
||||
parse_method: upstagedocumentparse
|
||||
# Use Naver Clova OCR
|
||||
- module_type: clova
|
||||
file_type: all_files
|
||||
table_detection: true
|
||||
# Use Llama Parse
|
||||
- module_type: llamaparse
|
||||
file_type: all_files
|
||||
result_type: markdown
|
||||
language: ko
|
||||
use_vendor_multimodal_model: true
|
||||
vendor_multimodal_model_name: openai-gpt-4o-mini
|
||||
@@ -1,26 +0,0 @@
|
||||
modules:
|
||||
# PDF
|
||||
- module_type: langchain_parse
|
||||
file_type: pdf
|
||||
parse_method: pdfminer
|
||||
# CSV
|
||||
- module_type: langchain_parse
|
||||
file_type: csv
|
||||
parse_method: csv
|
||||
# JSON
|
||||
- module_type: langchain_parse
|
||||
file_type: json
|
||||
parse_method: json
|
||||
jq_schema: .content
|
||||
# Markdown
|
||||
- module_type: langchain_parse
|
||||
file_type: md
|
||||
parse_method: unstructuredmarkdown
|
||||
# HTML
|
||||
- module_type: langchain_parse
|
||||
file_type: html
|
||||
parse_method: bshtml
|
||||
# XML
|
||||
- module_type: langchain_parse
|
||||
file_type: xml
|
||||
parse_method: unstructuredxml
|
||||
@@ -1,12 +0,0 @@
|
||||
modules:
|
||||
- module_type: table_hybrid_parse
|
||||
file_type: pdf
|
||||
text_parse_module: langchain_parse
|
||||
text_params:
|
||||
parse_method: pdfplumber
|
||||
table_parse_module: llamaparse
|
||||
table_params:
|
||||
result_type: markdown
|
||||
language: ko
|
||||
use_vendor_multimodal_model: true
|
||||
vendor_multimodal_model_name: openai-gpt-4o-mini
|
||||
@@ -1,11 +0,0 @@
|
||||
modules:
|
||||
- module_type: llama_parse
|
||||
file_type: all_files
|
||||
result_type: markdown
|
||||
language: ko
|
||||
- module_type: clova
|
||||
file_type: all_files
|
||||
table_detection: true
|
||||
- module_type: langchain_parse
|
||||
file_type: all_files
|
||||
parse_method: upstagedocumentparse
|
||||
@@ -1,8 +0,0 @@
|
||||
modules:
|
||||
- module_type: llamaparse
|
||||
file_type: all_files
|
||||
result_type: markdown
|
||||
language: ko
|
||||
use_vendor_multimodal_model: true
|
||||
vendor_multimodal_model_name: openai-gpt-4o-mini
|
||||
use_own_key: true
|
||||
@@ -1,10 +0,0 @@
|
||||
modules:
|
||||
- module_type: langchain_parse
|
||||
file_type: all_files
|
||||
parse_method: upstagedocumentparse
|
||||
- module_type: llama_parse
|
||||
file_type: all_files
|
||||
result_type: markdown
|
||||
- module_type: clova
|
||||
file_type: all_files
|
||||
table_detection: true
|
||||
@@ -1,4 +0,0 @@
|
||||
modules:
|
||||
- module_type: langchain_parse
|
||||
file_type: pdf
|
||||
parse_method: pdfminer
|
||||
@@ -1,50 +0,0 @@
|
||||
vectordb:
|
||||
- name: mpnet_base_chroma
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_all_mpnet_base_v2
|
||||
collection_name: huggingface_all_mpnet_base_v2
|
||||
path: ${PROJECT_DIR}/data/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
top_k: 20
|
||||
modules:
|
||||
- module_type: bm25
|
||||
- module_type: vectordb
|
||||
vectordb: mpnet_base_chroma
|
||||
- module_type: hybrid_rrf
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
top_k: 3
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: upr
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics: [bleu, meteor, rouge, sem_score]
|
||||
generator_modules:
|
||||
- module_type: vllm
|
||||
llm: mistralai/Mistral-7B-Instruct-v0.2
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics: [bleu, meteor, rouge, sem_score]
|
||||
modules:
|
||||
- module_type: vllm
|
||||
llm: mistralai/Mistral-7B-Instruct-v0.2
|
||||
temperature: [0.1, 0.5, 1.1]
|
||||
@@ -1,101 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 10
|
||||
top_k: 5
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: monot5
|
||||
- module_type: upr
|
||||
- module_type: rankgpt
|
||||
- module_type: colbert_reranker
|
||||
- module_type: sentence_transformer_reranker
|
||||
- module_type: flag_embedding_reranker
|
||||
- module_type: flag_embedding_llm_reranker
|
||||
- module_type: time_reranker
|
||||
- module_type: openvino_reranker
|
||||
- module_type: flashrank_reranker
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- module_type: long_context_reorder
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,154 +0,0 @@
|
||||
vectordb:
|
||||
- name: chroma_bge_m3
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_bge_m3
|
||||
collection_name: openai
|
||||
path: ${PROJECT_DIR}/resources/chroma
|
||||
node_lines:
|
||||
- node_line_name: pre_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: query_expansion
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
retrieval_modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_bge_m3
|
||||
modules:
|
||||
- module_type: pass_query_expansion
|
||||
- module_type: query_decompose
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
- module_type: hyde
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
max_token: 64
|
||||
- module_type: multi_query_expansion
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
temperature: [ 0.2, 1.0 ]
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_bge_m3
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 5
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: monot5
|
||||
- module_type: upr
|
||||
- module_type: rankgpt
|
||||
- module_type: colbert_reranker
|
||||
- module_type: sentence_transformer_reranker
|
||||
- module_type: flag_embedding_reranker
|
||||
- module_type: flag_embedding_llm_reranker
|
||||
- module_type: time_reranker
|
||||
- module_type: openvino_reranker
|
||||
- module_type: flashrank_reranker
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: recency_filter
|
||||
threshold_datetime: 2015-01-01 3:45:07
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- module_type: window_replacement
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval # LLM Judge Metric. Default Model: gpt-4-turbo
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0, 1.5]
|
||||
- module_type: openai_llm
|
||||
llm: gpt-4o-mini
|
||||
temperature: 0.8
|
||||
@@ -1,121 +0,0 @@
|
||||
vectordb:
|
||||
- name: chroma_bge_m3
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_bge_m3
|
||||
collection_name: openai
|
||||
path: ${PROJECT_DIR}/resources/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_bge_m3
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 10
|
||||
top_k: 5
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: monot5
|
||||
- module_type: upr
|
||||
- module_type: rankgpt
|
||||
- module_type: colbert_reranker
|
||||
- module_type: sentence_transformer_reranker
|
||||
- module_type: flag_embedding_reranker
|
||||
- module_type: flag_embedding_llm_reranker
|
||||
- module_type: time_reranker
|
||||
- module_type: openvino_reranker
|
||||
- module_type: flashrank_reranker
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- module_type: long_context_reorder
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,105 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 10
|
||||
top_k: 5
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: monot5
|
||||
- module_type: upr
|
||||
- module_type: cohere_reranker
|
||||
- module_type: rankgpt
|
||||
- module_type: jina_reranker
|
||||
- module_type: colbert_reranker
|
||||
- module_type: sentence_transformer_reranker
|
||||
- module_type: flag_embedding_reranker
|
||||
- module_type: flag_embedding_llm_reranker
|
||||
- module_type: time_reranker
|
||||
- module_type: openvino_reranker
|
||||
- module_type: voyageai_reranker
|
||||
- module_type: mixedbreadai_reranker
|
||||
- module_type: flashrank_reranker
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- module_type: long_context_reorder
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,151 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: pre_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: query_expansion
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
retrieval_modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
modules:
|
||||
- module_type: pass_query_expansion
|
||||
- module_type: query_decompose
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
- module_type: hyde
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
max_token: 64
|
||||
- module_type: multi_query_expansion
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
temperature: [ 0.2, 1.0 ]
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 5
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: monot5
|
||||
- module_type: upr
|
||||
- module_type: cohere_reranker
|
||||
- module_type: rankgpt
|
||||
- module_type: jina_reranker
|
||||
- module_type: colbert_reranker
|
||||
- module_type: sentence_transformer_reranker
|
||||
- module_type: flag_embedding_reranker
|
||||
- module_type: flag_embedding_llm_reranker
|
||||
- module_type: time_reranker
|
||||
- module_type: openvino_reranker
|
||||
- module_type: voyageai_reranker
|
||||
- module_type: mixedbreadai_reranker
|
||||
- module_type: flashrank_reranker
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: recency_filter
|
||||
threshold_datetime: 2015-01-01 3:45:07
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- module_type: window_replacement
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval # LLM Judge Metric. Default Model: gpt-4-turbo
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0, 1.5]
|
||||
- module_type: openai_llm
|
||||
llm: gpt-4o-mini
|
||||
temperature: 0.8
|
||||
@@ -1,118 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 10
|
||||
top_k: 5
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: monot5
|
||||
- module_type: upr
|
||||
- module_type: cohere_reranker
|
||||
- module_type: rankgpt
|
||||
- module_type: jina_reranker
|
||||
- module_type: colbert_reranker
|
||||
- module_type: sentence_transformer_reranker
|
||||
- module_type: flag_embedding_reranker
|
||||
- module_type: flag_embedding_llm_reranker
|
||||
- module_type: time_reranker
|
||||
- module_type: openvino_reranker
|
||||
- module_type: voyageai_reranker
|
||||
- module_type: mixedbreadai_reranker
|
||||
- module_type: flashrank_reranker
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- module_type: long_context_reorder
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,83 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- module_type: long_context_reorder
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,129 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: pre_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: query_expansion
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
retrieval_modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
modules:
|
||||
- module_type: pass_query_expansion
|
||||
- module_type: query_decompose
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
- module_type: hyde
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
max_token: 64
|
||||
- module_type: multi_query_expansion
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
temperature: [ 0.2, 1.0 ]
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
- module_type: vectordb
|
||||
embedding_model: openai
|
||||
embedding_batch: 256
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: recency_filter
|
||||
threshold_datetime: 2015-01-01 3:45:07
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- module_type: window_replacement
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval # LLM Judge Metric. Default Model: gpt-4-turbo
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0, 1.5]
|
||||
- module_type: openai_llm
|
||||
llm: gpt-4o-mini
|
||||
temperature: 0.8
|
||||
@@ -1,95 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- module_type: long_context_reorder
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,33 +0,0 @@
|
||||
vectordb:
|
||||
- name: mpnet_base_chroma
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_all_mpnet_base_v2
|
||||
collection_name: huggingface_all_mpnet_base_v2
|
||||
path: ${PROJECT_DIR}/data/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
top_k: 3
|
||||
modules:
|
||||
- module_type: vectordb
|
||||
vectordb: mpnet_base_chroma
|
||||
- node_line_name: post_retrieve_node_line
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics: [ meteor, rouge, bert_score ]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics: [ bleu, rouge, bert_score ]
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: bedrock
|
||||
model: amazon.titan-text-express-v1
|
||||
profile_name: your_profile_name # Plz replace this with your profile name
|
||||
@@ -1,31 +0,0 @@
|
||||
vectordb:
|
||||
- name: baai_chroma
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_baai_bge_small
|
||||
collection_name: huggingface_baai_bge_small
|
||||
path: ${PROJECT_DIR}/data/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
top_k: 3
|
||||
modules:
|
||||
- module_type: vectordb
|
||||
vectordb: baai_chroma
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics: [ meteor, rouge, bert_score ]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics: [ bleu, rouge, bert_score ]
|
||||
modules:
|
||||
- module_type: vllm
|
||||
llm: mistralai/Mistral-7B-Instruct-v0.2
|
||||
@@ -1,34 +0,0 @@
|
||||
vectordb:
|
||||
- name: mpnet_base_chroma
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_all_mpnet_base_v2
|
||||
collection_name: huggingface_all_mpnet_base_v2
|
||||
path: ${PROJECT_DIR}/data/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
top_k: 3
|
||||
modules:
|
||||
- module_type: vectordb
|
||||
vectordb: mpnet_base_chroma
|
||||
- node_line_name: post_retrieve_node_line
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics: [ meteor, rouge, bert_score ]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics: [ bleu, rouge, bert_score ]
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: ollama
|
||||
model: llama3
|
||||
batch: 1
|
||||
request_timeout: 100 # You can increase this value if your model is big (slow)
|
||||
@@ -1,25 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
top_k: 3
|
||||
modules:
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics: [bleu, meteor, rouge]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics: [bleu, rouge]
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
@@ -1,47 +0,0 @@
|
||||
vectordb:
|
||||
- name: default
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: openai
|
||||
collection_name: openai
|
||||
path: ${PROJECT_DIR}/data/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
modules:
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
top_k: 3
|
||||
strategy:
|
||||
metrics:
|
||||
- retrieval_f1
|
||||
- retrieval_recall
|
||||
- retrieval_precision
|
||||
- node_line_name: post_retrieve_node_line
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
|
||||
strategy:
|
||||
generator_modules:
|
||||
- batch: 2
|
||||
llm: openai
|
||||
module_type: llama_index_llm
|
||||
metrics:
|
||||
- bleu
|
||||
- meteor
|
||||
- rouge
|
||||
- node_type: generator
|
||||
modules:
|
||||
- batch: 2
|
||||
llm: openai
|
||||
model: gpt-3.5-turbo-16k
|
||||
module_type: llama_index_llm
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
@@ -1,159 +0,0 @@
|
||||
vectordb:
|
||||
- name: chroma_large
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: openai_embed_3_large
|
||||
collection_name: openai_embed_3_large
|
||||
path: ${PROJECT_DIR}/resources/chroma
|
||||
node_lines:
|
||||
- node_line_name: pre_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: query_expansion
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
retrieval_modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, ko_kiwi, space, gpt2, ko_okt, ko_kkma, sudachipy ]
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_large
|
||||
modules:
|
||||
- module_type: pass_query_expansion
|
||||
- module_type: query_decompose
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-3.5-turbo-16k, gpt-3.5-turbo-1106 ]
|
||||
- module_type: hyde
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-3.5-turbo-16k ]
|
||||
max_token: 64
|
||||
- module_type: multi_query_expansion
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
temperature: [ 0.2, 1.0 ]
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_large
|
||||
embedding_batch: 256
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 5
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: monot5
|
||||
- module_type: upr
|
||||
- module_type: cohere_reranker
|
||||
- module_type: rankgpt
|
||||
- module_type: jina_reranker
|
||||
- module_type: colbert_reranker
|
||||
- module_type: sentence_transformer_reranker
|
||||
- module_type: flag_embedding_reranker
|
||||
- module_type: flag_embedding_llm_reranker
|
||||
- module_type: time_reranker
|
||||
- module_type: openvino_reranker
|
||||
- module_type: voyageai_reranker
|
||||
- module_type: mixedbreadai_reranker
|
||||
- module_type: flashrank_reranker
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: recency_filter
|
||||
threshold_datetime: 2015-01-01 3:45:07
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-3.5-turbo-16k
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-3.5-turbo-16k
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-3.5-turbo-16k, gpt-3.5-turbo-1106]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- module_type: window_replacement
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-3.5-turbo-16k, gpt-3.5-turbo-1106]
|
||||
temperature: [0.5, 1.0, 1.5]
|
||||
- module_type: openai_llm
|
||||
llm: gpt-3.5-turbo
|
||||
temperature: 0.8
|
||||
@@ -1,93 +0,0 @@
|
||||
vectordb:
|
||||
- name: chroma_bge_m3
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_bge_m3
|
||||
collection_name: openai
|
||||
path: ${PROJECT_DIR}/resources/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_bge_m3
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
modules:
|
||||
- module_type: koreranker
|
||||
- module_type: flag_embedding_llm_reranker # Requires enough GPU resources
|
||||
- module_type: pass_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_recall, retrieval_precision, retrieval_map ]
|
||||
top_k: 3
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,157 +0,0 @@
|
||||
vectordb:
|
||||
- name: chroma_bge_m3
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_bge_m3
|
||||
collection_name: openai
|
||||
path: ${PROJECT_DIR}/resources/chroma
|
||||
node_lines:
|
||||
- node_line_name: pre_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: query_expansion
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
retrieval_modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_bge_m3
|
||||
modules:
|
||||
- module_type: pass_query_expansion
|
||||
- module_type: hyde
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
max_token: 64
|
||||
prompt: "질문에 답하기 위한 단락을 작성해 주세요."
|
||||
- module_type: multi_query_expansion
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
temperature: [ 0.2, 1.0 ]
|
||||
prompt: |
|
||||
당신은 인공지능 언어 모델 어시스턴트입니다.
|
||||
주어진 사용자 질문을 이용해 세 가지 버전의 새 질문을 생성하여 벡터 데이터베이스에서 관련 문서를 검색하는 것이 과제입니다.
|
||||
주어진 질문에 대한 다양한 관점을 생성함으로써 사용자가 거리 기반 유사도 검색의 한계를 극복할 수 있도록 돕는 것이 목표입니다.
|
||||
다음과 같은 대체 질문을 줄 바꿈으로 구분하여 제공하십시오.
|
||||
원래 질문: {query}
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_bge_m3
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
modules:
|
||||
- module_type: koreranker
|
||||
- module_type: flag_embedding_llm_reranker # Requires enough GPU resources
|
||||
- module_type: pass_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_recall, retrieval_precision, retrieval_map ]
|
||||
top_k: 3
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
여러 문맥 정보는 다음과 같습니다.\n
|
||||
---------------------\n
|
||||
{context_str}\n
|
||||
---------------------\n
|
||||
사전 지식이 아닌 여러 정보가 주어졌습니다,
|
||||
질문에 대답하세요.\n
|
||||
질문: {query_str}\n
|
||||
답변:
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
원래 질문은 다음과 같습니다: {query_str}
|
||||
기존 답변은 다음과 같습니다: {existing_answer}
|
||||
아래에서 기존 답변을 정제할 수 있는 기회가 있습니다.
|
||||
(필요한 경우에만) 아래에 몇 가지 맥락을 추가하여 기존 답변을 정제할 수 있습니다.
|
||||
------------
|
||||
{context_msg}
|
||||
------------
|
||||
새로운 문맥이 주어지면 기존 답변을 수정하여 질문에 대한 답변을 정제합니다.
|
||||
맥락이 쓸모 없다면, 기존 답변을 그대로 답변하세요.
|
||||
정제된 답변:
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
- metric_name: g_eval # LLM Judge Metric. Default Model: gpt-4-turbo
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,126 +0,0 @@
|
||||
vectordb:
|
||||
- name: chroma_bge_m3
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_bge_m3
|
||||
collection_name: openai
|
||||
path: ${PROJECT_DIR}/resources/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_bge_m3
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
modules:
|
||||
- module_type: koreranker
|
||||
- module_type: flag_embedding_llm_reranker # Requires enough GPU resources
|
||||
- module_type: pass_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_recall, retrieval_precision, retrieval_map ]
|
||||
top_k: 3
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
여러 문맥 정보는 다음과 같습니다.\n
|
||||
---------------------\n
|
||||
{context_str}\n
|
||||
---------------------\n
|
||||
사전 지식이 아닌 여러 정보가 주어졌습니다,
|
||||
질문에 대답하세요.\n
|
||||
질문: {query_str}\n
|
||||
답변:
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
원래 질문은 다음과 같습니다: {query_str}
|
||||
기존 답변은 다음과 같습니다: {existing_answer}
|
||||
아래에서 기존 답변을 정제할 수 있는 기회가 있습니다.
|
||||
(필요한 경우에만) 아래에 몇 가지 맥락을 추가하여 기존 답변을 정제할 수 있습니다.
|
||||
------------
|
||||
{context_msg}
|
||||
------------
|
||||
새로운 문맥이 주어지면 기존 답변을 수정하여 질문에 대한 답변을 정제합니다.
|
||||
맥락이 쓸모 없다면, 기존 답변을 그대로 답변하세요.
|
||||
정제된 답변:
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,87 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
modules:
|
||||
- module_type: koreranker
|
||||
- module_type: flag_embedding_llm_reranker # Requires enough GPU resources
|
||||
- module_type: cohere_reranker # Set Environment Variable: COHERE_API_KEY
|
||||
- module_type: pass_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_recall, retrieval_precision, retrieval_map ]
|
||||
top_k: 3
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,151 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: pre_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: query_expansion
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
retrieval_modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
modules:
|
||||
- module_type: pass_query_expansion
|
||||
- module_type: hyde
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai #
|
||||
model: [ gpt-4o-mini ] #
|
||||
max_token: 64
|
||||
prompt: "질문에 답하기 위한 단락을 작성해 주세요."
|
||||
- module_type: multi_query_expansion
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
temperature: [ 0.2, 1.0 ]
|
||||
prompt: |
|
||||
당신은 인공지능 언어 모델 어시스턴트입니다.
|
||||
주어진 사용자 질문을 이용해 세 가지 버전의 새 질문을 생성하여 벡터 데이터베이스에서 관련 문서를 검색하는 것이 과제입니다.
|
||||
주어진 질문에 대한 다양한 관점을 생성함으로써 사용자가 거리 기반 유사도 검색의 한계를 극복할 수 있도록 돕는 것이 목표입니다.
|
||||
다음과 같은 대체 질문을 줄 바꿈으로 구분하여 제공하십시오.
|
||||
원래 질문: {query}
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ] # ko_kiwi, ko_okt
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
modules:
|
||||
- module_type: koreranker
|
||||
- module_type: flag_embedding_llm_reranker # Requires enough GPU resources
|
||||
- module_type: cohere_reranker # Set Environment Variable: COHERE_API_KEY
|
||||
- module_type: pass_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_recall, retrieval_precision, retrieval_map ]
|
||||
top_k: 3
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
여러 문맥 정보는 다음과 같습니다.\n
|
||||
---------------------\n
|
||||
{context_str}\n
|
||||
---------------------\n
|
||||
사전 지식이 아닌 여러 정보가 주어졌습니다,
|
||||
질문에 대답하세요.\n
|
||||
질문: {query_str}\n
|
||||
답변:
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
원래 질문은 다음과 같습니다: {query_str}
|
||||
기존 답변은 다음과 같습니다: {existing_answer}
|
||||
아래에서 기존 답변을 정제할 수 있는 기회가 있습니다.
|
||||
(필요한 경우에만) 아래에 몇 가지 맥락을 추가하여 기존 답변을 정제할 수 있습니다.
|
||||
------------
|
||||
{context_msg}
|
||||
------------
|
||||
새로운 문맥이 주어지면 기존 답변을 수정하여 질문에 대한 답변을 정제합니다.
|
||||
맥락이 쓸모 없다면, 기존 답변을 그대로 답변하세요.
|
||||
정제된 답변:
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
- metric_name: g_eval # LLM Judge Metric. Default Model: gpt-4-turbo
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,120 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
modules:
|
||||
- module_type: koreranker
|
||||
- module_type: flag_embedding_llm_reranker # Requires enough GPU resources
|
||||
- module_type: cohere_reranker # Set Environment Variable: COHERE_API_KEY
|
||||
- module_type: pass_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_recall, retrieval_precision, retrieval_map ]
|
||||
top_k: 3
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
여러 문맥 정보는 다음과 같습니다.\n
|
||||
---------------------\n
|
||||
{context_str}\n
|
||||
---------------------\n
|
||||
사전 지식이 아닌 여러 정보가 주어졌습니다,
|
||||
질문에 대답하세요.\n
|
||||
질문: {query_str}\n
|
||||
답변:
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
원래 질문은 다음과 같습니다: {query_str}
|
||||
기존 답변은 다음과 같습니다: {existing_answer}
|
||||
아래에서 기존 답변을 정제할 수 있는 기회가 있습니다.
|
||||
(필요한 경우에만) 아래에 몇 가지 맥락을 추가하여 기존 답변을 정제할 수 있습니다.
|
||||
------------
|
||||
{context_msg}
|
||||
------------
|
||||
새로운 문맥이 주어지면 기존 답변을 수정하여 질문에 대한 답변을 정제합니다.
|
||||
맥락이 쓸모 없다면, 기존 답변을 그대로 답변하세요.
|
||||
정제된 답변:
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,78 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,142 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: pre_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: query_expansion
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
retrieval_modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
modules:
|
||||
- module_type: pass_query_expansion
|
||||
- module_type: hyde
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
max_token: 64
|
||||
prompt: "질문에 답하기 위한 단락을 작성해 주세요."
|
||||
- module_type: multi_query_expansion
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
temperature: [ 0.2, 1.0 ]
|
||||
prompt: |
|
||||
당신은 인공지능 언어 모델 어시스턴트입니다.
|
||||
주어진 사용자 질문을 이용해 세 가지 버전의 새 질문을 생성하여 벡터 데이터베이스에서 관련 문서를 검색하는 것이 과제입니다.
|
||||
주어진 질문에 대한 다양한 관점을 생성함으로써 사용자가 거리 기반 유사도 검색의 한계를 극복할 수 있도록 돕는 것이 목표입니다.
|
||||
다음과 같은 대체 질문을 줄 바꿈으로 구분하여 제공하십시오.
|
||||
원래 질문: {query}
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
여러 문맥 정보는 다음과 같습니다.\n
|
||||
---------------------\n
|
||||
{context_str}\n
|
||||
---------------------\n
|
||||
사전 지식이 아닌 여러 정보가 주어졌습니다,
|
||||
질문에 대답하세요.\n
|
||||
질문: {query_str}\n
|
||||
답변:
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
원래 질문은 다음과 같습니다: {query_str}
|
||||
기존 답변은 다음과 같습니다: {existing_answer}
|
||||
아래에서 기존 답변을 정제할 수 있는 기회가 있습니다.
|
||||
(필요한 경우에만) 아래에 몇 가지 맥락을 추가하여 기존 답변을 정제할 수 있습니다.
|
||||
------------
|
||||
{context_msg}
|
||||
------------
|
||||
새로운 문맥이 주어지면 기존 답변을 수정하여 질문에 대한 답변을 정제합니다.
|
||||
맥락이 쓸모 없다면, 기존 답변을 그대로 답변하세요.
|
||||
정제된 답변:
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
- metric_name: g_eval # LLM Judge Metric. Default Model: gpt-4-turbo
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,111 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
여러 문맥 정보는 다음과 같습니다.\n
|
||||
---------------------\n
|
||||
{context_str}\n
|
||||
---------------------\n
|
||||
사전 지식이 아닌 여러 정보가 주어졌습니다,
|
||||
질문에 대답하세요.\n
|
||||
질문: {query_str}\n
|
||||
답변:
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
원래 질문은 다음과 같습니다: {query_str}
|
||||
기존 답변은 다음과 같습니다: {existing_answer}
|
||||
아래에서 기존 답변을 정제할 수 있는 기회가 있습니다.
|
||||
(필요한 경우에만) 아래에 몇 가지 맥락을 추가하여 기존 답변을 정제할 수 있습니다.
|
||||
------------
|
||||
{context_msg}
|
||||
------------
|
||||
새로운 문맥이 주어지면 기존 답변을 수정하여 질문에 대한 답변을 정제합니다.
|
||||
맥락이 쓸모 없다면, 기존 답변을 그대로 답변하세요.
|
||||
정제된 답변:
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,30 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
top_k: 3
|
||||
modules:
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics: [ bleu, meteor, rouge ]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: "주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
@@ -1,25 +0,0 @@
|
||||
# sample_dataset handling
|
||||
|
||||
The sample_dataset folder does not includes a `qa.parquet`, `corpus.parquet` file that is significantly large and cannot be uploaded directly to Git due to size limitations.
|
||||
|
||||
To prepare and use datasets available in the sample_dataset folder, specifically `triviaqa`, `hotpotqa`, `msmarco` and `eli5`, you can follow the outlined methods below.
|
||||
|
||||
## Usage
|
||||
|
||||
The example provided uses `triviaqa`, but the same approach applies to `msmarco`, `eli5` and `hotpotqa`.
|
||||
|
||||
### 1. Run with a specified save path
|
||||
To execute the Python script from the terminal and save the dataset to a specified path, use the command:
|
||||
|
||||
```bash
|
||||
python ./sample_dataset/triviaqa/load_triviaqa_dataset.py --save_path /path/to/save/dataset
|
||||
```
|
||||
This runs the `load_triviaqa_dataset.py` script located in the `./sample_dataset/triviaqa/` directory,
|
||||
using the `--save_path` argument to specify the dataset's save location.
|
||||
|
||||
### 2. Run without specifying a save path
|
||||
If you run the script without the `--save_path` argument, the dataset will be saved to a default location, which is the directory containing the `load_triviaqa_dataset.py` file, essentially `./sample_dataset/triviaqa/`:
|
||||
```bash
|
||||
python ./sample_dataset/triviaqa/load_triviaqa_dataset.py
|
||||
```
|
||||
This behavior allows for a straightforward execution without needing to specify a path, making it convenient for quick tests or when working directly within the target directory.
|
||||
@@ -1,35 +0,0 @@
|
||||
import os
|
||||
import pathlib
|
||||
|
||||
import click
|
||||
from datasets import load_dataset
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--save_path",
|
||||
type=str,
|
||||
default=pathlib.PurePath(__file__).parent,
|
||||
help="Path to save sample eli5 dataset.",
|
||||
)
|
||||
def load_eli5_dataset(save_path):
|
||||
# set file path
|
||||
file_path = "MarkrAI/eli5_sample_autorag"
|
||||
|
||||
# load dataset
|
||||
corpus_dataset = load_dataset(file_path, "corpus")["train"].to_pandas()
|
||||
qa_train_dataset = load_dataset(file_path, "qa")["train"].to_pandas()
|
||||
qa_test_dataset = load_dataset(file_path, "qa")["test"].to_pandas()
|
||||
|
||||
# save data
|
||||
if os.path.exists(os.path.join(save_path, "corpus.parquet")) is True:
|
||||
raise ValueError("corpus.parquet already exists")
|
||||
if os.path.exists(os.path.join(save_path, "qa.parquet")) is True:
|
||||
raise ValueError("qa.parquet already exists")
|
||||
corpus_dataset.to_parquet(os.path.join(save_path, "corpus.parquet"))
|
||||
qa_train_dataset.to_parquet(os.path.join(save_path, "qa_train.parquet"))
|
||||
qa_test_dataset.to_parquet(os.path.join(save_path, "qa_test.parquet"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
load_eli5_dataset()
|
||||
@@ -1,35 +0,0 @@
|
||||
import os
|
||||
import pathlib
|
||||
|
||||
import click
|
||||
from datasets import load_dataset
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--save_path",
|
||||
type=str,
|
||||
default=pathlib.PurePath(__file__).parent,
|
||||
help="Path to save sample hotpotqa dataset.",
|
||||
)
|
||||
def load_hotpotqa_dataset(save_path):
|
||||
# set file path
|
||||
file_path = "gnekt/hotpotqa_small_sample_autorag"
|
||||
|
||||
# load dataset
|
||||
corpus_dataset = load_dataset(file_path, "corpus")["train"].to_pandas()
|
||||
qa_validation_dataset = load_dataset(file_path, "qa")["validation"].to_pandas()
|
||||
|
||||
# save corpus data
|
||||
if os.path.exists(os.path.join(save_path, "corpus.parquet")) is True:
|
||||
raise ValueError("corpus.parquet already exists")
|
||||
if os.path.exists(os.path.join(save_path, "qa.parquet")) is True:
|
||||
raise ValueError("qa.parquet already exists")
|
||||
corpus_dataset.to_parquet(os.path.join(save_path, "corpus.parquet"), index=False)
|
||||
qa_validation_dataset.to_parquet(
|
||||
os.path.join(save_path, "qa_validation.parquet"), index=False
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
load_hotpotqa_dataset()
|
||||
@@ -1,37 +0,0 @@
|
||||
import os
|
||||
import pathlib
|
||||
|
||||
import click
|
||||
from datasets import load_dataset
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--save_path",
|
||||
type=str,
|
||||
default=pathlib.PurePath(__file__).parent,
|
||||
help="Path to save sample msmarco dataset.",
|
||||
)
|
||||
def load_msmarco_dataset(save_path):
|
||||
# set file path
|
||||
file_path = "MarkrAI/msmarco_sample_autorag"
|
||||
|
||||
# load dataset
|
||||
corpus_dataset = load_dataset(file_path, "corpus")["train"].to_pandas()
|
||||
qa_train_dataset = load_dataset(file_path, "qa")["train"].to_pandas()
|
||||
qa_test_dataset = load_dataset(file_path, "qa")["test"].to_pandas()
|
||||
|
||||
# save corpus data
|
||||
if os.path.exists(os.path.join(save_path, "corpus.parquet")) is True:
|
||||
raise ValueError("corpus.parquet already exists")
|
||||
if os.path.exists(os.path.join(save_path, "qa.parquet")) is True:
|
||||
raise ValueError("qa.parquet already exists")
|
||||
corpus_dataset.to_parquet(os.path.join(save_path, "corpus.parquet"), index=False)
|
||||
qa_train_dataset.to_parquet(
|
||||
os.path.join(save_path, "qa_train.parquet"), index=False
|
||||
)
|
||||
qa_test_dataset.to_parquet(os.path.join(save_path, "qa_test.parquet"), index=False)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
load_msmarco_dataset()
|
||||
@@ -1,37 +0,0 @@
|
||||
import os
|
||||
import pathlib
|
||||
|
||||
import click
|
||||
from datasets import load_dataset
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--save_path",
|
||||
type=str,
|
||||
default=pathlib.PurePath(__file__).parent,
|
||||
help="Path to save sample triviaqa dataset.",
|
||||
)
|
||||
def load_triviaqa_dataset(save_path):
|
||||
# set file path
|
||||
file_path = "MarkrAI/triviaqa_sample_autorag"
|
||||
|
||||
# load dataset
|
||||
corpus_dataset = load_dataset(file_path, "corpus")["train"].to_pandas()
|
||||
qa_train_dataset = load_dataset(file_path, "qa")["train"].to_pandas()
|
||||
qa_test_dataset = load_dataset(file_path, "qa")["test"].to_pandas()
|
||||
|
||||
# save corpus data
|
||||
if os.path.exists(os.path.join(save_path, "corpus.parquet")) is True:
|
||||
raise ValueError("corpus.parquet already exists")
|
||||
if os.path.exists(os.path.join(save_path, "qa.parquet")) is True:
|
||||
raise ValueError("qa.parquet already exists")
|
||||
corpus_dataset.to_parquet(os.path.join(save_path, "corpus.parquet"), index=False)
|
||||
qa_train_dataset.to_parquet(
|
||||
os.path.join(save_path, "qa_train.parquet"), index=False
|
||||
)
|
||||
qa_test_dataset.to_parquet(os.path.join(save_path, "qa_test.parquet"), index=False)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
load_triviaqa_dataset()
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loaded 100 of 228 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user