Fix Dockerfile build issue
This commit is contained in:
@@ -1,2 +1 @@
|
||||
OPENAI_API_KEY=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
|
||||
LLAMA_CLOUD_API_KEY=llx-MkHkuDxnSxXEHvsIPAtjEZl4iSB8pHS1mgYDVZQlA690LUub
|
||||
25
Dockerfile
25
Dockerfile
@@ -7,8 +7,9 @@ ENV PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
PIP_NO_CACHE_DIR=1
|
||||
|
||||
# Copy only requirements files first to leverage Docker cache
|
||||
COPY pyproject.toml ./
|
||||
# Copy only necessary files
|
||||
COPY pyproject.toml ./
|
||||
COPY requirements.txt ./
|
||||
|
||||
# Install system and Python dependencies in a single layer
|
||||
RUN apt-get update && \
|
||||
@@ -19,10 +20,20 @@ RUN apt-get update && \
|
||||
pip install --no-cache-dir --upgrade pip setuptools setuptools-scm && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy project files
|
||||
COPY . .
|
||||
|
||||
# Install base project
|
||||
RUN pip install --no-cache-dir -e .
|
||||
RUN pip install -r requirements.txt
|
||||
|
||||
CMD ["bash"]
|
||||
# Copy project files
|
||||
COPY autorag /usr/src/app/autorag
|
||||
COPY main.py /usr/src/app/main.py
|
||||
COPY making.sh /usr/src/app/making.sh
|
||||
COPY entrypoint.sh /usr/src/app/entrypoint.sh
|
||||
|
||||
# Set permissions for entrypoint
|
||||
RUN chmod +x /usr/src/app/entrypoint.sh
|
||||
|
||||
# Use entrypoint.sh as the container entrypoint
|
||||
ENTRYPOINT ["/usr/src/app/entrypoint.sh"]
|
||||
|
||||
# Set a default command (optional, can be overridden)
|
||||
CMD ["bash"]
|
||||
136
README.md
136
README.md
@@ -1,51 +1,68 @@
|
||||
# AutoRAG Evaluation
|
||||
# 📊 AutoRAG Evaluation
|
||||
|
||||
이 문서는 AutoRAG을 활용하여 RAG 파이프라인을 설정하고 평가하는 과정에 대한 안내입니다.
|
||||
AutoRAG Evaluation은 RAG(Relevance-Augmented Generation) 파이프라인을 평가하기 위한 도구입니다.
|
||||
이 문서는 데이터 생성부터 RAG 평가 및 대시보드 활용까지의 전체 과정에 대한 안내를 제공합니다.
|
||||
|
||||
---
|
||||
|
||||
## 📌 환경 세팅
|
||||
## **🚀 1. 환경 설정**
|
||||
|
||||
1. `.env` 파일을 설정합니다. (`.env.sample` 참고)
|
||||
2. Docker 이미지를 빌드합니다.
|
||||
### **1️⃣ `.env` 파일 설정**
|
||||
|
||||
`.env.sample` 파일을 참고하여 `.env` 파일을 생성하고 필요한 환경 변수를 설정합니다.
|
||||
|
||||
### **2️⃣ Docker 이미지 빌드**
|
||||
|
||||
```bash
|
||||
docker build -t autorag-base .
|
||||
docker build -t autorag-base .
|
||||
```
|
||||
|
||||
3. Docker Compose를 실행하여 서비스를 시작합니다.
|
||||
### **3️⃣ Docker Compose 백그라운드 실행**
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
4. Hugginface embedding, Ollama LLM을 위한 추가 모듈 설치를 시작합니다.
|
||||
### **4️⃣ 실행 모드 (Docker)**
|
||||
|
||||
AutoRAG Evaluation은 실행 모드에 따라 다르게 동작합니다.
|
||||
Docker 실행 시 아래 명령어를 사용하여 모드를 선택하세요.
|
||||
|
||||
```bash
|
||||
pip install -r requirements_custom.txt
|
||||
docker run autorag-base <mode>
|
||||
```
|
||||
|
||||
| 실행 모드 | 설명 |
|
||||
| ---------- | -------------------------------- |
|
||||
| `data-gen` | 데이터 생성 (`making.sh` 실행) |
|
||||
| `evaluate` | RAG 평가 (`main.py` 실행) |
|
||||
| `bash` | 컨테이너 내부 접근 (`/bin/bash`) |
|
||||
|
||||
예시:
|
||||
|
||||
```bash
|
||||
docker run autorag-base data-gen
|
||||
docker run autorag-base evaluate
|
||||
docker run autorag-base bash
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📂 데이터 생성
|
||||
## **📂 2. 데이터 생성**
|
||||
|
||||
RAG 평가를 위해 **QA 데이터 세트**와 **코퍼스 데이터 세트**가 필요합니다.
|
||||
|
||||
### 1️⃣ 프로젝트 폴더 생성
|
||||
### **1️⃣ 프로젝트 폴더 생성**
|
||||
|
||||
```bash
|
||||
cd projects
|
||||
mkdir -p "project_name"
|
||||
cd "project_name"
|
||||
mkdir -p raw_data config
|
||||
mkdir -p example_project/{raw_data,config}
|
||||
```
|
||||
|
||||
- **`raw_data/`**: 분석할 원본 데이터를 저장 (`.pdf` 등)
|
||||
- **`config/`**: 파싱(`parse.yaml`), 청킹(`chunk.yaml`) 설정 파일을 저장
|
||||
- **`raw_data/`**: 원본 데이터 저장 (`.pdf`, `.txt` 등)
|
||||
- **`config/`**: 설정 파일 저장 (`parse.yaml`, `chunk.yaml` 등)
|
||||
|
||||
### 2️⃣ 파싱 설정 (`parse.yaml`)
|
||||
|
||||
파싱 모듈을 설정합니다.
|
||||
### **2️⃣ 파싱 설정 (`parse.yaml`)**
|
||||
|
||||
```yaml
|
||||
modules:
|
||||
@@ -53,11 +70,7 @@ modules:
|
||||
parse_method: pdfminer
|
||||
```
|
||||
|
||||
여러 개의 파싱 모듈을 동시에 사용할 수도 있습니다.
|
||||
|
||||
### 3️⃣ 청킹 설정 (`chunk.yaml`)
|
||||
|
||||
청킹 모듈을 설정합니다.
|
||||
### **3️⃣ 청킹 설정 (`chunk.yaml`)**
|
||||
|
||||
```yaml
|
||||
modules:
|
||||
@@ -68,25 +81,21 @@ modules:
|
||||
add_file_name: en
|
||||
```
|
||||
|
||||
여러 개의 청킹 모듈을 사용할 경우, QA 데이터와 매핑해야 합니다.
|
||||
|
||||
### 4️⃣ QA 데이터 생성
|
||||
|
||||
`raw_data/`에 저장된 파일을 바탕으로 **파싱 → 청킹 → QA 데이터 생성**을 진행합니다.
|
||||
QA 데이터는 `GPT-4o-mini` 모델을 사용하여 **20건**을 생성합니다.
|
||||
### **4️⃣ QA 데이터 생성**
|
||||
|
||||
```bash
|
||||
cd autorag-workspace
|
||||
sh making.sh
|
||||
bash making.sh
|
||||
```
|
||||
|
||||
> **참고:** `GPT-4o-mini` 모델을 사용하여 자동으로 QA 데이터를 생성합니다. (기본 20건)
|
||||
|
||||
---
|
||||
|
||||
## 🔍 RAG Pipeline 평가
|
||||
## **🔍 3. RAG Pipeline 평가**
|
||||
|
||||
### 1️⃣ Ollama 모델 다운로드
|
||||
### **1️⃣ Ollama 모델 다운로드**
|
||||
|
||||
WSL(Windows Subsystem for Linux)에서 실행합니다.
|
||||
WSL(Windows Subsystem for Linux)에서 실행해야 합니다.
|
||||
|
||||
```bash
|
||||
docker exec -it autorag-ollama bash
|
||||
@@ -94,61 +103,32 @@ ollama pull phi4
|
||||
ollama list
|
||||
```
|
||||
|
||||
### 2️⃣ AutoRAG 평가 실행
|
||||
### **2️⃣ 평가 실행**
|
||||
|
||||
```bash
|
||||
cd Autorag-workspace
|
||||
python main.py
|
||||
```
|
||||
|
||||
### 3️⃣ 평가 결과 확인
|
||||
### **3️⃣ 평가 결과 확인**
|
||||
|
||||
평가 결과는 프로젝트 폴더 내 `benchmark_*` 경로에 저장됩니다.
|
||||
|
||||
#### ✅ 전체 파이프라인 평가 결과
|
||||
#### **✅ 전체 평가 결과**
|
||||
|
||||
```bash
|
||||
cd projects/프로젝트이름/benchmark_{*}/*/
|
||||
summary.csv
|
||||
cd projects/example_project/benchmark_*/summary.csv
|
||||
```
|
||||
|
||||
#### ✅ 세부 평가 결과
|
||||
#### **✅ 세부 평가 결과**
|
||||
|
||||
- **검색기 노드 라인 평가 결과**
|
||||
```bash
|
||||
cd ./retrieve_node_line
|
||||
summary.csv
|
||||
```
|
||||
- **검색 노드 평가 결과**
|
||||
```bash
|
||||
cd ./retrieve_node_line/retrival
|
||||
summary.csv
|
||||
```
|
||||
- **리랭커 노드 평가 결과**
|
||||
```bash
|
||||
cd ./retrieve_node_line/passage_reranker
|
||||
summary.csv
|
||||
```
|
||||
- **생성기 노드 라인 평가 결과**
|
||||
```bash
|
||||
cd ./post_retrieve_node_line
|
||||
summary.csv
|
||||
```
|
||||
- **생성 노드 평가 결과**
|
||||
```bash
|
||||
cd ./post_retrieve_node_line/generator
|
||||
summary.csv
|
||||
```
|
||||
| 평가 항목 | 파일 경로 |
|
||||
| ------------------------- | --------------------------------------------------- |
|
||||
| **검색기 노드 라인 평가** | `./retrieve_node_line/summary.csv` |
|
||||
| **검색 노드 평가** | `./retrieve_node_line/retrieval/summary.csv` |
|
||||
| **리랭커 노드 평가** | `./retrieve_node_line/passage_reranker/summary.csv` |
|
||||
| **생성기 노드 라인 평가** | `./post_retrieve_node_line/summary.csv` |
|
||||
| **생성 노드 평가** | `./post_retrieve_node_line/generator/summary.csv` |
|
||||
|
||||
> 📌 **참고:** `./projects/example_01` 폴더는 데이터 생성부터 평가까지 진행된 예제입니다.
|
||||
> 📌 `./projects/example_01` 폴더에는 데이터 생성부터 평가까지 진행된 예제 파일이 포함되어 있습니다.
|
||||
|
||||
---
|
||||
|
||||
## 📊 평가 대시보드 실행
|
||||
|
||||
```bash
|
||||
cd Autorag-workspace
|
||||
sh dashboard.sh
|
||||
```
|
||||
|
||||
AutoRAG 평가 결과를 자세히 확인할 수 있습니다.
|
||||
|
||||
@@ -1,209 +0,0 @@
|
||||
import importlib.resources
|
||||
import logging
|
||||
import os
|
||||
import pathlib
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import click
|
||||
import nest_asyncio
|
||||
|
||||
from autorag import dashboard
|
||||
from autorag.deploy import extract_best_config as original_extract_best_config
|
||||
from autorag.deploy.api import ApiRunner
|
||||
from autorag.evaluator import Evaluator
|
||||
from autorag.validator import Validator
|
||||
|
||||
logger = logging.getLogger("AutoRAG")
|
||||
|
||||
autorag_dir = os.path.dirname(os.path.realpath(__file__))
|
||||
version_file = os.path.join(autorag_dir, "VERSION")
|
||||
with open(version_file, "r") as f:
|
||||
__version__ = f.read().strip()
|
||||
|
||||
|
||||
@click.group()
|
||||
@click.version_option(__version__)
|
||||
def cli():
|
||||
pass
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--config",
|
||||
"-c",
|
||||
help="Path to config yaml file. Must be yaml or yml file.",
|
||||
type=str,
|
||||
)
|
||||
@click.option(
|
||||
"--qa_data_path", help="Path to QA dataset. Must be parquet file.", type=str
|
||||
)
|
||||
@click.option(
|
||||
"--corpus_data_path", help="Path to corpus dataset. Must be parquet file.", type=str
|
||||
)
|
||||
@click.option(
|
||||
"--project_dir", help="Path to project directory.", type=str, default=None
|
||||
)
|
||||
@click.option(
|
||||
"--skip_validation",
|
||||
help="Skip validation or not. Default is False.",
|
||||
type=bool,
|
||||
default=False,
|
||||
)
|
||||
def evaluate(config, qa_data_path, corpus_data_path, project_dir, skip_validation):
|
||||
if not config.endswith(".yaml") and not config.endswith(".yml"):
|
||||
raise ValueError(f"Config file {config} is not a yaml or yml file.")
|
||||
if not os.path.exists(config):
|
||||
raise ValueError(f"Config file {config} does not exist.")
|
||||
evaluator = Evaluator(qa_data_path, corpus_data_path, project_dir=project_dir)
|
||||
evaluator.start_trial(config, skip_validation=skip_validation)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--config_path", type=str, help="Path to extracted config yaml file.", default=None
|
||||
)
|
||||
@click.option("--host", type=str, default="0.0.0.0", help="Host address")
|
||||
@click.option("--port", type=int, default=8000, help="Port number")
|
||||
@click.option(
|
||||
"--trial_dir",
|
||||
type=click.Path(file_okay=False, dir_okay=True, exists=True),
|
||||
default=None,
|
||||
help="Path to trial directory.",
|
||||
)
|
||||
@click.option(
|
||||
"--project_dir", help="Path to project directory.", type=str, default=None
|
||||
)
|
||||
@click.option(
|
||||
"--remote", help="Run the API server in remote mode.", type=bool, default=False
|
||||
)
|
||||
def run_api(config_path, host, port, trial_dir, project_dir, remote: bool):
|
||||
if trial_dir is None:
|
||||
runner = ApiRunner.from_yaml(config_path, project_dir=project_dir)
|
||||
else:
|
||||
runner = ApiRunner.from_trial_folder(trial_dir)
|
||||
logger.info(f"Running API server at {host}:{port}...")
|
||||
nest_asyncio.apply()
|
||||
runner.run_api_server(host, port, remote=remote)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--yaml_path", type=click.Path(path_type=Path), help="Path to the YAML file."
|
||||
)
|
||||
@click.option(
|
||||
"--project_dir",
|
||||
type=click.Path(path_type=Path),
|
||||
help="Path to the project directory.",
|
||||
)
|
||||
@click.option(
|
||||
"--trial_path", type=click.Path(path_type=Path), help="Path to the trial directory."
|
||||
)
|
||||
def run_web(
|
||||
yaml_path: Optional[str], project_dir: Optional[str], trial_path: Optional[str]
|
||||
):
|
||||
try:
|
||||
with importlib.resources.path("autorag", "web.py") as web_path:
|
||||
web_py_path = str(web_path)
|
||||
except ImportError:
|
||||
raise ImportError(
|
||||
"Could not locate the web.py file within the autorag package."
|
||||
" Please ensure that autorag is correctly installed."
|
||||
)
|
||||
|
||||
if not yaml_path and not trial_path:
|
||||
raise ValueError("yaml_path or trial_path must be given.")
|
||||
elif yaml_path and trial_path:
|
||||
raise ValueError("yaml_path and trial_path cannot be given at the same time.")
|
||||
elif yaml_path and not project_dir:
|
||||
subprocess.run(
|
||||
["streamlit", "run", web_py_path, "--", "--yaml_path", yaml_path]
|
||||
)
|
||||
elif yaml_path and project_dir:
|
||||
subprocess.run(
|
||||
[
|
||||
"streamlit",
|
||||
"run",
|
||||
web_py_path,
|
||||
"--",
|
||||
"--yaml_path",
|
||||
yaml_path,
|
||||
"--project_dir",
|
||||
project_dir,
|
||||
]
|
||||
)
|
||||
elif trial_path:
|
||||
subprocess.run(
|
||||
["streamlit", "run", web_py_path, "--", "--trial_path", trial_path]
|
||||
)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--trial_dir",
|
||||
type=click.Path(dir_okay=True, file_okay=False, exists=True),
|
||||
required=True,
|
||||
)
|
||||
@click.option(
|
||||
"--port", type=int, default=7690, help="Port number. The default is 7690."
|
||||
)
|
||||
def run_dashboard(trial_dir: str, port: int):
|
||||
dashboard.run(trial_dir, port=port)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option("--trial_path", type=click.Path(), help="Path to the trial directory.")
|
||||
@click.option(
|
||||
"--output_path",
|
||||
type=click.Path(),
|
||||
help="Path to the output directory." " Must be .yaml or .yml file.",
|
||||
)
|
||||
def extract_best_config(trial_path: str, output_path: str):
|
||||
original_extract_best_config(trial_path, output_path)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option("--trial_path", help="Path to trial directory.", type=str)
|
||||
def restart_evaluate(trial_path):
|
||||
if not os.path.exists(trial_path):
|
||||
raise ValueError(f"trial_path {trial_path} does not exist.")
|
||||
project_dir = str(pathlib.PurePath(trial_path).parent)
|
||||
qa_data_path = os.path.join(project_dir, "data", "qa.parquet")
|
||||
corpus_data_path = os.path.join(project_dir, "data", "corpus.parquet")
|
||||
evaluator = Evaluator(qa_data_path, corpus_data_path, project_dir)
|
||||
evaluator.restart_trial(trial_path)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--config",
|
||||
"-c",
|
||||
help="Path to config yaml file. Must be yaml or yml file.",
|
||||
type=str,
|
||||
)
|
||||
@click.option(
|
||||
"--qa_data_path", help="Path to QA dataset. Must be parquet file.", type=str
|
||||
)
|
||||
@click.option(
|
||||
"--corpus_data_path", help="Path to corpus dataset. Must be parquet file.", type=str
|
||||
)
|
||||
def validate(config, qa_data_path, corpus_data_path):
|
||||
if not config.endswith(".yaml") and not config.endswith(".yml"):
|
||||
raise ValueError(f"Config file {config} is not a parquet file.")
|
||||
if not os.path.exists(config):
|
||||
raise ValueError(f"Config file {config} does not exist.")
|
||||
validator = Validator(qa_data_path=qa_data_path, corpus_data_path=corpus_data_path)
|
||||
validator.validate(config)
|
||||
|
||||
|
||||
cli.add_command(evaluate, "evaluate")
|
||||
cli.add_command(run_api, "run_api")
|
||||
cli.add_command(run_web, "run_web")
|
||||
cli.add_command(run_dashboard, "dashboard")
|
||||
cli.add_command(extract_best_config, "extract_best_config")
|
||||
cli.add_command(restart_evaluate, "restart_evaluate")
|
||||
cli.add_command(validate, "validate")
|
||||
|
||||
if __name__ == "__main__":
|
||||
cli()
|
||||
@@ -1,199 +0,0 @@
|
||||
import ast
|
||||
import logging
|
||||
import os
|
||||
from typing import Dict, List
|
||||
|
||||
import matplotlib.pyplot as plt
|
||||
import pandas as pd
|
||||
import panel as pn
|
||||
import seaborn as sns
|
||||
import yaml
|
||||
from bokeh.models import NumberFormatter, BooleanFormatter
|
||||
|
||||
from autorag.utils.util import dict_to_markdown, dict_to_markdown_table
|
||||
|
||||
pn.extension(
|
||||
"terminal",
|
||||
"tabulator",
|
||||
"mathjax",
|
||||
"ipywidgets",
|
||||
console_output="disable",
|
||||
sizing_mode="stretch_width",
|
||||
css_files=[
|
||||
"https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.2/css/all.min.css"
|
||||
],
|
||||
)
|
||||
logger = logging.getLogger("AutoRAG")
|
||||
|
||||
|
||||
def find_node_dir(trial_dir: str) -> List[str]:
|
||||
trial_summary_df = pd.read_csv(os.path.join(trial_dir, "summary.csv"))
|
||||
result_paths = []
|
||||
for idx, row in trial_summary_df.iterrows():
|
||||
node_line_name = row["node_line_name"]
|
||||
node_type = row["node_type"]
|
||||
result_paths.append(os.path.join(trial_dir, node_line_name, node_type))
|
||||
return result_paths
|
||||
|
||||
|
||||
def get_metric_values(node_summary_df: pd.DataFrame) -> Dict:
|
||||
non_metric_column_names = [
|
||||
"filename",
|
||||
"module_name",
|
||||
"module_params",
|
||||
"execution_time",
|
||||
"average_output_token",
|
||||
"is_best",
|
||||
]
|
||||
best_row = node_summary_df.loc[node_summary_df["is_best"]].drop(
|
||||
columns=non_metric_column_names, errors="ignore"
|
||||
)
|
||||
assert len(best_row) == 1, "The best module must be only one."
|
||||
return best_row.iloc[0].to_dict()
|
||||
|
||||
|
||||
def make_trial_summary_md(trial_dir):
|
||||
markdown_text = f"""# Trial Result Summary
|
||||
- Trial Directory : {trial_dir}
|
||||
|
||||
"""
|
||||
node_dirs = find_node_dir(trial_dir)
|
||||
for node_dir in node_dirs:
|
||||
node_summary_filepath = os.path.join(node_dir, "summary.csv")
|
||||
node_type = os.path.basename(node_dir)
|
||||
node_summary_df = pd.read_csv(node_summary_filepath)
|
||||
best_row = node_summary_df.loc[node_summary_df["is_best"]].iloc[0]
|
||||
metric_dict = get_metric_values(node_summary_df)
|
||||
markdown_text += f"""---
|
||||
|
||||
## {node_type} best module
|
||||
|
||||
### Module Name
|
||||
|
||||
{best_row['module_name']}
|
||||
|
||||
### Module Params
|
||||
|
||||
{dict_to_markdown(ast.literal_eval(best_row['module_params']), level=3)}
|
||||
|
||||
### Metric Values
|
||||
|
||||
{dict_to_markdown_table(metric_dict, key_column_name='metric_name', value_column_name='metric_value')}
|
||||
|
||||
"""
|
||||
|
||||
return markdown_text
|
||||
|
||||
|
||||
def node_view(node_dir: str):
|
||||
non_metric_column_names = [
|
||||
"filename",
|
||||
"module_name",
|
||||
"module_params",
|
||||
"execution_time",
|
||||
"average_output_token",
|
||||
"is_best",
|
||||
]
|
||||
summary_df = pd.read_csv(os.path.join(node_dir, "summary.csv"))
|
||||
bokeh_formatters = {
|
||||
"float": NumberFormatter(format="0.000"),
|
||||
"bool": BooleanFormatter(),
|
||||
}
|
||||
first_df = pd.read_parquet(os.path.join(node_dir, "0.parquet"), engine="pyarrow")
|
||||
|
||||
each_module_df_widget = pn.widgets.Tabulator(
|
||||
pd.DataFrame(columns=first_df.columns),
|
||||
name="Module DataFrame",
|
||||
formatters=bokeh_formatters,
|
||||
pagination="local",
|
||||
page_size=20,
|
||||
widths=150,
|
||||
)
|
||||
|
||||
def change_module_widget(event):
|
||||
if event.column == "detail":
|
||||
filename = summary_df["filename"].iloc[event.row]
|
||||
filepath = os.path.join(node_dir, filename)
|
||||
each_module_df = pd.read_parquet(filepath, engine="pyarrow")
|
||||
each_module_df_widget.value = each_module_df
|
||||
|
||||
df_widget = pn.widgets.Tabulator(
|
||||
summary_df,
|
||||
name="Summary DataFrame",
|
||||
formatters=bokeh_formatters,
|
||||
buttons={"detail": '<i class="fa fa-eye"></i>'},
|
||||
widths=150,
|
||||
)
|
||||
df_widget.on_click(change_module_widget)
|
||||
|
||||
try:
|
||||
fig, ax = plt.subplots(figsize=(10, 5))
|
||||
metric_df = summary_df.drop(columns=non_metric_column_names, errors="ignore")
|
||||
sns.stripplot(data=metric_df, ax=ax)
|
||||
strip_plot_pane = pn.pane.Matplotlib(fig, tight=True)
|
||||
|
||||
fig2, ax2 = plt.subplots(figsize=(10, 5))
|
||||
sns.boxplot(data=metric_df, ax=ax2)
|
||||
box_plot_pane = pn.pane.Matplotlib(fig2, tight=True)
|
||||
plot_pane = pn.Row(strip_plot_pane, box_plot_pane)
|
||||
|
||||
layout = pn.Column(
|
||||
"## Summary distribution plot",
|
||||
plot_pane,
|
||||
"## Summary DataFrame",
|
||||
df_widget,
|
||||
"## Module Result DataFrame",
|
||||
each_module_df_widget,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Skipping make boxplot and stripplot with error {e}")
|
||||
layout = pn.Column("## Summary DataFrame", df_widget)
|
||||
layout.servable()
|
||||
return layout
|
||||
|
||||
|
||||
CSS = """
|
||||
div.card-margin:nth-child(1) {
|
||||
max-height: 300px;
|
||||
}
|
||||
div.card-margin:nth-child(2) {
|
||||
max-height: 400px;
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def yaml_to_markdown(yaml_filepath):
|
||||
markdown_content = ""
|
||||
with open(yaml_filepath, "r", encoding="utf-8") as file:
|
||||
try:
|
||||
content = yaml.safe_load(file)
|
||||
markdown_content += f"## {os.path.basename(yaml_filepath)}\n```yaml\n{yaml.safe_dump(content, allow_unicode=True)}\n```\n\n"
|
||||
except yaml.YAMLError as exc:
|
||||
print(f"Error in {yaml_filepath}: {exc}")
|
||||
return markdown_content
|
||||
|
||||
|
||||
def run(trial_dir: str, port: int = 7690):
|
||||
trial_summary_md = make_trial_summary_md(trial_dir=trial_dir)
|
||||
trial_summary_tab = pn.pane.Markdown(trial_summary_md, sizing_mode="stretch_width")
|
||||
|
||||
node_views = [
|
||||
(str(os.path.basename(node_dir)), node_view(node_dir))
|
||||
for node_dir in find_node_dir(trial_dir)
|
||||
]
|
||||
|
||||
yaml_file_markdown = yaml_to_markdown(os.path.join(trial_dir, "config.yaml"))
|
||||
|
||||
yaml_file = pn.pane.Markdown(yaml_file_markdown, sizing_mode="stretch_width")
|
||||
|
||||
tabs = pn.Tabs(
|
||||
("Summary", trial_summary_tab),
|
||||
*node_views,
|
||||
("Used YAML file", yaml_file),
|
||||
dynamic=True,
|
||||
)
|
||||
|
||||
template = pn.template.FastListTemplate(
|
||||
site="AutoRAG", title="Dashboard", main=[tabs], raw_css=[CSS]
|
||||
).servable()
|
||||
template.show(port=port)
|
||||
@@ -1,8 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
export BOKEH_ALLOW_WS_ORIGIN="localhost:7690,172.16.9.208:7690"
|
||||
|
||||
python -m autorag.cli dashboard \
|
||||
--trial_dir ../projects/daesan-dangjin_01/benchmark/1
|
||||
|
||||
echo "📊 AutoRAG 대시보드 http://localhost:7690/..."
|
||||
@@ -1,32 +0,0 @@
|
||||
modules:
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ Token, Sentence ]
|
||||
chunk_size: [ 1024, 512 ]
|
||||
chunk_overlap: 24
|
||||
add_file_name: en
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ SentenceWindow ]
|
||||
window_size: 3
|
||||
add_file_name: en
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ Semantic_llama_index ]
|
||||
embed_model: openai
|
||||
buffer_size: 1
|
||||
breakpoint_percentile_threshold: 95
|
||||
add_file_name: en
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ SemanticDoubleMerging ]
|
||||
add_file_name: en
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ SimpleFile ]
|
||||
add_file_name: en
|
||||
- module_type: langchain_chunk
|
||||
chunk_method: sentencetransformerstoken
|
||||
- module_type: langchain_chunk
|
||||
chunk_method: recursivecharacter
|
||||
separators: [ " ", "\n" ]
|
||||
- module_type: langchain_chunk
|
||||
chunk_method: character
|
||||
separator: ". "
|
||||
- module_type: langchain_chunk
|
||||
chunk_method: Konlpy
|
||||
@@ -1,19 +0,0 @@
|
||||
modules:
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ Token, Sentence ]
|
||||
chunk_size: [ 1024, 512 ]
|
||||
chunk_overlap: 24
|
||||
add_file_name: ko
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ SentenceWindow ]
|
||||
sentence_splitter: kiwi
|
||||
add_file_name: ko
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ Semantic_llama_index ]
|
||||
embed_model: openai
|
||||
add_file_name: ko
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: [ SimpleFile ]
|
||||
add_file_name: ko
|
||||
- module_type: langchain_chunk
|
||||
chunk_method: KonlpyTextSplitter
|
||||
@@ -1,3 +0,0 @@
|
||||
modules:
|
||||
- module_type: llama_index_chunk
|
||||
chunk_method: Token
|
||||
@@ -1,25 +0,0 @@
|
||||
# You can use only one of the following modules at a time.
|
||||
modules:
|
||||
# Use Directory Parse
|
||||
- module_type: langchain_parse
|
||||
file_type: all_files
|
||||
parse_method: directory
|
||||
# Use Unstructured
|
||||
- module_type: langchain_parse
|
||||
file_type: all_files
|
||||
parse_method: unstructured
|
||||
# Use Upsatge Document Parse
|
||||
- module_type: langchain_parse
|
||||
file_type: all_files
|
||||
parse_method: upstagedocumentparse
|
||||
# Use Naver Clova OCR
|
||||
- module_type: clova
|
||||
file_type: all_files
|
||||
table_detection: true
|
||||
# Use Llama Parse
|
||||
- module_type: llamaparse
|
||||
file_type: all_files
|
||||
result_type: markdown
|
||||
language: ko
|
||||
use_vendor_multimodal_model: true
|
||||
vendor_multimodal_model_name: openai-gpt-4o-mini
|
||||
@@ -1,26 +0,0 @@
|
||||
modules:
|
||||
# PDF
|
||||
- module_type: langchain_parse
|
||||
file_type: pdf
|
||||
parse_method: pdfminer
|
||||
# CSV
|
||||
- module_type: langchain_parse
|
||||
file_type: csv
|
||||
parse_method: csv
|
||||
# JSON
|
||||
- module_type: langchain_parse
|
||||
file_type: json
|
||||
parse_method: json
|
||||
jq_schema: .content
|
||||
# Markdown
|
||||
- module_type: langchain_parse
|
||||
file_type: md
|
||||
parse_method: unstructuredmarkdown
|
||||
# HTML
|
||||
- module_type: langchain_parse
|
||||
file_type: html
|
||||
parse_method: bshtml
|
||||
# XML
|
||||
- module_type: langchain_parse
|
||||
file_type: xml
|
||||
parse_method: unstructuredxml
|
||||
@@ -1,12 +0,0 @@
|
||||
modules:
|
||||
- module_type: table_hybrid_parse
|
||||
file_type: pdf
|
||||
text_parse_module: langchain_parse
|
||||
text_params:
|
||||
parse_method: pdfplumber
|
||||
table_parse_module: llamaparse
|
||||
table_params:
|
||||
result_type: markdown
|
||||
language: ko
|
||||
use_vendor_multimodal_model: true
|
||||
vendor_multimodal_model_name: openai-gpt-4o-mini
|
||||
@@ -1,11 +0,0 @@
|
||||
modules:
|
||||
- module_type: llama_parse
|
||||
file_type: all_files
|
||||
result_type: markdown
|
||||
language: ko
|
||||
- module_type: clova
|
||||
file_type: all_files
|
||||
table_detection: true
|
||||
- module_type: langchain_parse
|
||||
file_type: all_files
|
||||
parse_method: upstagedocumentparse
|
||||
@@ -1,8 +0,0 @@
|
||||
modules:
|
||||
- module_type: llamaparse
|
||||
file_type: all_files
|
||||
result_type: markdown
|
||||
language: ko
|
||||
use_vendor_multimodal_model: true
|
||||
vendor_multimodal_model_name: openai-gpt-4o-mini
|
||||
use_own_key: true
|
||||
@@ -1,10 +0,0 @@
|
||||
modules:
|
||||
- module_type: langchain_parse
|
||||
file_type: all_files
|
||||
parse_method: upstagedocumentparse
|
||||
- module_type: llama_parse
|
||||
file_type: all_files
|
||||
result_type: markdown
|
||||
- module_type: clova
|
||||
file_type: all_files
|
||||
table_detection: true
|
||||
@@ -1,4 +0,0 @@
|
||||
modules:
|
||||
- module_type: langchain_parse
|
||||
file_type: pdf
|
||||
parse_method: pdfminer
|
||||
@@ -1,50 +0,0 @@
|
||||
vectordb:
|
||||
- name: mpnet_base_chroma
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_all_mpnet_base_v2
|
||||
collection_name: huggingface_all_mpnet_base_v2
|
||||
path: ${PROJECT_DIR}/data/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
top_k: 20
|
||||
modules:
|
||||
- module_type: bm25
|
||||
- module_type: vectordb
|
||||
vectordb: mpnet_base_chroma
|
||||
- module_type: hybrid_rrf
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
top_k: 3
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: upr
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics: [bleu, meteor, rouge, sem_score]
|
||||
generator_modules:
|
||||
- module_type: vllm
|
||||
llm: mistralai/Mistral-7B-Instruct-v0.2
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics: [bleu, meteor, rouge, sem_score]
|
||||
modules:
|
||||
- module_type: vllm
|
||||
llm: mistralai/Mistral-7B-Instruct-v0.2
|
||||
temperature: [0.1, 0.5, 1.1]
|
||||
@@ -1,101 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 10
|
||||
top_k: 5
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: monot5
|
||||
- module_type: upr
|
||||
- module_type: rankgpt
|
||||
- module_type: colbert_reranker
|
||||
- module_type: sentence_transformer_reranker
|
||||
- module_type: flag_embedding_reranker
|
||||
- module_type: flag_embedding_llm_reranker
|
||||
- module_type: time_reranker
|
||||
- module_type: openvino_reranker
|
||||
- module_type: flashrank_reranker
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- module_type: long_context_reorder
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,154 +0,0 @@
|
||||
vectordb:
|
||||
- name: chroma_bge_m3
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_bge_m3
|
||||
collection_name: openai
|
||||
path: ${PROJECT_DIR}/resources/chroma
|
||||
node_lines:
|
||||
- node_line_name: pre_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: query_expansion
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
retrieval_modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_bge_m3
|
||||
modules:
|
||||
- module_type: pass_query_expansion
|
||||
- module_type: query_decompose
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
- module_type: hyde
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
max_token: 64
|
||||
- module_type: multi_query_expansion
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
temperature: [ 0.2, 1.0 ]
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_bge_m3
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 5
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: monot5
|
||||
- module_type: upr
|
||||
- module_type: rankgpt
|
||||
- module_type: colbert_reranker
|
||||
- module_type: sentence_transformer_reranker
|
||||
- module_type: flag_embedding_reranker
|
||||
- module_type: flag_embedding_llm_reranker
|
||||
- module_type: time_reranker
|
||||
- module_type: openvino_reranker
|
||||
- module_type: flashrank_reranker
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: recency_filter
|
||||
threshold_datetime: 2015-01-01 3:45:07
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- module_type: window_replacement
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval # LLM Judge Metric. Default Model: gpt-4-turbo
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0, 1.5]
|
||||
- module_type: openai_llm
|
||||
llm: gpt-4o-mini
|
||||
temperature: 0.8
|
||||
@@ -1,121 +0,0 @@
|
||||
vectordb:
|
||||
- name: chroma_bge_m3
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_bge_m3
|
||||
collection_name: openai
|
||||
path: ${PROJECT_DIR}/resources/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_bge_m3
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 10
|
||||
top_k: 5
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: monot5
|
||||
- module_type: upr
|
||||
- module_type: rankgpt
|
||||
- module_type: colbert_reranker
|
||||
- module_type: sentence_transformer_reranker
|
||||
- module_type: flag_embedding_reranker
|
||||
- module_type: flag_embedding_llm_reranker
|
||||
- module_type: time_reranker
|
||||
- module_type: openvino_reranker
|
||||
- module_type: flashrank_reranker
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- module_type: long_context_reorder
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,105 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 10
|
||||
top_k: 5
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: monot5
|
||||
- module_type: upr
|
||||
- module_type: cohere_reranker
|
||||
- module_type: rankgpt
|
||||
- module_type: jina_reranker
|
||||
- module_type: colbert_reranker
|
||||
- module_type: sentence_transformer_reranker
|
||||
- module_type: flag_embedding_reranker
|
||||
- module_type: flag_embedding_llm_reranker
|
||||
- module_type: time_reranker
|
||||
- module_type: openvino_reranker
|
||||
- module_type: voyageai_reranker
|
||||
- module_type: mixedbreadai_reranker
|
||||
- module_type: flashrank_reranker
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- module_type: long_context_reorder
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,151 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: pre_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: query_expansion
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
retrieval_modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
modules:
|
||||
- module_type: pass_query_expansion
|
||||
- module_type: query_decompose
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
- module_type: hyde
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
max_token: 64
|
||||
- module_type: multi_query_expansion
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
temperature: [ 0.2, 1.0 ]
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 5
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: monot5
|
||||
- module_type: upr
|
||||
- module_type: cohere_reranker
|
||||
- module_type: rankgpt
|
||||
- module_type: jina_reranker
|
||||
- module_type: colbert_reranker
|
||||
- module_type: sentence_transformer_reranker
|
||||
- module_type: flag_embedding_reranker
|
||||
- module_type: flag_embedding_llm_reranker
|
||||
- module_type: time_reranker
|
||||
- module_type: openvino_reranker
|
||||
- module_type: voyageai_reranker
|
||||
- module_type: mixedbreadai_reranker
|
||||
- module_type: flashrank_reranker
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: recency_filter
|
||||
threshold_datetime: 2015-01-01 3:45:07
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- module_type: window_replacement
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval # LLM Judge Metric. Default Model: gpt-4-turbo
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0, 1.5]
|
||||
- module_type: openai_llm
|
||||
llm: gpt-4o-mini
|
||||
temperature: 0.8
|
||||
@@ -1,118 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 10
|
||||
top_k: 5
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: monot5
|
||||
- module_type: upr
|
||||
- module_type: cohere_reranker
|
||||
- module_type: rankgpt
|
||||
- module_type: jina_reranker
|
||||
- module_type: colbert_reranker
|
||||
- module_type: sentence_transformer_reranker
|
||||
- module_type: flag_embedding_reranker
|
||||
- module_type: flag_embedding_llm_reranker
|
||||
- module_type: time_reranker
|
||||
- module_type: openvino_reranker
|
||||
- module_type: voyageai_reranker
|
||||
- module_type: mixedbreadai_reranker
|
||||
- module_type: flashrank_reranker
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- module_type: long_context_reorder
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,83 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- module_type: long_context_reorder
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,129 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: pre_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: query_expansion
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
retrieval_modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
modules:
|
||||
- module_type: pass_query_expansion
|
||||
- module_type: query_decompose
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
- module_type: hyde
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
max_token: 64
|
||||
- module_type: multi_query_expansion
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
temperature: [ 0.2, 1.0 ]
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
- module_type: vectordb
|
||||
embedding_model: openai
|
||||
embedding_batch: 256
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: recency_filter
|
||||
threshold_datetime: 2015-01-01 3:45:07
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- module_type: window_replacement
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval # LLM Judge Metric. Default Model: gpt-4-turbo
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0, 1.5]
|
||||
- module_type: openai_llm
|
||||
llm: gpt-4o-mini
|
||||
temperature: 0.8
|
||||
@@ -1,95 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- module_type: long_context_reorder
|
||||
prompt:
|
||||
- "Answer to given questions with the following passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- "There is a passages related to user question. Please response carefully to the following question. \n\n Passage: {retrieved_contents} \n\n Question: {query} \n\n Answer the question. Think step by step." # Zero-shot CoT prompt
|
||||
- "{retrieved_contents} \n\n Read the passage carefully, and answer this question. \n\n Question: {query} \n\n Answer the question. Be concise." # concise prompt
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,33 +0,0 @@
|
||||
vectordb:
|
||||
- name: mpnet_base_chroma
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_all_mpnet_base_v2
|
||||
collection_name: huggingface_all_mpnet_base_v2
|
||||
path: ${PROJECT_DIR}/data/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
top_k: 3
|
||||
modules:
|
||||
- module_type: vectordb
|
||||
vectordb: mpnet_base_chroma
|
||||
- node_line_name: post_retrieve_node_line
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics: [ meteor, rouge, bert_score ]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics: [ bleu, rouge, bert_score ]
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: bedrock
|
||||
model: amazon.titan-text-express-v1
|
||||
profile_name: your_profile_name # Plz replace this with your profile name
|
||||
@@ -1,31 +0,0 @@
|
||||
vectordb:
|
||||
- name: baai_chroma
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_baai_bge_small
|
||||
collection_name: huggingface_baai_bge_small
|
||||
path: ${PROJECT_DIR}/data/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
top_k: 3
|
||||
modules:
|
||||
- module_type: vectordb
|
||||
vectordb: baai_chroma
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics: [ meteor, rouge, bert_score ]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics: [ bleu, rouge, bert_score ]
|
||||
modules:
|
||||
- module_type: vllm
|
||||
llm: mistralai/Mistral-7B-Instruct-v0.2
|
||||
@@ -1,34 +0,0 @@
|
||||
vectordb:
|
||||
- name: mpnet_base_chroma
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_all_mpnet_base_v2
|
||||
collection_name: huggingface_all_mpnet_base_v2
|
||||
path: ${PROJECT_DIR}/data/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
top_k: 3
|
||||
modules:
|
||||
- module_type: vectordb
|
||||
vectordb: mpnet_base_chroma
|
||||
- node_line_name: post_retrieve_node_line
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics: [ meteor, rouge, bert_score ]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics: [ bleu, rouge, bert_score ]
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: ollama
|
||||
model: llama3
|
||||
batch: 1
|
||||
request_timeout: 100 # You can increase this value if your model is big (slow)
|
||||
@@ -1,25 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
top_k: 3
|
||||
modules:
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics: [bleu, meteor, rouge]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics: [bleu, rouge]
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
@@ -1,47 +0,0 @@
|
||||
vectordb:
|
||||
- name: default
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: openai
|
||||
collection_name: openai
|
||||
path: ${PROJECT_DIR}/data/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
modules:
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
top_k: 3
|
||||
strategy:
|
||||
metrics:
|
||||
- retrieval_f1
|
||||
- retrieval_recall
|
||||
- retrieval_precision
|
||||
- node_line_name: post_retrieve_node_line
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: "Read the passages and answer the given question. \n Question: {query} \n Passage: {retrieved_contents} \n Answer : "
|
||||
strategy:
|
||||
generator_modules:
|
||||
- batch: 2
|
||||
llm: openai
|
||||
module_type: llama_index_llm
|
||||
metrics:
|
||||
- bleu
|
||||
- meteor
|
||||
- rouge
|
||||
- node_type: generator
|
||||
modules:
|
||||
- batch: 2
|
||||
llm: openai
|
||||
model: gpt-3.5-turbo-16k
|
||||
module_type: llama_index_llm
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
@@ -1,159 +0,0 @@
|
||||
vectordb:
|
||||
- name: chroma_large
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: openai_embed_3_large
|
||||
collection_name: openai_embed_3_large
|
||||
path: ${PROJECT_DIR}/resources/chroma
|
||||
node_lines:
|
||||
- node_line_name: pre_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: query_expansion
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
retrieval_modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, ko_kiwi, space, gpt2, ko_okt, ko_kkma, sudachipy ]
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_large
|
||||
modules:
|
||||
- module_type: pass_query_expansion
|
||||
- module_type: query_decompose
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-3.5-turbo-16k, gpt-3.5-turbo-1106 ]
|
||||
- module_type: hyde
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-3.5-turbo-16k ]
|
||||
max_token: 64
|
||||
- module_type: multi_query_expansion
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
temperature: [ 0.2, 1.0 ]
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_large
|
||||
embedding_batch: 256
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 5
|
||||
modules:
|
||||
- module_type: pass_reranker
|
||||
- module_type: tart
|
||||
- module_type: monot5
|
||||
- module_type: upr
|
||||
- module_type: cohere_reranker
|
||||
- module_type: rankgpt
|
||||
- module_type: jina_reranker
|
||||
- module_type: colbert_reranker
|
||||
- module_type: sentence_transformer_reranker
|
||||
- module_type: flag_embedding_reranker
|
||||
- module_type: flag_embedding_llm_reranker
|
||||
- module_type: time_reranker
|
||||
- module_type: openvino_reranker
|
||||
- module_type: voyageai_reranker
|
||||
- module_type: mixedbreadai_reranker
|
||||
- module_type: flashrank_reranker
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: recency_filter
|
||||
threshold_datetime: 2015-01-01 3:45:07
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-3.5-turbo-16k
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-3.5-turbo-16k
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-3.5-turbo-16k, gpt-3.5-turbo-1106]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- module_type: window_replacement
|
||||
prompt: [ "Tell me something about the question: {query} \n\n {retrieved_contents}",
|
||||
"Question: {query} \n Something to read: {retrieved_contents} \n What's your answer?" ]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
- metric_name: g_eval
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-3.5-turbo-16k, gpt-3.5-turbo-1106]
|
||||
temperature: [0.5, 1.0, 1.5]
|
||||
- module_type: openai_llm
|
||||
llm: gpt-3.5-turbo
|
||||
temperature: 0.8
|
||||
@@ -1,93 +0,0 @@
|
||||
vectordb:
|
||||
- name: chroma_bge_m3
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_bge_m3
|
||||
collection_name: openai
|
||||
path: ${PROJECT_DIR}/resources/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_bge_m3
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
modules:
|
||||
- module_type: koreranker
|
||||
- module_type: flag_embedding_llm_reranker # Requires enough GPU resources
|
||||
- module_type: pass_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_recall, retrieval_precision, retrieval_map ]
|
||||
top_k: 3
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,157 +0,0 @@
|
||||
vectordb:
|
||||
- name: chroma_bge_m3
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_bge_m3
|
||||
collection_name: openai
|
||||
path: ${PROJECT_DIR}/resources/chroma
|
||||
node_lines:
|
||||
- node_line_name: pre_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: query_expansion
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
retrieval_modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_bge_m3
|
||||
modules:
|
||||
- module_type: pass_query_expansion
|
||||
- module_type: hyde
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
max_token: 64
|
||||
prompt: "질문에 답하기 위한 단락을 작성해 주세요."
|
||||
- module_type: multi_query_expansion
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
temperature: [ 0.2, 1.0 ]
|
||||
prompt: |
|
||||
당신은 인공지능 언어 모델 어시스턴트입니다.
|
||||
주어진 사용자 질문을 이용해 세 가지 버전의 새 질문을 생성하여 벡터 데이터베이스에서 관련 문서를 검색하는 것이 과제입니다.
|
||||
주어진 질문에 대한 다양한 관점을 생성함으로써 사용자가 거리 기반 유사도 검색의 한계를 극복할 수 있도록 돕는 것이 목표입니다.
|
||||
다음과 같은 대체 질문을 줄 바꿈으로 구분하여 제공하십시오.
|
||||
원래 질문: {query}
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_bge_m3
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
modules:
|
||||
- module_type: koreranker
|
||||
- module_type: flag_embedding_llm_reranker # Requires enough GPU resources
|
||||
- module_type: pass_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_recall, retrieval_precision, retrieval_map ]
|
||||
top_k: 3
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
여러 문맥 정보는 다음과 같습니다.\n
|
||||
---------------------\n
|
||||
{context_str}\n
|
||||
---------------------\n
|
||||
사전 지식이 아닌 여러 정보가 주어졌습니다,
|
||||
질문에 대답하세요.\n
|
||||
질문: {query_str}\n
|
||||
답변:
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
원래 질문은 다음과 같습니다: {query_str}
|
||||
기존 답변은 다음과 같습니다: {existing_answer}
|
||||
아래에서 기존 답변을 정제할 수 있는 기회가 있습니다.
|
||||
(필요한 경우에만) 아래에 몇 가지 맥락을 추가하여 기존 답변을 정제할 수 있습니다.
|
||||
------------
|
||||
{context_msg}
|
||||
------------
|
||||
새로운 문맥이 주어지면 기존 답변을 수정하여 질문에 대한 답변을 정제합니다.
|
||||
맥락이 쓸모 없다면, 기존 답변을 그대로 답변하세요.
|
||||
정제된 답변:
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
- metric_name: g_eval # LLM Judge Metric. Default Model: gpt-4-turbo
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,126 +0,0 @@
|
||||
vectordb:
|
||||
- name: chroma_bge_m3
|
||||
db_type: chroma
|
||||
client_type: persistent
|
||||
embedding_model: huggingface_bge_m3
|
||||
collection_name: openai
|
||||
path: ${PROJECT_DIR}/resources/chroma
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: chroma_bge_m3
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
modules:
|
||||
- module_type: koreranker
|
||||
- module_type: flag_embedding_llm_reranker # Requires enough GPU resources
|
||||
- module_type: pass_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_recall, retrieval_precision, retrieval_map ]
|
||||
top_k: 3
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
여러 문맥 정보는 다음과 같습니다.\n
|
||||
---------------------\n
|
||||
{context_str}\n
|
||||
---------------------\n
|
||||
사전 지식이 아닌 여러 정보가 주어졌습니다,
|
||||
질문에 대답하세요.\n
|
||||
질문: {query_str}\n
|
||||
답변:
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
원래 질문은 다음과 같습니다: {query_str}
|
||||
기존 답변은 다음과 같습니다: {existing_answer}
|
||||
아래에서 기존 답변을 정제할 수 있는 기회가 있습니다.
|
||||
(필요한 경우에만) 아래에 몇 가지 맥락을 추가하여 기존 답변을 정제할 수 있습니다.
|
||||
------------
|
||||
{context_msg}
|
||||
------------
|
||||
새로운 문맥이 주어지면 기존 답변을 수정하여 질문에 대한 답변을 정제합니다.
|
||||
맥락이 쓸모 없다면, 기존 답변을 그대로 답변하세요.
|
||||
정제된 답변:
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,87 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
modules:
|
||||
- module_type: koreranker
|
||||
- module_type: flag_embedding_llm_reranker # Requires enough GPU resources
|
||||
- module_type: cohere_reranker # Set Environment Variable: COHERE_API_KEY
|
||||
- module_type: pass_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_recall, retrieval_precision, retrieval_map ]
|
||||
top_k: 3
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,151 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: pre_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: query_expansion
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
retrieval_modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
modules:
|
||||
- module_type: pass_query_expansion
|
||||
- module_type: hyde
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai #
|
||||
model: [ gpt-4o-mini ] #
|
||||
max_token: 64
|
||||
prompt: "질문에 답하기 위한 단락을 작성해 주세요."
|
||||
- module_type: multi_query_expansion
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
temperature: [ 0.2, 1.0 ]
|
||||
prompt: |
|
||||
당신은 인공지능 언어 모델 어시스턴트입니다.
|
||||
주어진 사용자 질문을 이용해 세 가지 버전의 새 질문을 생성하여 벡터 데이터베이스에서 관련 문서를 검색하는 것이 과제입니다.
|
||||
주어진 질문에 대한 다양한 관점을 생성함으로써 사용자가 거리 기반 유사도 검색의 한계를 극복할 수 있도록 돕는 것이 목표입니다.
|
||||
다음과 같은 대체 질문을 줄 바꿈으로 구분하여 제공하십시오.
|
||||
원래 질문: {query}
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ] # ko_kiwi, ko_okt
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
modules:
|
||||
- module_type: koreranker
|
||||
- module_type: flag_embedding_llm_reranker # Requires enough GPU resources
|
||||
- module_type: cohere_reranker # Set Environment Variable: COHERE_API_KEY
|
||||
- module_type: pass_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_recall, retrieval_precision, retrieval_map ]
|
||||
top_k: 3
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
여러 문맥 정보는 다음과 같습니다.\n
|
||||
---------------------\n
|
||||
{context_str}\n
|
||||
---------------------\n
|
||||
사전 지식이 아닌 여러 정보가 주어졌습니다,
|
||||
질문에 대답하세요.\n
|
||||
질문: {query_str}\n
|
||||
답변:
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
원래 질문은 다음과 같습니다: {query_str}
|
||||
기존 답변은 다음과 같습니다: {existing_answer}
|
||||
아래에서 기존 답변을 정제할 수 있는 기회가 있습니다.
|
||||
(필요한 경우에만) 아래에 몇 가지 맥락을 추가하여 기존 답변을 정제할 수 있습니다.
|
||||
------------
|
||||
{context_msg}
|
||||
------------
|
||||
새로운 문맥이 주어지면 기존 답변을 수정하여 질문에 대한 답변을 정제합니다.
|
||||
맥락이 쓸모 없다면, 기존 답변을 그대로 답변하세요.
|
||||
정제된 답변:
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
- metric_name: g_eval # LLM Judge Metric. Default Model: gpt-4-turbo
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,120 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_reranker
|
||||
modules:
|
||||
- module_type: koreranker
|
||||
- module_type: flag_embedding_llm_reranker # Requires enough GPU resources
|
||||
- module_type: cohere_reranker # Set Environment Variable: COHERE_API_KEY
|
||||
- module_type: pass_reranker
|
||||
strategy:
|
||||
metrics: [ retrieval_recall, retrieval_precision, retrieval_map ]
|
||||
top_k: 3
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
여러 문맥 정보는 다음과 같습니다.\n
|
||||
---------------------\n
|
||||
{context_str}\n
|
||||
---------------------\n
|
||||
사전 지식이 아닌 여러 정보가 주어졌습니다,
|
||||
질문에 대답하세요.\n
|
||||
질문: {query_str}\n
|
||||
답변:
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
원래 질문은 다음과 같습니다: {query_str}
|
||||
기존 답변은 다음과 같습니다: {existing_answer}
|
||||
아래에서 기존 답변을 정제할 수 있는 기회가 있습니다.
|
||||
(필요한 경우에만) 아래에 몇 가지 맥락을 추가하여 기존 답변을 정제할 수 있습니다.
|
||||
------------
|
||||
{context_msg}
|
||||
------------
|
||||
새로운 문맥이 주어지면 기존 답변을 수정하여 질문에 대한 답변을 정제합니다.
|
||||
맥락이 쓸모 없다면, 기존 답변을 그대로 답변하세요.
|
||||
정제된 답변:
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,78 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,142 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: pre_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: query_expansion
|
||||
strategy:
|
||||
metrics: [retrieval_f1, retrieval_recall, retrieval_precision]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
retrieval_modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ porter_stemmer, space, gpt2 ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
modules:
|
||||
- module_type: pass_query_expansion
|
||||
- module_type: hyde
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
max_token: 64
|
||||
prompt: "질문에 답하기 위한 단락을 작성해 주세요."
|
||||
- module_type: multi_query_expansion
|
||||
generator_module_type: llama_index_llm
|
||||
llm: openai
|
||||
temperature: [ 0.2, 1.0 ]
|
||||
prompt: |
|
||||
당신은 인공지능 언어 모델 어시스턴트입니다.
|
||||
주어진 사용자 질문을 이용해 세 가지 버전의 새 질문을 생성하여 벡터 데이터베이스에서 관련 문서를 검색하는 것이 과제입니다.
|
||||
주어진 질문에 대한 다양한 관점을 생성함으로써 사용자가 거리 기반 유사도 검색의 한계를 극복할 수 있도록 돕는 것이 목표입니다.
|
||||
다음과 같은 대체 질문을 줄 바꿈으로 구분하여 제공하십시오.
|
||||
원래 질문: {query}
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
여러 문맥 정보는 다음과 같습니다.\n
|
||||
---------------------\n
|
||||
{context_str}\n
|
||||
---------------------\n
|
||||
사전 지식이 아닌 여러 정보가 주어졌습니다,
|
||||
질문에 대답하세요.\n
|
||||
질문: {query_str}\n
|
||||
답변:
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
원래 질문은 다음과 같습니다: {query_str}
|
||||
기존 답변은 다음과 같습니다: {existing_answer}
|
||||
아래에서 기존 답변을 정제할 수 있는 기회가 있습니다.
|
||||
(필요한 경우에만) 아래에 몇 가지 맥락을 추가하여 기존 답변을 정제할 수 있습니다.
|
||||
------------
|
||||
{context_msg}
|
||||
------------
|
||||
새로운 문맥이 주어지면 기존 답변을 수정하여 질문에 대한 답변을 정제합니다.
|
||||
맥락이 쓸모 없다면, 기존 답변을 그대로 답변하세요.
|
||||
정제된 답변:
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
- metric_name: g_eval # LLM Judge Metric. Default Model: gpt-4-turbo
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,111 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision,
|
||||
retrieval_ndcg, retrieval_map, retrieval_mrr ]
|
||||
speed_threshold: 10
|
||||
top_k: 10
|
||||
modules:
|
||||
- module_type: bm25
|
||||
bm25_tokenizer: [ ko_kiwi, ko_okt, ko_kkma ]
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- module_type: hybrid_rrf
|
||||
weight_range: (4,80)
|
||||
- module_type: hybrid_cc
|
||||
normalize_method: [ mm, tmm, z, dbsf ]
|
||||
weight_range: (0.0, 1.0)
|
||||
test_weight_size: 101
|
||||
- node_type: passage_augmenter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
top_k: 5
|
||||
embedding_model: openai
|
||||
modules:
|
||||
- module_type: pass_passage_augmenter
|
||||
- module_type: prev_next_augmenter
|
||||
mode: next
|
||||
- node_type: passage_filter
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
speed_threshold: 5
|
||||
modules:
|
||||
- module_type: pass_passage_filter
|
||||
- module_type: similarity_threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: similarity_percentile_cutoff
|
||||
percentile: 0.6
|
||||
- module_type: threshold_cutoff
|
||||
threshold: 0.85
|
||||
- module_type: percentile_cutoff
|
||||
percentile: 0.6
|
||||
- node_type: passage_compressor
|
||||
strategy:
|
||||
metrics: [retrieval_token_f1, retrieval_token_recall, retrieval_token_precision]
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: pass_compressor
|
||||
- module_type: tree_summarize
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
여러 문맥 정보는 다음과 같습니다.\n
|
||||
---------------------\n
|
||||
{context_str}\n
|
||||
---------------------\n
|
||||
사전 지식이 아닌 여러 정보가 주어졌습니다,
|
||||
질문에 대답하세요.\n
|
||||
질문: {query_str}\n
|
||||
답변:
|
||||
- module_type: refine
|
||||
llm: openai
|
||||
model: gpt-4o-mini
|
||||
prompt: |
|
||||
원래 질문은 다음과 같습니다: {query_str}
|
||||
기존 답변은 다음과 같습니다: {existing_answer}
|
||||
아래에서 기존 답변을 정제할 수 있는 기회가 있습니다.
|
||||
(필요한 경우에만) 아래에 몇 가지 맥락을 추가하여 기존 답변을 정제할 수 있습니다.
|
||||
------------
|
||||
{context_msg}
|
||||
------------
|
||||
새로운 문맥이 주어지면 기존 답변을 수정하여 질문에 대한 답변을 정제합니다.
|
||||
맥락이 쓸모 없다면, 기존 답변을 그대로 답변하세요.
|
||||
정제된 답변:
|
||||
- module_type: longllmlingua
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: bleu
|
||||
- metric_name: meteor
|
||||
- metric_name: rouge
|
||||
- metric_name: sem_score
|
||||
embedding_model: openai
|
||||
speed_threshold: 10
|
||||
generator_modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [gpt-4o-mini]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- module_type: long_context_reorder
|
||||
prompt: ["주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"]
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
speed_threshold: 10
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: [openai]
|
||||
model: [gpt-4o-mini]
|
||||
temperature: [0.5, 1.0]
|
||||
@@ -1,30 +0,0 @@
|
||||
node_lines:
|
||||
- node_line_name: retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: retrieval
|
||||
strategy:
|
||||
metrics: [ retrieval_f1, retrieval_recall, retrieval_precision ]
|
||||
top_k: 3
|
||||
modules:
|
||||
- module_type: vectordb
|
||||
vectordb: default
|
||||
- node_line_name: post_retrieve_node_line # Arbitrary node line name
|
||||
nodes:
|
||||
- node_type: prompt_maker
|
||||
strategy:
|
||||
metrics: [ bleu, meteor, rouge ]
|
||||
modules:
|
||||
- module_type: fstring
|
||||
prompt: "주어진 passage만을 이용하여 question에 따라 답하시오 passage: {retrieved_contents} \n\n Question: {query} \n\n Answer:"
|
||||
- node_type: generator
|
||||
strategy:
|
||||
metrics:
|
||||
- metric_name: rouge
|
||||
- embedding_model: openai
|
||||
metric_name: sem_score
|
||||
- metric_name: bert_score
|
||||
lang: ko
|
||||
modules:
|
||||
- module_type: llama_index_llm
|
||||
llm: openai
|
||||
model: [ gpt-4o-mini ]
|
||||
@@ -1,25 +0,0 @@
|
||||
# sample_dataset handling
|
||||
|
||||
The sample_dataset folder does not includes a `qa.parquet`, `corpus.parquet` file that is significantly large and cannot be uploaded directly to Git due to size limitations.
|
||||
|
||||
To prepare and use datasets available in the sample_dataset folder, specifically `triviaqa`, `hotpotqa`, `msmarco` and `eli5`, you can follow the outlined methods below.
|
||||
|
||||
## Usage
|
||||
|
||||
The example provided uses `triviaqa`, but the same approach applies to `msmarco`, `eli5` and `hotpotqa`.
|
||||
|
||||
### 1. Run with a specified save path
|
||||
To execute the Python script from the terminal and save the dataset to a specified path, use the command:
|
||||
|
||||
```bash
|
||||
python ./sample_dataset/triviaqa/load_triviaqa_dataset.py --save_path /path/to/save/dataset
|
||||
```
|
||||
This runs the `load_triviaqa_dataset.py` script located in the `./sample_dataset/triviaqa/` directory,
|
||||
using the `--save_path` argument to specify the dataset's save location.
|
||||
|
||||
### 2. Run without specifying a save path
|
||||
If you run the script without the `--save_path` argument, the dataset will be saved to a default location, which is the directory containing the `load_triviaqa_dataset.py` file, essentially `./sample_dataset/triviaqa/`:
|
||||
```bash
|
||||
python ./sample_dataset/triviaqa/load_triviaqa_dataset.py
|
||||
```
|
||||
This behavior allows for a straightforward execution without needing to specify a path, making it convenient for quick tests or when working directly within the target directory.
|
||||
@@ -1,35 +0,0 @@
|
||||
import os
|
||||
import pathlib
|
||||
|
||||
import click
|
||||
from datasets import load_dataset
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--save_path",
|
||||
type=str,
|
||||
default=pathlib.PurePath(__file__).parent,
|
||||
help="Path to save sample eli5 dataset.",
|
||||
)
|
||||
def load_eli5_dataset(save_path):
|
||||
# set file path
|
||||
file_path = "MarkrAI/eli5_sample_autorag"
|
||||
|
||||
# load dataset
|
||||
corpus_dataset = load_dataset(file_path, "corpus")["train"].to_pandas()
|
||||
qa_train_dataset = load_dataset(file_path, "qa")["train"].to_pandas()
|
||||
qa_test_dataset = load_dataset(file_path, "qa")["test"].to_pandas()
|
||||
|
||||
# save data
|
||||
if os.path.exists(os.path.join(save_path, "corpus.parquet")) is True:
|
||||
raise ValueError("corpus.parquet already exists")
|
||||
if os.path.exists(os.path.join(save_path, "qa.parquet")) is True:
|
||||
raise ValueError("qa.parquet already exists")
|
||||
corpus_dataset.to_parquet(os.path.join(save_path, "corpus.parquet"))
|
||||
qa_train_dataset.to_parquet(os.path.join(save_path, "qa_train.parquet"))
|
||||
qa_test_dataset.to_parquet(os.path.join(save_path, "qa_test.parquet"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
load_eli5_dataset()
|
||||
@@ -1,35 +0,0 @@
|
||||
import os
|
||||
import pathlib
|
||||
|
||||
import click
|
||||
from datasets import load_dataset
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--save_path",
|
||||
type=str,
|
||||
default=pathlib.PurePath(__file__).parent,
|
||||
help="Path to save sample hotpotqa dataset.",
|
||||
)
|
||||
def load_hotpotqa_dataset(save_path):
|
||||
# set file path
|
||||
file_path = "gnekt/hotpotqa_small_sample_autorag"
|
||||
|
||||
# load dataset
|
||||
corpus_dataset = load_dataset(file_path, "corpus")["train"].to_pandas()
|
||||
qa_validation_dataset = load_dataset(file_path, "qa")["validation"].to_pandas()
|
||||
|
||||
# save corpus data
|
||||
if os.path.exists(os.path.join(save_path, "corpus.parquet")) is True:
|
||||
raise ValueError("corpus.parquet already exists")
|
||||
if os.path.exists(os.path.join(save_path, "qa.parquet")) is True:
|
||||
raise ValueError("qa.parquet already exists")
|
||||
corpus_dataset.to_parquet(os.path.join(save_path, "corpus.parquet"), index=False)
|
||||
qa_validation_dataset.to_parquet(
|
||||
os.path.join(save_path, "qa_validation.parquet"), index=False
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
load_hotpotqa_dataset()
|
||||
@@ -1,37 +0,0 @@
|
||||
import os
|
||||
import pathlib
|
||||
|
||||
import click
|
||||
from datasets import load_dataset
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--save_path",
|
||||
type=str,
|
||||
default=pathlib.PurePath(__file__).parent,
|
||||
help="Path to save sample msmarco dataset.",
|
||||
)
|
||||
def load_msmarco_dataset(save_path):
|
||||
# set file path
|
||||
file_path = "MarkrAI/msmarco_sample_autorag"
|
||||
|
||||
# load dataset
|
||||
corpus_dataset = load_dataset(file_path, "corpus")["train"].to_pandas()
|
||||
qa_train_dataset = load_dataset(file_path, "qa")["train"].to_pandas()
|
||||
qa_test_dataset = load_dataset(file_path, "qa")["test"].to_pandas()
|
||||
|
||||
# save corpus data
|
||||
if os.path.exists(os.path.join(save_path, "corpus.parquet")) is True:
|
||||
raise ValueError("corpus.parquet already exists")
|
||||
if os.path.exists(os.path.join(save_path, "qa.parquet")) is True:
|
||||
raise ValueError("qa.parquet already exists")
|
||||
corpus_dataset.to_parquet(os.path.join(save_path, "corpus.parquet"), index=False)
|
||||
qa_train_dataset.to_parquet(
|
||||
os.path.join(save_path, "qa_train.parquet"), index=False
|
||||
)
|
||||
qa_test_dataset.to_parquet(os.path.join(save_path, "qa_test.parquet"), index=False)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
load_msmarco_dataset()
|
||||
@@ -1,37 +0,0 @@
|
||||
import os
|
||||
import pathlib
|
||||
|
||||
import click
|
||||
from datasets import load_dataset
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--save_path",
|
||||
type=str,
|
||||
default=pathlib.PurePath(__file__).parent,
|
||||
help="Path to save sample triviaqa dataset.",
|
||||
)
|
||||
def load_triviaqa_dataset(save_path):
|
||||
# set file path
|
||||
file_path = "MarkrAI/triviaqa_sample_autorag"
|
||||
|
||||
# load dataset
|
||||
corpus_dataset = load_dataset(file_path, "corpus")["train"].to_pandas()
|
||||
qa_train_dataset = load_dataset(file_path, "qa")["train"].to_pandas()
|
||||
qa_test_dataset = load_dataset(file_path, "qa")["test"].to_pandas()
|
||||
|
||||
# save corpus data
|
||||
if os.path.exists(os.path.join(save_path, "corpus.parquet")) is True:
|
||||
raise ValueError("corpus.parquet already exists")
|
||||
if os.path.exists(os.path.join(save_path, "qa.parquet")) is True:
|
||||
raise ValueError("qa.parquet already exists")
|
||||
corpus_dataset.to_parquet(os.path.join(save_path, "corpus.parquet"), index=False)
|
||||
qa_train_dataset.to_parquet(
|
||||
os.path.join(save_path, "qa_train.parquet"), index=False
|
||||
)
|
||||
qa_test_dataset.to_parquet(os.path.join(save_path, "qa_test.parquet"), index=False)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
load_triviaqa_dataset()
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user