nbrightandClaude Opus 5
e529c449b5
Six papers converted to Markdown with the local doc2md tool, figures extracted and annotated. The tool's venv had a CPU-only torch, so marker-pdf silently ran on CPU and stalled; swapping in 2.5.1+cu121 dropped a paper from "hung after six minutes" to three. Gemini then described all 86 figures in place, below each original caption. The PDFs themselves are gitignored - 88 MB of public arXiv downloads that convert_papers.sh regenerates. The .md and figures are tracked, because the annotations took a separate pass and do not reproduce byte-for-byte. sum-parts-explained.html gains two tabs: - PointVector. Why representing a scalar feature as a rotated 3D vector buys anisotropic aggregation without attention's cost, and why the paper predicts two independent angles rather than a rotation matrix whose nine elements are interdependent. - Bare Earth. Reframes the task as ground vs not-ground, and separates the five boundaries by their nature. Four of them are cuts; the slope boundary is the one that must NOT be cut, which is why "horizontal means ground" destroys road cut and fill. Notes that SUM Parts is flat Helsinki and cannot teach slopes at all, so that part needs a geometric filter rather than more training. NEXT.md carries the goal forward: separate bare earth from the rest as OBJ meshes, then reclassify the remainder. Removing the ground first is sound - it is 24-40% of the points, and without it the remaining objects fall apart into separate connected components instead of being joined through the floor. The gap that blocks step 4 is named: mesh_to_ply.py samples points without recording which face each came from, so there is no way back to the mesh yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
88 lines
2.8 KiB
Bash
88 lines
2.8 KiB
Bash
#!/usr/bin/env bash
|
|
# Convert the reference papers to Markdown with the local doc2md tool
|
|
#
|
|
# Two things had to line up to make this work:
|
|
#
|
|
# 1. Call it from bash, not PowerShell. PowerShell prepends a BOM when piping
|
|
# to a native process, has no `<` redirection, and PS 5.1 lacks
|
|
# StandardInputEncoding to turn the BOM off. (main.py now reads
|
|
# utf-8-sig, so the BOM is tolerated - but bash avoids the issue entirely.)
|
|
#
|
|
# 2. The tool's venv had a CPU-only torch (2.12.0+cpu). marker-pdf picks its
|
|
# device from torch.cuda.is_available(), so it silently ran on CPU and
|
|
# stalled. Replaced with 2.5.1+cu121; no code change was needed.
|
|
#
|
|
# Runs one paper at a time - they share a single GPU.
|
|
set -uo pipefail
|
|
|
|
TOOL="/d/Teknom/jjangoo/tools/doc.convert.doc2md"
|
|
PAPERS="/d/MYCLAUDE_PROJECT/sum-parts-test/docs/papers"
|
|
OUT="$PAPERS/md"
|
|
WIN_PAPERS="D:/MYCLAUDE_PROJECT/sum-parts-test/docs/papers"
|
|
WIN_OUT="D:/MYCLAUDE_PROJECT/sum-parts-test/docs/papers/md"
|
|
|
|
mkdir -p "$OUT"
|
|
cd "$TOOL" || exit 1
|
|
|
|
for pdf in "$PAPERS"/*.pdf; do
|
|
name=$(basename "$pdf" .pdf)
|
|
|
|
if [ -f "$OUT/$name.md" ]; then
|
|
echo "=== $name (이미 변환됨, 건너뜀) ==="
|
|
continue
|
|
fi
|
|
|
|
echo "=== $name ==="
|
|
printf '%s' "{\"file\":\"$WIN_PAPERS/$name.pdf\",\"outputDir\":\"$WIN_OUT\"}" > /tmp/doc2md_req.json
|
|
|
|
start=$(date +%s)
|
|
TORCH_DEVICE=cuda ./.venv/Scripts/python.exe main.py < /tmp/doc2md_req.json \
|
|
> "/tmp/doc2md_$name.out" 2>"/tmp/doc2md_$name.err"
|
|
rc=$?
|
|
elapsed=$(( $(date +%s) - start ))
|
|
|
|
if [ $rc -ne 0 ]; then
|
|
echo " FAILED rc=$rc (${elapsed}s)"
|
|
tail -5 "/tmp/doc2md_$name.err"
|
|
continue
|
|
fi
|
|
|
|
# the result line is JSON on stdout
|
|
python3 - "/tmp/doc2md_$name.out" "$elapsed" <<'PY'
|
|
import json, sys
|
|
from pathlib import Path
|
|
|
|
txt = Path(sys.argv[1]).read_text(encoding="utf-8", errors="replace")
|
|
elapsed = sys.argv[2]
|
|
for line in txt.splitlines():
|
|
line = line.strip()
|
|
if not line.startswith("{"):
|
|
continue
|
|
try:
|
|
d = json.loads(line)
|
|
except Exception:
|
|
continue
|
|
if d.get("type") == "result":
|
|
o = d["output"]
|
|
md = Path(o["file"])
|
|
size = md.stat().st_size / 1024 if md.exists() else 0
|
|
print(f" ok {elapsed}s {size:,.0f} KB "
|
|
f"images={len(o.get('images') or [])} "
|
|
f"pages={len(o.get('pages') or [])} "
|
|
f"diagrams={o.get('hasDiagrams')}")
|
|
elif d.get("type") == "error":
|
|
print(f" ERROR {d.get('code')}: {d.get('message')}")
|
|
PY
|
|
done
|
|
|
|
echo
|
|
echo "=== 결과 ==="
|
|
for md in "$OUT"/*.md; do
|
|
[ -f "$md" ] || continue
|
|
n=$(basename "$md")
|
|
kb=$(( $(stat -c%s "$md") / 1024 ))
|
|
heads=$(grep -cE '^#{1,3} ' "$md")
|
|
rows=$(grep -c '^|' "$md")
|
|
echo " $(printf '%-34s' "$n") ${kb}KB 헤딩 ${heads} 표행 ${rows}"
|
|
done
|