Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
3003c1b
Add A2E + Japanese audio test suite
claude Feb 20, 2026
cdca526
Add ONNX error diagnostic and VAD handler patch tools
claude Feb 20, 2026
081f904
Add standalone A2E Japanese audio test script
claude Feb 20, 2026
35838a6
Add LLM handler patch for Gemini dict content TypeError
claude Feb 20, 2026
b50178e
Update Gemini model to gemini-2.5-flash (2.0-flash deprecated)
claude Feb 20, 2026
cbc1c7c
Add ASR language patch to force Japanese in SenseVoice
claude Feb 20, 2026
59387bf
Add config patch script to Japanize existing chat_with_lam.yaml
claude Feb 20, 2026
a58395b
Fix ASR 2nd inference 24x slowdown causing system freeze
claude Feb 20, 2026
0875af7
Add audio2exp-service microservice and frontend A2E integration
claude Feb 21, 2026
37ebe2c
Support flat model directory layout for A2E checkpoint
claude Feb 21, 2026
23f10de
Add complete A2E-integrated concierge-controller.ts replacement
claude Feb 21, 2026
2388a86
Replace concierge-controller.ts with actual gourmet-sp version
claude Feb 21, 2026
949bbae
Fix start.sh: uvicorn → gunicorn for Flask WSGI app
claude Feb 21, 2026
76bd40c
Add comprehensive system architecture documentation
claude Feb 21, 2026
cde7c54
fix: correct data format mismatch in applyExpressionFromTts for lip sync
claude Feb 21, 2026
2e09277
fix: integrate ExpressionManager for A2E lip sync (replace broken lam…
claude Feb 21, 2026
3b91d52
fix: inline ExpressionManager to eliminate external import dependency
claude Feb 21, 2026
c2a881c
fix: use LAMAvatar's buffer system instead of non-existent guavaRenderer
claude Feb 21, 2026
461ee0a
fix: add A2E chain diagnostics after analyzing actual LAMAvatar.astro…
claude Feb 21, 2026
0df04b9
fix: match original gourmet-sp behavior - remove destructive clearFra…
claude Feb 21, 2026
4332c8f
fix: prevent autoplay deadlock in all play-and-wait patterns (STT fix)
claude Feb 22, 2026
978b5d3
fix(frontend-patch): applyExpressionFromTts の両形式対応と try/catch 追加
claude Feb 22, 2026
c9cf14c
feat: rewrite A2E patch based on production code - restore GVRM integ…
claude Feb 22, 2026
8887cc2
fix: use gourmet-sp version of concierge-controller.ts (LAMAvatar, no…
claude Feb 22, 2026
65db8dc
feat: improve A2E lip sync quality - amplify mouth blendshapes + inte…
claude Feb 22, 2026
50f4e4d
fix: handle new a2e_engine response format (plain arrays vs {weights}…
claude Feb 22, 2026
c15162e
fix: use proper INFER pipeline for A2E decoder + add renderer diagnostic
claude Feb 22, 2026
8f99c70
fix: resolve INFER pipeline startup errors for A2E service
claude Feb 22, 2026
a8a68c3
chore: add .gitignore for audio2exp-service model files
claude Feb 22, 2026
e1b8d30
fix: disable Flask auto dotenv loading to avoid encoding errors
claude Feb 22, 2026
2e16f78
fix: enable TTS playback for text input in concierge mode
claude Feb 22, 2026
dc85ffd
docs: add session handoff document for next AI session
claude Feb 22, 2026
bc1fe80
test: add CI-friendly pytest suite for A2E service PoC validation
claude Feb 22, 2026
7a2ab7c
fix: add .gcloudignore to include models/ in Cloud Build context
claude Feb 22, 2026
ac3a861
docs: update .gitignore comment to reflect bake-in strategy
claude Feb 22, 2026
a9fc0b5
fix: use Cloud Run PORT env var instead of hardcoded 8081
claude Feb 22, 2026
a633fe1
fix: lazy-load A2E engine in background thread
claude Feb 22, 2026
a4eb9ee
Pre-bake wav2vec2-base-960h in Docker image to avoid runtime download
claude Feb 22, 2026
9f5bd88
Use CPU-only PyTorch + add torchaudio for INFER pipeline
claude Feb 22, 2026
e36190d
fix(audio2exp): add warmup timeout and forward-pass timing logs
claude Feb 23, 2026
2bd1198
fix(docs): correct Cloud Run region from asia-northeast1 to us-central1
claude Feb 23, 2026
81fac9b
fix(deploy): update memory to 4Gi and add CLAUDE.md project notes
claude Feb 23, 2026
528dd5f
docs: update CLAUDE.md as handover document for next session
claude Feb 23, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Project Notes - 引継ぎ文書

## 現在の状況: audio2exp-service デプロイ(進行中)

### やったこと
1. audio2exp-service を修正し、再ビルド・再デプロイを実施
2. `--memory 2Gi` ではメモリ不足で3回失敗 → `4Gi` に増やして完走
3. デプロイ完走後のヘルスチェックで **NG** → 原因調査・対処が必要

### 現在のステータス
- **デプロイ**: 完走済み(メモリ4Gi)
- **ヘルスチェック**: NG(未解決)
- **次のアクション**: ヘルスチェックNG原因の調査・修正・再デプロイ

### ルール
- 推測で回答せず、必ず会話ログ・ファイル・記録を確認してから回答すること
- 確定していない中途半端な情報を書き出さないこと
377 changes: 377 additions & 0 deletions docs/SESSION_HANDOFF.md

Large diffs are not rendered by default.

855 changes: 855 additions & 0 deletions docs/SYSTEM_ARCHITECTURE.md

Large diffs are not rendered by default.

12 changes: 12 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
markers = [
"unit: Unit tests (no external dependencies)",
"api: API contract tests (Flask test client)",
"integration: Integration tests (requires models)",
"slow: Slow tests (model loading, inference)",
]
addopts = "-v --tb=short -m 'not integration and not slow'"
271 changes: 271 additions & 0 deletions scripts/test_a2e_japanese_audio.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,271 @@
"""
日本語音声 A2E テスト - 簡易スタンドアロン版

OpenAvatarChat で data_bundle.py の修正が正しく機能するかテストします。

使い方:
cd C:\Users\hamad\OpenAvatarChat
conda activate oac
python scripts/test_a2e_japanese_audio.py

このスクリプトを C:\Users\hamad\OpenAvatarChat\scripts\ にコピーして実行してください。
"""

import sys
import os
import time
import traceback
from pathlib import Path

# OpenAvatarChatのルートディレクトリを検出
SCRIPT_DIR = Path(__file__).parent
OAC_DIR = SCRIPT_DIR.parent # scripts/ の親 = OpenAvatarChat/

def print_header(title):
print(f"\n{'='*60}")
print(f" {title}")
print(f"{'='*60}")


def test_1_environment():
"""テスト1: 環境チェック"""
print_header("TEST 1: Environment Check")
errors = []

# Python version
print(f" Python: {sys.version}")

# NumPy
try:
import numpy as np
print(f" NumPy: {np.__version__}")
except ImportError:
errors.append("NumPy not installed")

# PyTorch
try:
import torch
print(f" PyTorch: {torch.__version__}")
print(f" CUDA available: {torch.cuda.is_available()}")
except ImportError:
errors.append("PyTorch not installed")

# transformers
try:
import transformers
print(f" Transformers: {transformers.__version__}")
except ImportError:
errors.append("transformers not installed")

# onnxruntime
try:
import onnxruntime
print(f" ONNXRuntime: {onnxruntime.__version__}")
except ImportError:
print(" ONNXRuntime: not installed (optional)")

if errors:
for e in errors:
print(f" [ERROR] {e}")
return False

print(" [PASS] Environment OK")
return True


def test_2_model_files():
"""テスト2: モデルファイル存在確認"""
print_header("TEST 2: Model Files Check")

checks = {
"LAM_audio2exp dir": OAC_DIR / "models" / "LAM_audio2exp",
"wav2vec2-base-960h dir": OAC_DIR / "models" / "wav2vec2-base-960h",
"pretrained_models dir": OAC_DIR / "models" / "LAM_audio2exp" / "pretrained_models",
}

all_ok = True
for label, path in checks.items():
exists = path.exists()
status = "OK" if exists else "MISSING"
print(f" [{status}] {label}: {path}")
if not exists:
all_ok = False

if all_ok:
print(" [PASS] All model directories found")
else:
print(" [FAIL] Some model files missing")
return all_ok


def test_3_data_bundle_fix():
"""テスト3: data_bundle.py の list/tuple → ndarray 変換テスト"""
print_header("TEST 3: data_bundle.py Fix Verification")

try:
import numpy as np

# data_bundle.py のパスを確認
db_path = OAC_DIR / "src" / "chat_engine" / "data_models" / "runtime_data" / "data_bundle.py"
if not db_path.exists():
print(f" [SKIP] File not found: {db_path}")
return True # ファイルがなければスキップ

# ファイル内容をチェック
content = db_path.read_text(encoding="utf-8")
if "isinstance(data, (list, tuple))" in content:
print(" [OK] list/tuple conversion patch found in data_bundle.py")
else:
print(" [WARN] list/tuple conversion patch NOT found in data_bundle.py")
print(" Add this before 'if isinstance(data, np.ndarray)'::")
print(" if isinstance(data, (list, tuple)):")
print(" data = np.array(data, dtype=np.float32)")
return False

# 実際に変換が動作するかテスト
test_list = [0.1, 0.2, 0.3, 0.4, 0.5]
test_tuple = (0.1, 0.2, 0.3)
arr_from_list = np.array(test_list, dtype=np.float32)
arr_from_tuple = np.array(test_tuple, dtype=np.float32)

assert isinstance(arr_from_list, np.ndarray), "list→ndarray conversion failed"
assert isinstance(arr_from_tuple, np.ndarray), "tuple→ndarray conversion failed"
assert arr_from_list.dtype == np.float32, "dtype should be float32"
print(f" [OK] list→ndarray: {test_list} → shape={arr_from_list.shape}")
print(f" [OK] tuple→ndarray: {test_tuple} → shape={arr_from_tuple.shape}")

print(" [PASS] data_bundle.py fix is correct")
return True

except Exception as e:
print(f" [FAIL] {e}")
traceback.print_exc()
return False


def test_4_wav2vec2_load():
"""テスト4: Wav2Vec2モデルの読み込みテスト"""
print_header("TEST 4: Wav2Vec2 Model Loading")

try:
import torch
from transformers import Wav2Vec2Model, Wav2Vec2Processor
import numpy as np

wav2vec_dir = OAC_DIR / "models" / "wav2vec2-base-960h"
if wav2vec_dir.exists() and (wav2vec_dir / "config.json").exists():
model_path = str(wav2vec_dir)
print(f" Loading from local: {model_path}")
else:
model_path = "facebook/wav2vec2-base-960h"
print(f" Loading from HuggingFace: {model_path}")

t0 = time.time()
model = Wav2Vec2Model.from_pretrained(model_path)
model.eval()
elapsed = time.time() - t0
print(f" Model loaded in {elapsed:.1f}s")

# ダミー音声でテスト (1秒の無音)
dummy_audio = np.zeros(16000, dtype=np.float32)
try:
processor = Wav2Vec2Processor.from_pretrained(model_path)
except Exception:
processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-base-960h")

inputs = processor(dummy_audio, sampling_rate=16000, return_tensors="pt", padding=True)
with torch.no_grad():
outputs = model(**inputs)

features = outputs.last_hidden_state
print(f" Output shape: {tuple(features.shape)}")
print(f" [PASS] Wav2Vec2 working correctly")
return True

except Exception as e:
print(f" [FAIL] {e}")
traceback.print_exc()
return False


def test_5_a2e_import():
"""テスト5: A2Eモジュールのインポートテスト"""
print_header("TEST 5: A2E Module Import")

# sys.pathにOpenAvatarChatのパスを追加
paths_to_add = [
str(OAC_DIR / "src"),
str(OAC_DIR / "src" / "handlers"),
str(OAC_DIR / "src" / "handlers" / "avatar" / "lam"),
str(OAC_DIR / "src" / "handlers" / "avatar" / "lam" / "LAM_Audio2Expression"),
]
for p in paths_to_add:
if p not in sys.path and os.path.exists(p):
sys.path.insert(0, p)

imported = False

# 方法1: A2E直接インポート
try:
from LAM_Audio2Expression.engines.infer import Audio2ExpressionInfer
print(" [OK] A2E infer module imported")
imported = True
except ImportError as e:
print(f" [INFO] Direct A2E import failed: {e}")

# 方法2: handler経由
if not imported:
try:
from avatar.lam.avatar_handler_lam_audio2expression import HandlerAvatarLAM
print(" [OK] A2E handler module imported")
imported = True
except ImportError as e:
print(f" [INFO] Handler import failed: {e}")

if imported:
print(" [PASS] A2E module is importable")
else:
print(" [WARN] A2E module not importable (may need specific env)")
print(" This is OK if other tests pass")

return True # インポート失敗でも致命的ではない


def main():
print("=" * 60)
print(" A2E Japanese Audio Test - Standalone")
print(f" OAC Dir: {OAC_DIR}")
print(f" Time: {time.strftime('%Y-%m-%d %H:%M:%S')}")
print("=" * 60)

results = {}
results["environment"] = test_1_environment()
results["model_files"] = test_2_model_files()
results["data_bundle_fix"] = test_3_data_bundle_fix()
results["wav2vec2"] = test_4_wav2vec2_load()
results["a2e_import"] = test_5_a2e_import()

# サマリー
print_header("SUMMARY")
passed = 0
total = len(results)
for name, ok in results.items():
status = "PASS" if ok else "FAIL"
print(f" [{status}] {name}")
if ok:
passed += 1

print(f"\n Result: {passed}/{total} passed")

if passed == total:
print("\n All tests passed!")
print(" Next step: Start OpenAvatarChat and test with Japanese voice:")
print(" python src/demo.py --config config/chat_with_lam_jp.yaml")
else:
print("\n Some tests failed. Fix the issues above and re-run.")

return 0 if passed == total else 1


if __name__ == "__main__":
sys.exit(main())
Loading