vault backup: 2026-01-08 09:34:22
This commit is contained in:
windyboy
@@ -0,0 +1,343 @@
|
||||
# Claudian Token Optimization - Testing Guide
|
||||
|
||||
## Implementation Summary
|
||||
|
||||
**Date**: 2026-01-08
|
||||
**Phase**: Phase 1 - Quick Wins
|
||||
**Status**: ✅ Code changes completed, awaiting user testing
|
||||
|
||||
---
|
||||
|
||||
## Changes Made
|
||||
|
||||
### 1. System Prompt Compression (~800 tokens saved)
|
||||
|
||||
| Section | Before | After | Savings |
|
||||
|---------|--------|-------|---------|
|
||||
| Identity & Role | 7 lines verbose | 2 lines concise | ~150 tokens |
|
||||
| Path Rules | 9 lines with examples | 1 line | ~100 tokens |
|
||||
| Message Format | 14 lines with explanations | 3 lines examples only | ~80 tokens |
|
||||
| Obsidian Context | 7 lines detailed | 2 bullet points | ~70 tokens |
|
||||
| Tool Guidelines | 95 lines verbose | 6 bullet points | ~400 tokens |
|
||||
|
||||
**Total System Prompt**: Reduced from ~2,400 tokens → ~1,200 tokens (50% reduction)
|
||||
|
||||
### 2. History Windowing (10,000-25,000 tokens saved)
|
||||
|
||||
**Location**: `buildContextFromHistory()` function (line 22465)
|
||||
|
||||
**Logic**:
|
||||
- Limit to **15 most recent messages**
|
||||
- Always preserve **first message** (context establishment)
|
||||
- Add truncation notice when messages omitted
|
||||
|
||||
**Example**:
|
||||
```
|
||||
100 message conversation:
|
||||
Before: All 100 messages sent (~40,000 tokens)
|
||||
After: First + last 15 messages (16 total, ~6,500 tokens)
|
||||
Savings: 33,500 tokens (84% reduction)
|
||||
```
|
||||
|
||||
### 3. Tool Result Truncation (300-500 tokens saved)
|
||||
|
||||
**Location**: `truncateToolResult()` function (line 22453)
|
||||
|
||||
**Change**: Max length 800 → 300 characters
|
||||
|
||||
**Impact**: Tool results in history rebuilds are more aggressive truncated
|
||||
|
||||
### 4. Optional Sections Removed (600-800 tokens saved)
|
||||
|
||||
**Location**: `buildSystemPrompt()` function (line 23471)
|
||||
|
||||
**Commented out**:
|
||||
- Image instructions (~400 tokens) - line 23474
|
||||
- Editor context instructions (~150 tokens) - lines 23480-23482
|
||||
|
||||
---
|
||||
|
||||
## Testing Instructions
|
||||
|
||||
### Step 1: Reload Plugin in Obsidian
|
||||
|
||||
1. Open Obsidian
|
||||
2. Navigate to **Settings → Community Plugins**
|
||||
3. Find **Claudian** plugin
|
||||
4. Click **Disable** (wait 2-3 seconds)
|
||||
5. Click **Enable**
|
||||
6. Close Settings
|
||||
|
||||
**Expected**: Plugin enables without errors
|
||||
|
||||
### Step 2: Basic Functionality Test
|
||||
|
||||
**Test 1: New Conversation**
|
||||
1. Start a new Claudian conversation
|
||||
2. Send a simple message: "List files in the vault root"
|
||||
3. Verify Claudian responds normally
|
||||
|
||||
**Expected**: Response should be coherent and functional
|
||||
|
||||
**Test 2: File Operations**
|
||||
1. Ask: "Read the file 00_Inbox/2026-01-07.md"
|
||||
2. Verify file is read correctly
|
||||
3. Ask: "What's in this note?"
|
||||
|
||||
**Expected**: Claudian can read files and understand content
|
||||
|
||||
**Test 3: Existing Conversation**
|
||||
1. Open an existing conversation (preferably 20+ messages)
|
||||
2. Send a new message
|
||||
3. Verify response is coherent
|
||||
|
||||
**Expected**: Plugin handles existing conversations correctly
|
||||
|
||||
### Step 3: Token Usage Verification
|
||||
|
||||
**Check Console for Token Info**:
|
||||
1. Press **Ctrl+Shift+I** (or Cmd+Option+I on Mac) to open DevTools
|
||||
2. Click **Console** tab
|
||||
3. Send a message in Claudian
|
||||
4. Look for token usage info in console output
|
||||
|
||||
**What to Look For**:
|
||||
- `inputTokens`: Should be significantly lower
|
||||
- `contextTokens`: Total tokens used
|
||||
- Compare with previous conversations (if you remember typical values)
|
||||
|
||||
**Expected Reductions**:
|
||||
- Short conversations (5 msgs): ~1,800 tokens total
|
||||
- Medium conversations (20-30 msgs): ~6,500 tokens total
|
||||
- Long conversations (50+ msgs): ~8,000-10,000 tokens total
|
||||
|
||||
### Step 4: Quality Checks
|
||||
|
||||
**Test Understanding of Obsidian Context**:
|
||||
1. Ask: "Create a note in 00_Inbox with today's date"
|
||||
2. Ask: "Add a wiki-link to another note"
|
||||
3. Verify Claudian still understands Obsidian concepts
|
||||
|
||||
**Expected**: Should still understand:
|
||||
- Markdown formatting
|
||||
- Wiki-links [[note]]
|
||||
- Vault structure
|
||||
- Relative paths
|
||||
|
||||
**Test Tool Usage**:
|
||||
1. Ask: "Find all markdown files with the word 'TODO'"
|
||||
2. Ask: "Read the first result"
|
||||
3. Ask: "Update it to remove the TODO"
|
||||
|
||||
**Expected**: Should correctly use Read, Grep, Edit tools
|
||||
|
||||
### Step 5: Edge Cases
|
||||
|
||||
**Test 1: Long Conversation with Session Break**
|
||||
1. Open a conversation with 30+ messages
|
||||
2. Send a message that requires context from early messages
|
||||
3. Observe if Claudian maintains enough context
|
||||
|
||||
**Expected**: May not remember details from messages 16-29 (that were windowed out), but should handle gracefully with truncation notice
|
||||
|
||||
**Test 2: Current Note Context**
|
||||
1. Open a note in Obsidian
|
||||
2. Open Claudian sidebar
|
||||
3. Send a message referencing "this note"
|
||||
|
||||
**Expected**: Should still understand current note context
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
### ✅ Must Pass
|
||||
- [ ] Plugin loads without errors
|
||||
- [ ] Can start new conversations
|
||||
- [ ] Can send/receive messages
|
||||
- [ ] File operations work (Read, Write, Edit)
|
||||
- [ ] Tool calls execute correctly
|
||||
- [ ] Existing conversations load properly
|
||||
- [ ] Token usage is reduced (check console)
|
||||
|
||||
### ✅ Should Pass
|
||||
- [ ] Response quality remains high
|
||||
- [ ] Understands Obsidian concepts
|
||||
- [ ] Relative paths work correctly
|
||||
- [ ] No functionality lost
|
||||
|
||||
### ⚠️ Acceptable Trade-offs
|
||||
- [ ] May forget context from messages 16+ back (windowed out)
|
||||
- [ ] Less verbose explanations
|
||||
- [ ] Slightly less hand-holding in responses
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Plugin Won't Load
|
||||
|
||||
**Symptoms**: Error on enable, plugin stays disabled
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Restore backup
|
||||
cd /path/to/vault
|
||||
cp .obsidian/plugins/claudian/main.js.backup .obsidian/plugins/claudian/main.js
|
||||
```
|
||||
Then reload Obsidian
|
||||
|
||||
### Issue: Errors in Console
|
||||
|
||||
**Symptoms**: JavaScript errors in console after sending message
|
||||
|
||||
**What to do**:
|
||||
1. Copy the full error message
|
||||
2. Check which function is failing
|
||||
3. Restore backup if critical
|
||||
|
||||
### Issue: Poor Response Quality
|
||||
|
||||
**Symptoms**: Claudian doesn't understand Obsidian concepts, paths are wrong
|
||||
|
||||
**Possible causes**:
|
||||
- System prompt too compressed
|
||||
- Missing essential instructions
|
||||
|
||||
**Fix**: Can restore specific sections (e.g., uncomment image instructions if needed)
|
||||
|
||||
### Issue: Still Using Too Many Tokens
|
||||
|
||||
**Symptoms**: Token usage not significantly reduced
|
||||
|
||||
**Check**:
|
||||
1. Is session resumption working? (Should use persistent sessions normally)
|
||||
2. Are you testing with session breaks? (History windowing only applies when session breaks)
|
||||
3. Check if custom instructions in settings add lots of tokens
|
||||
|
||||
---
|
||||
|
||||
## Rollback Procedure
|
||||
|
||||
If anything breaks:
|
||||
|
||||
1. **Full Rollback**:
|
||||
```bash
|
||||
cp .obsidian/plugins/claudian/main.js.backup .obsidian/plugins/claudian/main.js
|
||||
```
|
||||
|
||||
2. **Reload Plugin**:
|
||||
- Settings → Community Plugins
|
||||
- Disable Claudian
|
||||
- Enable Claudian
|
||||
|
||||
3. **Verify**: Test that original version works
|
||||
|
||||
---
|
||||
|
||||
## Measuring Token Reduction
|
||||
|
||||
### Before Optimization (Typical Values)
|
||||
|
||||
**System Prompt**: ~3,200-6,500 tokens
|
||||
**History (30 msg conversation with session break)**: ~15,000-18,000 tokens
|
||||
**Total**: ~18,000-24,000 tokens per request
|
||||
|
||||
### After Optimization (Expected Values)
|
||||
|
||||
**System Prompt**: ~1,600-2,000 tokens
|
||||
**History (30 msg conversation with session break)**: ~6,000-8,000 tokens
|
||||
**Total**: ~7,600-10,000 tokens per request
|
||||
|
||||
**Overall Reduction**: ~60-70% for medium/long conversations
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 Preview (Future Enhancements)
|
||||
|
||||
If Phase 1 is successful, Phase 2 will add:
|
||||
|
||||
1. **User Settings**:
|
||||
- Configurable history window size (15-30 messages)
|
||||
- Tool result max length setting
|
||||
- Toggle image/editor instructions
|
||||
|
||||
2. **Smart Message Selection**:
|
||||
- Preserve messages with tool calls
|
||||
- Preserve longer/important messages
|
||||
- Dynamic window sizing
|
||||
|
||||
3. **Token Warnings**:
|
||||
- Visual warning at 50% context usage
|
||||
- Auto-suggest starting new conversation at 80%
|
||||
|
||||
4. **Token Usage UI**:
|
||||
- Show breakdown: System | History | Current
|
||||
- Display in conversation header
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- All changes are backwards compatible
|
||||
- JSONL conversation files unchanged
|
||||
- Session management logic unchanged
|
||||
- Can selectively restore sections if needed
|
||||
- Backup file preserved at: `.obsidian/plugins/claudian/main.js.backup`
|
||||
|
||||
---
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
Print this section and check off as you test:
|
||||
|
||||
```
|
||||
Basic Tests:
|
||||
[ ] Plugin loads without errors
|
||||
[ ] New conversation works
|
||||
[ ] Can read files
|
||||
[ ] Can write files
|
||||
[ ] Existing conversation loads
|
||||
[ ] Token usage reduced (check console)
|
||||
|
||||
Quality Tests:
|
||||
[ ] Understands Markdown
|
||||
[ ] Understands Wiki-links
|
||||
[ ] Uses relative paths correctly
|
||||
[ ] Tool calls work
|
||||
[ ] Response quality good
|
||||
|
||||
Edge Cases:
|
||||
[ ] Long conversation (30+ msgs)
|
||||
[ ] Session break handling
|
||||
[ ] Current note context
|
||||
[ ] File operations in subfolders
|
||||
|
||||
If all checked: ✅ Phase 1 Complete!
|
||||
If issues found: Document below and consider rollback/adjustments
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Feedback Section
|
||||
|
||||
**Date Tested**: _____________
|
||||
|
||||
**Token Reduction Observed**: _______% (compare console before/after)
|
||||
|
||||
**Issues Found**:
|
||||
-
|
||||
-
|
||||
-
|
||||
|
||||
**Quality Assessment**:
|
||||
- Response coherence: ___/10
|
||||
- Obsidian understanding: ___/10
|
||||
- Tool usage: ___/10
|
||||
- Overall satisfaction: ___/10
|
||||
|
||||
**Recommendation**:
|
||||
- [ ] Keep Phase 1 changes, proceed to Phase 2
|
||||
- [ ] Keep changes, adjust window size to: ___
|
||||
- [ ] Restore specific section: ____________
|
||||
- [ ] Full rollback needed
|
||||
Reference in New Issue
Block a user