344 lines
8.8 KiB
Markdown
344 lines
8.8 KiB
Markdown
# Claudian Token Optimization - Testing Guide
|
|
|
|
## Implementation Summary
|
|
|
|
**Date**: 2026-01-08
|
|
**Phase**: Phase 1 - Quick Wins
|
|
**Status**: ✅ Code changes completed, awaiting user testing
|
|
|
|
---
|
|
|
|
## Changes Made
|
|
|
|
### 1. System Prompt Compression (~800 tokens saved)
|
|
|
|
| Section | Before | After | Savings |
|
|
|---------|--------|-------|---------|
|
|
| Identity & Role | 7 lines verbose | 2 lines concise | ~150 tokens |
|
|
| Path Rules | 9 lines with examples | 1 line | ~100 tokens |
|
|
| Message Format | 14 lines with explanations | 3 lines examples only | ~80 tokens |
|
|
| Obsidian Context | 7 lines detailed | 2 bullet points | ~70 tokens |
|
|
| Tool Guidelines | 95 lines verbose | 6 bullet points | ~400 tokens |
|
|
|
|
**Total System Prompt**: Reduced from ~2,400 tokens → ~1,200 tokens (50% reduction)
|
|
|
|
### 2. History Windowing (10,000-25,000 tokens saved)
|
|
|
|
**Location**: `buildContextFromHistory()` function (line 22465)
|
|
|
|
**Logic**:
|
|
- Limit to **15 most recent messages**
|
|
- Always preserve **first message** (context establishment)
|
|
- Add truncation notice when messages omitted
|
|
|
|
**Example**:
|
|
```
|
|
100 message conversation:
|
|
Before: All 100 messages sent (~40,000 tokens)
|
|
After: First + last 15 messages (16 total, ~6,500 tokens)
|
|
Savings: 33,500 tokens (84% reduction)
|
|
```
|
|
|
|
### 3. Tool Result Truncation (300-500 tokens saved)
|
|
|
|
**Location**: `truncateToolResult()` function (line 22453)
|
|
|
|
**Change**: Max length 800 → 300 characters
|
|
|
|
**Impact**: Tool results in history rebuilds are more aggressive truncated
|
|
|
|
### 4. Optional Sections Removed (600-800 tokens saved)
|
|
|
|
**Location**: `buildSystemPrompt()` function (line 23471)
|
|
|
|
**Commented out**:
|
|
- Image instructions (~400 tokens) - line 23474
|
|
- Editor context instructions (~150 tokens) - lines 23480-23482
|
|
|
|
---
|
|
|
|
## Testing Instructions
|
|
|
|
### Step 1: Reload Plugin in Obsidian
|
|
|
|
1. Open Obsidian
|
|
2. Navigate to **Settings → Community Plugins**
|
|
3. Find **Claudian** plugin
|
|
4. Click **Disable** (wait 2-3 seconds)
|
|
5. Click **Enable**
|
|
6. Close Settings
|
|
|
|
**Expected**: Plugin enables without errors
|
|
|
|
### Step 2: Basic Functionality Test
|
|
|
|
**Test 1: New Conversation**
|
|
1. Start a new Claudian conversation
|
|
2. Send a simple message: "List files in the vault root"
|
|
3. Verify Claudian responds normally
|
|
|
|
**Expected**: Response should be coherent and functional
|
|
|
|
**Test 2: File Operations**
|
|
1. Ask: "Read the file 00_Inbox/2026-01-07.md"
|
|
2. Verify file is read correctly
|
|
3. Ask: "What's in this note?"
|
|
|
|
**Expected**: Claudian can read files and understand content
|
|
|
|
**Test 3: Existing Conversation**
|
|
1. Open an existing conversation (preferably 20+ messages)
|
|
2. Send a new message
|
|
3. Verify response is coherent
|
|
|
|
**Expected**: Plugin handles existing conversations correctly
|
|
|
|
### Step 3: Token Usage Verification
|
|
|
|
**Check Console for Token Info**:
|
|
1. Press **Ctrl+Shift+I** (or Cmd+Option+I on Mac) to open DevTools
|
|
2. Click **Console** tab
|
|
3. Send a message in Claudian
|
|
4. Look for token usage info in console output
|
|
|
|
**What to Look For**:
|
|
- `inputTokens`: Should be significantly lower
|
|
- `contextTokens`: Total tokens used
|
|
- Compare with previous conversations (if you remember typical values)
|
|
|
|
**Expected Reductions**:
|
|
- Short conversations (5 msgs): ~1,800 tokens total
|
|
- Medium conversations (20-30 msgs): ~6,500 tokens total
|
|
- Long conversations (50+ msgs): ~8,000-10,000 tokens total
|
|
|
|
### Step 4: Quality Checks
|
|
|
|
**Test Understanding of Obsidian Context**:
|
|
1. Ask: "Create a note in 00_Inbox with today's date"
|
|
2. Ask: "Add a wiki-link to another note"
|
|
3. Verify Claudian still understands Obsidian concepts
|
|
|
|
**Expected**: Should still understand:
|
|
- Markdown formatting
|
|
- Wiki-links [[note]]
|
|
- Vault structure
|
|
- Relative paths
|
|
|
|
**Test Tool Usage**:
|
|
1. Ask: "Find all markdown files with the word 'TODO'"
|
|
2. Ask: "Read the first result"
|
|
3. Ask: "Update it to remove the TODO"
|
|
|
|
**Expected**: Should correctly use Read, Grep, Edit tools
|
|
|
|
### Step 5: Edge Cases
|
|
|
|
**Test 1: Long Conversation with Session Break**
|
|
1. Open a conversation with 30+ messages
|
|
2. Send a message that requires context from early messages
|
|
3. Observe if Claudian maintains enough context
|
|
|
|
**Expected**: May not remember details from messages 16-29 (that were windowed out), but should handle gracefully with truncation notice
|
|
|
|
**Test 2: Current Note Context**
|
|
1. Open a note in Obsidian
|
|
2. Open Claudian sidebar
|
|
3. Send a message referencing "this note"
|
|
|
|
**Expected**: Should still understand current note context
|
|
|
|
---
|
|
|
|
## Success Criteria
|
|
|
|
### ✅ Must Pass
|
|
- [ ] Plugin loads without errors
|
|
- [ ] Can start new conversations
|
|
- [ ] Can send/receive messages
|
|
- [ ] File operations work (Read, Write, Edit)
|
|
- [ ] Tool calls execute correctly
|
|
- [ ] Existing conversations load properly
|
|
- [ ] Token usage is reduced (check console)
|
|
|
|
### ✅ Should Pass
|
|
- [ ] Response quality remains high
|
|
- [ ] Understands Obsidian concepts
|
|
- [ ] Relative paths work correctly
|
|
- [ ] No functionality lost
|
|
|
|
### ⚠️ Acceptable Trade-offs
|
|
- [ ] May forget context from messages 16+ back (windowed out)
|
|
- [ ] Less verbose explanations
|
|
- [ ] Slightly less hand-holding in responses
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Issue: Plugin Won't Load
|
|
|
|
**Symptoms**: Error on enable, plugin stays disabled
|
|
|
|
**Fix**:
|
|
```bash
|
|
# Restore backup
|
|
cd /path/to/vault
|
|
cp .obsidian/plugins/claudian/main.js.backup .obsidian/plugins/claudian/main.js
|
|
```
|
|
Then reload Obsidian
|
|
|
|
### Issue: Errors in Console
|
|
|
|
**Symptoms**: JavaScript errors in console after sending message
|
|
|
|
**What to do**:
|
|
1. Copy the full error message
|
|
2. Check which function is failing
|
|
3. Restore backup if critical
|
|
|
|
### Issue: Poor Response Quality
|
|
|
|
**Symptoms**: Claudian doesn't understand Obsidian concepts, paths are wrong
|
|
|
|
**Possible causes**:
|
|
- System prompt too compressed
|
|
- Missing essential instructions
|
|
|
|
**Fix**: Can restore specific sections (e.g., uncomment image instructions if needed)
|
|
|
|
### Issue: Still Using Too Many Tokens
|
|
|
|
**Symptoms**: Token usage not significantly reduced
|
|
|
|
**Check**:
|
|
1. Is session resumption working? (Should use persistent sessions normally)
|
|
2. Are you testing with session breaks? (History windowing only applies when session breaks)
|
|
3. Check if custom instructions in settings add lots of tokens
|
|
|
|
---
|
|
|
|
## Rollback Procedure
|
|
|
|
If anything breaks:
|
|
|
|
1. **Full Rollback**:
|
|
```bash
|
|
cp .obsidian/plugins/claudian/main.js.backup .obsidian/plugins/claudian/main.js
|
|
```
|
|
|
|
2. **Reload Plugin**:
|
|
- Settings → Community Plugins
|
|
- Disable Claudian
|
|
- Enable Claudian
|
|
|
|
3. **Verify**: Test that original version works
|
|
|
|
---
|
|
|
|
## Measuring Token Reduction
|
|
|
|
### Before Optimization (Typical Values)
|
|
|
|
**System Prompt**: ~3,200-6,500 tokens
|
|
**History (30 msg conversation with session break)**: ~15,000-18,000 tokens
|
|
**Total**: ~18,000-24,000 tokens per request
|
|
|
|
### After Optimization (Expected Values)
|
|
|
|
**System Prompt**: ~1,600-2,000 tokens
|
|
**History (30 msg conversation with session break)**: ~6,000-8,000 tokens
|
|
**Total**: ~7,600-10,000 tokens per request
|
|
|
|
**Overall Reduction**: ~60-70% for medium/long conversations
|
|
|
|
---
|
|
|
|
## Phase 2 Preview (Future Enhancements)
|
|
|
|
If Phase 1 is successful, Phase 2 will add:
|
|
|
|
1. **User Settings**:
|
|
- Configurable history window size (15-30 messages)
|
|
- Tool result max length setting
|
|
- Toggle image/editor instructions
|
|
|
|
2. **Smart Message Selection**:
|
|
- Preserve messages with tool calls
|
|
- Preserve longer/important messages
|
|
- Dynamic window sizing
|
|
|
|
3. **Token Warnings**:
|
|
- Visual warning at 50% context usage
|
|
- Auto-suggest starting new conversation at 80%
|
|
|
|
4. **Token Usage UI**:
|
|
- Show breakdown: System | History | Current
|
|
- Display in conversation header
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
- All changes are backwards compatible
|
|
- JSONL conversation files unchanged
|
|
- Session management logic unchanged
|
|
- Can selectively restore sections if needed
|
|
- Backup file preserved at: `.obsidian/plugins/claudian/main.js.backup`
|
|
|
|
---
|
|
|
|
## Testing Checklist
|
|
|
|
Print this section and check off as you test:
|
|
|
|
```
|
|
Basic Tests:
|
|
[ ] Plugin loads without errors
|
|
[ ] New conversation works
|
|
[ ] Can read files
|
|
[ ] Can write files
|
|
[ ] Existing conversation loads
|
|
[ ] Token usage reduced (check console)
|
|
|
|
Quality Tests:
|
|
[ ] Understands Markdown
|
|
[ ] Understands Wiki-links
|
|
[ ] Uses relative paths correctly
|
|
[ ] Tool calls work
|
|
[ ] Response quality good
|
|
|
|
Edge Cases:
|
|
[ ] Long conversation (30+ msgs)
|
|
[ ] Session break handling
|
|
[ ] Current note context
|
|
[ ] File operations in subfolders
|
|
|
|
If all checked: ✅ Phase 1 Complete!
|
|
If issues found: Document below and consider rollback/adjustments
|
|
```
|
|
|
|
---
|
|
|
|
## Feedback Section
|
|
|
|
**Date Tested**: _____________
|
|
|
|
**Token Reduction Observed**: _______% (compare console before/after)
|
|
|
|
**Issues Found**:
|
|
-
|
|
-
|
|
-
|
|
|
|
**Quality Assessment**:
|
|
- Response coherence: ___/10
|
|
- Obsidian understanding: ___/10
|
|
- Tool usage: ___/10
|
|
- Overall satisfaction: ___/10
|
|
|
|
**Recommendation**:
|
|
- [ ] Keep Phase 1 changes, proceed to Phase 2
|
|
- [ ] Keep changes, adjust window size to: ___
|
|
- [ ] Restore specific section: ____________
|
|
- [ ] Full rollback needed
|