Files
go-caatsm/internal/adapter/parser/aviation/tokenizer.go
T
windyboyandClaude Sonnet 4.5 c0a66cf845 Enhance aviation parser with security fixes and comprehensive refactoring
This commit implements a complete refactoring of the ICAO aviation parser,
addressing 15 identified issues across security, performance, code quality,
and documentation.

Security Enhancements (P0 - Critical):
- Add input size validation (max 1800 chars per AFTN standard)
- Implement ReDoS protection with 100ms regex timeout mechanism
- Add field validation to prevent nil pointer dereferences
- Document intentional error handling pattern for audit compliance

Performance & Design Improvements (P1 - Important):
- Remove unnecessary mutex from BodyParser (eliminates serialization)
- Fix tokenizer slash handling logic
- Remove global logger dependencies (zap.S() calls)

Code Quality Improvements (P2):
- Refactor parseRemainingLines with clear helper functions
- Document all regex patterns with ICAO format specifications
- Replace magic numbers with named constants (5 new constants)
- Add error message sanitization to prevent data leakage

Documentation & Polish (P3):
- Create comprehensive package documentation (doc.go)
- Verify naming consistency across all functions
- Add 54 comprehensive tests (all passing)
- Verify performance with benchmarks (~10µs for simple messages)

New Files:
- validation.go: Input validation utilities with AFTN limits
- validation_test.go: Comprehensive validation tests
- regex_timeout.go: ReDoS protection mechanism
- regex_timeout_test.go: Timeout protection tests
- suite_test.go: Ginkgo test suite registration
- doc.go: Package-level documentation

All changes maintain backward compatibility and existing architecture
while significantly enhancing security, maintainability, and code quality.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-26 17:55:25 +08:00

56 lines
1019 B
Go

package aviation
import "strings"
// Token represents a lexeme in the body with its byte offsets.
type Token struct {
Text string
Start int
End int
}
// Tokenizer splits text into tokens using a whitespace set.
// Whitespace characters split tokens but are not emitted.
// All other characters (including '/') are included in tokens.
type Tokenizer struct {
Whitespace string
}
// Tokenize tokenizes input and returns tokens with byte offsets.
func (t Tokenizer) Tokenize(input string) []Token {
if t.Whitespace == "" {
t.Whitespace = " \n\t\r"
}
var tokens []Token
start := -1
for idx, r := range input {
if strings.ContainsRune(t.Whitespace, r) {
if start != -1 {
tokens = append(tokens, Token{
Text: input[start:idx],
Start: start,
End: idx,
})
start = -1
}
continue
}
if start == -1 {
start = idx
}
}
if start != -1 {
tokens = append(tokens, Token{
Text: input[start:],
Start: start,
End: len(input),
})
}
return tokens
}