# Research Development Framework

A CLI-first research document management system optimized for Claude Code orchestration. RDF provides the data substrate (ingest, search, validate, govern) while Claude Code handles reasoning, planning, and synthesis.

---

## Quick Start

### Write an Essay (1 command)

```bash
./rdf essay "Freemasonry in the 1800's" --pages 20

# Output: projects/essays/ESSAY_xxx/essay_polished.md
```

### Write a Book (3 commands)

```bash
# 1. Generate outline from your idea
./rdf outline "Freemasonry during the 1800's" --chapters 5 --pages 100

# 2. Research and write (supervised mode pauses twice)
./rdf book projects/books/BOOK_xxx/outline.yaml --autonomy supervised

# 3. Find your chapters
ls projects/books/BOOK_xxx/
# → chapter_01.md, chapter_02.md, ...
```

### Universal Entry Point

```bash
# System infers essay vs book from parameters
./rdf write "Topic" --pages 20           # → essay
./rdf write "Topic" --chapters 5         # → book
```

---

## Architecture Overview

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                           CLAUDE CODE (Agent)                                │
│     Planning  │  Judgment  │  Synthesis  │  Quality Evaluation              │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                              RDF CLI LAYER                                   │
│   rdf write  │  rdf book  │  rdf essay  │  rdf research  │  rdf validate    │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           DATA SUBSTRATE                                     │
│   Ingest/OCR  │  Chunk  │  Embed  │  Search  │  Quote Extract  │  Validate  │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                              STORAGE                                         │
│              PostgreSQL + pgvector  │  File System (artifacts)              │
└─────────────────────────────────────────────────────────────────────────────┘
```

**Key Principle:** RDF does data operations, Claude Code does thinking.

---

## Project Structure

```
Research_development/
├── rdf                          # Unified CLI entrypoint
├── requirements.txt             # Python dependencies
├── tools.json                   # Tool registry for agent orchestration
│
├── NEW_DOCS/                    # Drop documents here for ingestion
│
├── docs/                        # Documentation
│   ├── README.md                # Documentation index
│   ├── SETUP.md                 # Installation guide
│   ├── CLI_USER_GUIDE.md        # Command reference
│   ├── WRITING_PROJECT_WORKFLOW.md  # Writing guide
│   └── ...
│
├── prompts/                     # Writing prompt library
│   ├── drafting/                # First draft prompts
│   ├── revision/                # Editing prompts
│   └── polish/                  # Final refinement prompts
│
├── projects/                    # Writing project outputs
│   ├── books/                   # Book projects
│   └── essays/                  # Essay projects
│
├── library/                     # Document storage
│   ├── ORGANIZED/               # Categorized documents
│   └── MARKDOWN_LIBRARY/        # Extracted markdown
│
├── pipeline/                    # Core Python modules
│   ├── ingest_documents.py      # Document import
│   ├── chunk_documents.py       # Text chunking
│   ├── generate_embeddings.py   # Vector embeddings
│   ├── search_export.py         # Search interface
│   ├── research_agent.py        # Autonomous research
│   ├── book.py                  # Book workflow
│   ├── essay.py                 # Essay workflow
│   └── ...                      # See pipeline/ for full list
│
├── config/                      # Configuration
│   └── project.yaml
│
├── database/                    # Database schemas
│   └── schema.sql
│
├── tests/                       # Test suite
├── scripts/                     # Utility scripts
├── tools/                       # Compilation utilities
│
├── .env                         # API keys
└── .env.db                      # Database credentials
```

---

## CLI Commands

All commands use the unified `rdf` entrypoint with `--format json` support.

### Core Commands (12)

| Command | Purpose |
|---------|---------|
| `rdf ingest <path>` | Add documents to library |
| `rdf search "<query>"` | Search the library |
| `rdf fetch <chunk_id>` | Retrieve bounded chunk text |
| `rdf research "<question>"` | Autonomous research |
| `rdf quotes "<topic>"` | Extract evidence |
| `rdf outline "<topic>"` | Generate book outline |
| `rdf book <outline.yaml>` | Book compilation workflow |
| `rdf essay "<topic>"` | Essay generation |
| `rdf write "<topic>"` | Universal entry point |
| `rdf validate <file>` | Claim verification |
| `rdf polish <file>` | Style refinement |
| `rdf queue <action>` | Review queue management |
| `rdf status` | Workflow state |

### Utility Commands (5)

| Command | Purpose |
|---------|---------|
| `rdf health` | Library health scan |
| `rdf edit-meta <DOC_ID>` | Metadata corrections |
| `rdf assess <DOC_ID>` | Mechanical document assessment |
| `rdf graph query "<concept>"` | Knowledge graph queries (JSON) |
| `rdf diff <a> <b>` | File comparison |

### Help

```bash
./rdf --help
./rdf commands              # List all commands as JSON
./rdf <command> --help      # Command-specific help
```

---

## Workflow Modes

### Claude Code Workflow (Recommended)

When using Claude Code as your writing agent, the workflow is **inherently human-guided**:
- Each phase is triggered explicitly by the user
- Claude Code pauses naturally after each command
- You review output before continuing to the next phase

**See:** [Writing Project Workflow](docs/WRITING_PROJECT_WORKFLOW.md) for the complete 9-phase guide.

### Direct CLI Mode (Essays Only)

For quick essay generation without human review at each step:

```bash
# Generate essay in one shot
./rdf essay "Topic" --pages 20

# Output: projects/essays/ESSAY_xxx/essay_polished.md
```

The `--autonomy` flag controls CLI pausing but is **not needed** when Claude Code orchestrates the workflow, since the conversation itself provides natural checkpoints.

---

## Configuration

### Environment Variables

```bash
# Database (required)
cat .env.db
# DB_HOST=localhost
# DB_NAME=research_dev_db
# DB_USER=research_dev_user
# DB_PASSWORD=your_password

# API keys (optional)
cat .env
# OPENAI_API_KEY=sk-...       # For embeddings
# TAVILY_API_KEY=tvly-...     # For web search (if allowed)
```

### Project Configuration

```yaml
# config/project.yaml
embeddings:
  provider: "openai"
  model: "text-embedding-3-small"

agent_safety:
  allow_web_search: false           # Library only by default
  validation_required_before_polish: true

defaults:
  autonomy_essay: "full"
  autonomy_book: "supervised"
  strict_library_only: true
```

---

## Common Workflows

### Add Documents

```bash
# Add PDFs to library
cp ~/documents/*.pdf NEW_DOCS/
./rdf ingest NEW_DOCS/

# Check library health
./rdf health --format json
```

### Research a Topic

```bash
# Quick search
./rdf search "consciousness in Steiner's work" --limit 10

# Deep research (library only)
./rdf research "How did Steiner's views evolve?" --strict

# Extract quotes
./rdf quotes "Steiner on consciousness" --format json
```

### Write Content

```bash
# Essay (full autonomy - runs to completion)
./rdf essay "The evolution of consciousness in esoteric thought" --pages 15

# Book (supervised - pauses for review)
./rdf outline "History of Western Esotericism" --chapters 5 --pages 100
./rdf book projects/books/BOOK_xxx/outline.yaml
```

### Validate and Polish

```bash
# Check claims against sources
./rdf validate chapter_01.md --format json

# Resolve any issues
./rdf queue list validation
./rdf queue approve val_001

# Polish prose
./rdf polish chapter_01.md --preset academic
```

---

## JSON Output

All commands return structured JSON with `--format json`:

```json
{
  "status": "success",
  "code": "SUCCESS",
  "message": "Operation completed",
  "data": {},
  "warnings": [],
  "queue_items_created": 0,
  "next_suggested_commands": []
}
```

Pause for review:
```json
{
  "status": "success",
  "code": "PAUSED_FOR_REVIEW",
  "message": "Research complete. Review gaps before drafting.",
  "data": {
    "checkpoint": "post_research",
    "resume_token": "RESUME_xxx_post_research",
    "decision_packet": { ... }
  }
}
```

---

## System Requirements

- **Python**: 3.10+ (3.11 recommended)
- **PostgreSQL**: 14+ with pgvector extension
- **Disk**: ~500MB for dependencies + document storage
- **RAM**: 4GB minimum

---

## Documentation

| Guide | Description |
|-------|-------------|
| [Feature Map](docs/FEATURE_MAP.md) | Complete specification |
| [Setup Guide](docs/SETUP.md) | Installation and configuration |
| [CLI User Guide](docs/CLI_USER_GUIDE.md) | Command reference |
| [Canonical Workflows](docs/CANONICAL_WORKFLOWS.md) | Standard workflows |
| [Database Schema](docs/DATABASE_SCHEMA.md) | PostgreSQL schema reference |
| [Developer Guide](docs/DEVELOPER_GUIDE.md) | Extending the framework |

---

## Testing

```bash
cd tests

# Quick smoke tests
python3 run_tests.py --quick

# Full test suite
python3 run_tests.py
```

---

## Support

1. Check [documentation](docs/)
2. Run commands with `--help`: `./rdf search --help`
3. View tool registry: `cat tools.json`
4. Run tests: `python3 tests/run_tests.py --quick`
