zhaojie/iov_data_analysis_agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

大更新，架构调整，数据分析能力提升，

Browse Source
This commit is contained in:
Jeason
2026-04-19 21:30:08 +08:00
parent 9d01f004d4
commit 00bd48e7e7
26 changed files with 4375 additions and 252 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.kiro/specs/analysis-dashboard-redesign/.config.kiro Normal file
View File

@@ -0,0 +1 @@
{"specId": "ea41aaef-0737-4255-bcad-90f156a5b2d5", "workflowType": "requirements-first", "specType": "feature"}

393
.kiro/specs/analysis-dashboard-redesign/design.md Normal file
View File

@@ -0,0 +1,393 @@
# Design Document: Analysis Dashboard Redesign
## Overview
This design transforms the Analysis Dashboard from a raw-log-centric 3-tab layout (Live Log, Report, Gallery) into a structured, evidence-driven 3-tab layout (Execution Process, Data Files, Report). The core architectural change is introducing a **Round_Data** structured data model that flows from the agent's execution loop through the API to the frontend, replacing the current raw text log approach.
Key design decisions:
- **Round_Data as the central abstraction**: Every analysis round produces a structured object containing reasoning, code, result summary, data evidence, and raw log. This single model drives the Execution Process tab, evidence linking, and data file tracking.
- **Auto-detection at the CodeExecutor level**: DataFrame detection and CSV export happen transparently in `CodeExecutor.execute_code()`, requiring no LLM cooperation. Prompt guidance is additive — it encourages the LLM to save files explicitly, but the system doesn't depend on it.
- **Gallery absorbed into Report**: Images are already rendered inline via `marked.js` Markdown parsing. Removing the Gallery tab is a subtraction, not an addition.
- **Evidence linking via HTML comments**: The LLM annotates report paragraphs with `<!-- evidence:round_N -->` comments during final report generation. The backend parses these to build a `supporting_data` mapping. This is a best-effort approach — missing annotations simply mean no "查看支撑数据" button.
## Architecture
```mermaid
flowchart TD
subgraph Backend
A[DataAnalysisAgent] -->|produces| B[Round_Data objects]
A -->|calls| C[CodeExecutor]
C -->|auto-detects DataFrames| D[CSV export to session dir]
C -->|captures evidence rows| B
C -->|parses DATA_FILE_SAVED markers| E[File metadata]
B -->|stored on| F[SessionData]
E -->|stored on| F
F -->|serves| G[GET /api/status]
F -->|serves| H[GET /api/data-files]
F -->|serves| I[GET /api/report]
end
subgraph Frontend
G -->|rounds array| J[Execution Process Tab]
H -->|file list + preview| K[Data Files Tab]
I -->|paragraphs + supporting_data| L[Report Tab]
end
```
### Data Flow
1. **Agent loop** (`DataAnalysisAgent.analyze`): Each round calls `CodeExecutor.execute_code()`, which returns an enriched result dict containing `evidence_rows`, `auto_exported_files`, and `prompt_saved_files`. The agent wraps this into a `Round_Data` dict and appends it to `SessionData.rounds`.
2. **Status polling**: Frontend polls `GET /api/status` every 2 seconds. The response now includes a `rounds` array. The frontend incrementally appends new `Round_Card` elements — it tracks the last-seen round count and only renders new entries.
3. **Data Files**: `GET /api/data-files` reads `SessionData.data_files` plus scans the session directory for CSV/XLSX files (fallback discovery). Preview reads the first 5 rows via pandas.
4. **Report with evidence**: `GET /api/report` parses `<!-- evidence:round_N -->` annotations, looks up `SessionData.rounds[N].evidence_rows`, and builds a `supporting_data` mapping keyed by paragraph ID.
## Components and Interfaces
### 1. CodeExecutor Enhancements (`utils/code_executor.py`)
**New behavior in `execute_code()`:**
```python
def execute_code(self, code: str) -> Dict[str, Any]:
"""Returns dict with keys: success, output, error, variables,
evidence_rows, auto_exported_files, prompt_saved_files"""
```
- **DataFrame snapshot before/after**: Before execution, capture `{name: id(obj)}` for all DataFrame variables. After execution, detect new names or changed `id()` values.
- **Evidence capture**: If the execution result is a DataFrame (via `result.result`), call `.head(10).to_dict(orient='records')` to produce `evidence_rows`. Also check the last assigned DataFrame variable in the namespace.
- **Auto-export**: For each newly detected DataFrame, export to `{session_dir}/{var_name}.csv` with dedup suffix. Record metadata in `auto_exported_files` list.
- **Marker parsing**: Scan `captured.stdout` for `[DATA_FILE_SAVED]` lines, parse filename/rows/description, record in `prompt_saved_files` list.
**Interface contract:**
```python
# evidence_rows: list[dict] — up to 10 rows as dicts
# auto_exported_files: list[dict] — [{variable_name, filename, rows, cols, columns}]
# prompt_saved_files: list[dict] — [{filename, rows, description}]
```
### 2. DataAnalysisAgent Changes (`data_analysis_agent.py`)
**Round_Data construction** in `_handle_generate_code()` and the main loop:
```python
round_data = {
"round": self.current_round,
"reasoning": yaml_data.get("reasoning", ""),
"code": code,
"result_summary": self._summarize_result(result),
"evidence_rows": result.get("evidence_rows", []),
"raw_log": feedback,
"auto_exported_files": result.get("auto_exported_files", []),
"prompt_saved_files": result.get("prompt_saved_files", []),
}
```
The agent appends `round_data` to `SessionData.rounds` (accessed via the progress callback or a direct reference). File metadata from both `auto_exported_files` and `prompt_saved_files` is merged into `SessionData.data_files`.
**`_summarize_result()`**: Produces a one-line summary from the execution result — e.g., "执行成功，输出 DataFrame (150行×8列)" or "执行失败: KeyError: 'col_x'".
### 3. SessionData Extension (`web/main.py`)
```python
class SessionData:
def __init__(self, session_id: str):
# ... existing fields ...
self.rounds: List[Dict] = [] # Round_Data objects
self.data_files: List[Dict] = [] # File metadata dicts
```
Persistence: `rounds` and `data_files` are written to `results.json` on analysis completion (existing pattern).
### 4. API Changes (`web/main.py`)
**`GET /api/status`** — add `rounds` to response:
```python
return {
# ... existing fields ...
"rounds": session.rounds,
}
```
**`GET /api/data-files`** — new endpoint:
```python
@app.get("/api/data-files")
async def list_data_files(session_id: str = Query(...)):
# Returns session.data_files + fallback directory scan
```
**`GET /api/data-files/preview`** — new endpoint:
```python
@app.get("/api/data-files/preview")
async def preview_data_file(session_id: str = Query(...), filename: str = Query(...)):
# Reads CSV/XLSX, returns {columns: [...], rows: [...first 5...]}
```
**`GET /api/data-files/download`** — new endpoint:
```python
@app.get("/api/data-files/download")
async def download_data_file(session_id: str = Query(...), filename: str = Query(...)):
# Returns FileResponse with appropriate MIME type
```
**`GET /api/report`** — enhanced response:
```python
return {
"content": content,
"base_path": web_base_path,
"paragraphs": paragraphs,
"supporting_data": supporting_data_map, # NEW: {paragraph_id: [evidence_rows]}
}
```
### 5. Prompt Changes (`prompts.py`)
Add to `data_analysis_system_prompt` after the existing code generation rules:
```
**中间数据保存规则**：
- 当你生成了有价值的中间数据（筛选子集、聚合表、聚类结果等），请主动保存为CSV/XLSX文件。
- 保存后必须打印标记行：`[DATA_FILE_SAVED] filename: {文件名}, rows: {行数}, description: {描述}`
- 示例：
```python
top_issues.to_csv(os.path.join(session_output_dir, "TOP问题汇总.csv"), index=False)
print(f"[DATA_FILE_SAVED] filename: TOP问题汇总.csv, rows: {len(top_issues)}, description: 各类型TOP问题聚合统计")
```
```
Add to `final_report_system_prompt` for evidence annotation:
```
**证据标注规则**：
- 当报告段落的结论来源于某一轮分析的数据，请在段落末尾添加HTML注释标注：`<!-- evidence:round_N -->`
- N 为产生该数据的分析轮次编号（从1开始）
- 示例：某段落描述了第3轮分析发现的车型分布规律，则在段落末尾添加 `<!-- evidence:round_3 -->`
```
### 6. Frontend Changes
**`index.html`**:
- Replace tab labels: "Live Log" → "执行过程", add "数据文件", keep "Report"
- Remove Gallery tab HTML and carousel container
- Add Execution Process tab container with round card template
- Add Data Files tab container with file card template
**`script.js`**:
- Remove gallery functions and state
- Add `renderRoundCards(rounds)` — incremental rendering using a `lastRenderedRound` counter
- Add `loadDataFiles()`, `previewDataFile(filename)`, `downloadDataFile(filename)`
- Modify `startPolling()` to call `renderRoundCards()` and `loadDataFiles()` on each cycle
- Add `showSupportingData(paraId)` for the evidence popover
- Modify `renderParagraphReport()` to add "查看支撑数据" buttons when `supporting_data[paraId]` exists
- Update `switchTab()` to handle `execution`, `datafiles`, `report`
**`clean_style.css`**:
- Add `.round-card`, `.round-card-header`, `.round-card-body` styles
- Add `.data-file-card`, `.data-preview-table` styles
- Add `.supporting-data-btn`, `.supporting-data-popover` styles
- Remove `.carousel-*` styles
## Data Models
### Round_Data (Python dict)
```python
{
"round": int, # 1-indexed round number
"reasoning": str, # LLM reasoning text (may be empty)
"code": str, # Generated Python code
"result_summary": str, # One-line execution summary
"evidence_rows": list[dict], # Up to 10 rows as [{col: val, ...}]
"raw_log": str, # Full execution feedback text
"auto_exported_files": list[dict], # Auto-detected DataFrame exports
"prompt_saved_files": list[dict], # LLM-guided file saves
}
```
### File Metadata (Python dict)
```python
{
"filename": str, # e.g., "top_issues.csv"
"description": str, # Human-readable description
"rows": int, # Row count
"cols": int, # Column count (optional, may be 0)
"columns": list[str], # Column names (optional)
"size_bytes": int, # File size
"source": str, # "auto" | "prompt" — how the file was created
}
```
### SessionData Extension
```python
class SessionData:
rounds: List[Dict] = [] # List of Round_Data dicts
data_files: List[Dict] = [] # List of File Metadata dicts
```
### API Response: GET /api/status (extended)
```json
{
"is_running": true,
"log": "...",
"has_report": false,
"rounds": [
{
"round": 1,
"reasoning": "正在执行阶段1...",
"code": "import pandas as pd\n...",
"result_summary": "执行成功，输出 DataFrame (150行×8列)",
"evidence_rows": [{"车型": "...", "模块": "..."}],
"raw_log": "..."
}
],
"progress_percentage": 25.0,
"current_round": 1,
"max_rounds": 20,
"status_message": "第1/20轮分析中..."
}
```
### API Response: GET /api/data-files
```json
{
"files": [
{
"filename": "top_issues.csv",
"description": "各类型TOP问题聚合统计",
"rows": 25,
"cols": 6,
"size_bytes": 2048
}
]
}
```
### API Response: GET /api/report (extended)
```json
{
"content": "...",
"base_path": "/outputs/session_xxx",
"paragraphs": [...],
"supporting_data": {
"p-3": [{"车型": "A", "模块": "TSP", "数量": 42}],
"p-7": [{"问题类型": "远控", "占比": "35%"}]
}
}
```
## Correctness Properties
*A property is a characteristic or behavior that should hold true across all valid executions of a system — essentially, a formal statement about what the system should do. Properties serve as the bridge between human-readable specifications and machine-verifiable correctness guarantees.*
### Property 1: Round_Data Structural Completeness and Ordering
*For any* sequence of analysis rounds (varying in count from 1 to N, with varying execution results including successes, failures, and missing YAML fields), every Round_Data object appended to `SessionData.rounds` SHALL contain all required fields (`round`, `reasoning`, `code`, `result_summary`, `evidence_rows`, `raw_log`) with correct types, and the list SHALL preserve insertion order (i.e., `rounds[i].round <= rounds[i+1].round` for all consecutive pairs).
**Validates: Requirements 1.1, 1.3, 1.4**
### Property 2: Evidence Capture Bounded and Correctly Serialized
*For any* DataFrame of arbitrary size (0 to 10,000 rows, 1 to 50 columns) produced by code execution, the evidence capture SHALL return a list of at most 10 dictionaries, where each dictionary's keys exactly match the DataFrame's column names, and the list length equals `min(10, len(dataframe))`.
**Validates: Requirements 4.1, 4.2, 4.3**
### Property 3: Filename Deduplication Uniqueness
*For any* sequence of auto-export operations (1 to 20) targeting the same variable name in the same session directory, all generated filenames SHALL be unique (no two exports produce the same filename), and no previously existing file SHALL be overwritten.
**Validates: Requirements 5.3**
### Property 4: Auto-Export Metadata Completeness
*For any* newly detected DataFrame variable (with arbitrary variable name, row count, column count, and column names), the auto-export metadata dict SHALL contain all required fields (`variable_name`, `filename`, `rows`, `cols`, `columns`) with values matching the source DataFrame's actual properties.
**Validates: Requirements 5.4, 5.5**
### Property 5: DATA_FILE_SAVED Marker Parsing Round-Trip
*For any* valid filename string (alphanumeric, Chinese characters, underscores, hyphens, with .csv or .xlsx extension), any positive integer row count, and any non-empty description string, formatting these values into the standardized marker format `[DATA_FILE_SAVED] filename: {name}, rows: {count}, description: {desc}` and then parsing the marker SHALL recover the original filename, row count, and description exactly.
**Validates: Requirements 6.3**
### Property 6: Data File Preview Bounded Rows
*For any* CSV file containing 0 to 10,000 rows and 1 to 50 columns, the preview function SHALL return a result with `columns` matching the file's column names exactly, and `rows` containing at most 5 dictionaries, where each dictionary's keys match the column names.
**Validates: Requirements 7.2**
### Property 7: Evidence Annotation Parsing Correctness
*For any* Markdown report text containing a mix of paragraphs with and without `<!-- evidence:round_N -->` annotations (where N varies from 1 to 100), the annotation parser SHALL: (a) correctly extract the round number for every annotated paragraph, (b) exclude non-annotated paragraphs from the `supporting_data` mapping, and (c) produce a mapping where each key is a valid paragraph ID and each value references a valid round number.
**Validates: Requirements 11.3, 11.4**
### Property 8: SessionData JSON Serialization Round-Trip
*For any* `SessionData` instance with arbitrary `rounds` (list of Round_Data dicts) and `data_files` (list of file metadata dicts), serializing these to JSON and deserializing back SHALL produce lists that are equal to the originals.
**Validates: Requirements 12.4**
## Error Handling
### CodeExecutor Errors
- **DataFrame evidence capture failure**: If `.head(10).to_dict(orient='records')` raises an exception (e.g., mixed types, memory issues), catch the exception and return an empty `evidence_rows` list. Log a warning but do not fail the execution.
- **Auto-export failure**: If CSV writing fails for a detected DataFrame (e.g., permission error, disk full), catch the exception, log a warning with the variable name, and skip that export. Other detected DataFrames should still be exported.
- **Marker parsing failure**: If a `[DATA_FILE_SAVED]` line doesn't match the expected format, skip it silently. Malformed markers should not crash the execution pipeline.
### API Errors
- **Missing session**: All new endpoints return HTTP 404 with `{"detail": "Session not found"}` for invalid session IDs.
- **Missing file**: `GET /api/data-files/preview` and `GET /api/data-files/download` return HTTP 404 with `{"detail": "File not found: {filename}"}` when the requested file doesn't exist in the session directory.
- **Corrupt CSV**: If a CSV file can't be read by pandas during preview, return HTTP 500 with `{"detail": "Failed to read file: {error}"}`.
### Frontend Errors
- **Polling with missing rounds**: If `rounds` is undefined or null in the status response, treat it as an empty array. Don't crash the rendering loop.
- **Evidence popover with empty data**: If `supporting_data[paraId]` is an empty array, don't show the button (same as missing).
- **Incremental rendering mismatch**: If `rounds.length < lastRenderedRound` (server restart scenario), reset `lastRenderedRound` to 0 and re-render all cards.
### Agent Errors
- **Missing reasoning field**: Already handled — store empty string (Requirement 1.4).
- **Evidence annotation missing**: Already handled — paragraphs without annotations simply don't get supporting data buttons. This is by design, not an error.
## Testing Strategy
### Property-Based Tests (Hypothesis)
The project already uses `hypothesis` with `max_examples=20` for fast execution (see `tests/test_properties.py`). New property tests will follow the same pattern.
**Library**: `hypothesis` (already installed)
**Configuration**: `max_examples=100` minimum per property (increased from existing 20 for new properties)
**Tag format**: `Feature: analysis-dashboard-redesign, Property {N}: {title}`
Properties to implement:
1. **Round_Data structural completeness** — Generate random execution results, verify Round_Data fields
2. **Evidence capture bounded** — Generate random DataFrames, verify evidence row count and format
3. **Filename deduplication** — Generate sequences of same-name exports, verify uniqueness
4. **Auto-export metadata** — Generate random DataFrames, verify metadata fields
5. **Marker parsing round-trip** — Generate random filenames/rows/descriptions, verify parse(format(x)) == x
6. **Preview bounded rows** — Generate random CSVs, verify preview row count and columns
7. **Evidence annotation parsing** — Generate random annotated Markdown, verify extraction
8. **SessionData JSON round-trip** — Generate random rounds/data_files, verify serialize/deserialize identity
### Unit Tests
- Prompt content assertions (6.1, 6.2, 11.2): Verify prompt strings contain required instruction text
- SessionData initialization (12.1, 12.2): Verify new attributes exist with correct defaults
- API response shape (2.1, 2.3): Verify status endpoint returns rounds array and log field
- Tab switching (9.4): Verify switchTab handles new tab identifiers
### Integration Tests
- End-to-end round capture: Run a mini analysis session, verify rounds are populated
- Data file API flow: Create files, call list/preview/download endpoints, verify responses
- Report evidence linking: Generate a report with annotations, call report API, verify supporting_data mapping
### Manual Testing
- UI layout verification (3.1-3.6, 8.1-8.5, 9.1-9.3, 10.1-10.4): Visual inspection of tab layout, round cards, data file cards, inline images, and supporting data popovers

159
.kiro/specs/analysis-dashboard-redesign/requirements.md Normal file
View File

@@ -0,0 +1,159 @@
# Requirements Document
## Introduction
This feature redesigns the Analysis Dashboard from the current 3-tab layout (Live Log, Report, Gallery) to a new 3-tab layout (Execution Process, Data Files, Report) with richer functionality. The redesign introduces structured round-by-round execution cards, intermediate data file browsing, inline image display within the report, and a data evidence/supporting data feature that links analysis conclusions to the specific data rows that support them. The Gallery tab is removed; its functionality is absorbed into the Report tab.
## Glossary
- **Dashboard**: The main analysis output panel in the web frontend (`index.html`) containing tabs for viewing analysis results.
- **Execution_Process_Tab**: The new first tab (执行过程) replacing the Live Log tab, displaying analysis rounds as collapsible cards.
- **Round_Card**: A collapsible UI card within the Execution_Process_Tab representing one analysis round, containing reasoning, code, result summary, data evidence, and raw log.
- **Data_Files_Tab**: The new second tab (数据文件) showing intermediate data files produced during analysis.
- **Report_Tab**: The enhanced third tab (报告) with inline images and supporting data links.
- **Data_Evidence**: Specific data rows extracted during analysis that support a particular analytical conclusion or claim.
- **CodeExecutor**: The Python class (`utils/code_executor.py`) responsible for executing generated analysis code in an IPython environment.
- **DataAnalysisAgent**: The Python class (`data_analysis_agent.py`) orchestrating the multi-round LLM-driven analysis workflow.
- **SessionData**: The Python class (`web/main.py`) tracking per-session state including running status, output directory, and analysis results.
- **Status_API**: The `GET /api/status` endpoint polled every 2 seconds by the frontend to retrieve analysis progress.
- **Data_Files_API**: The new set of API endpoints (`GET /api/data-files`, `GET /api/data-files/preview`, `GET /api/data-files/download`) for listing, previewing, and downloading intermediate data files.
- **Round_Data**: A structured JSON object representing one analysis round, containing fields for reasoning, code, execution result summary, data evidence rows, and raw log output.
- **Auto_Detection**: The mechanism by which CodeExecutor automatically detects new DataFrames created during code execution and exports them as files.
- **Prompt_Guidance**: Instructions embedded in the system prompt that direct the LLM to proactively save intermediate analysis results as files.
## Requirements
### Requirement 1: Structured Round Data Capture
**User Story:** As a user, I want each analysis round's data to be captured in a structured format, so that the frontend can render rich execution cards instead of raw log text.
#### Acceptance Criteria
1. WHEN an analysis round completes, THE DataAnalysisAgent SHALL produce a Round_Data object containing the following fields: round number, AI reasoning text, generated code, execution result summary, data evidence rows (list of dictionaries), and raw log output.
2. WHEN the DataAnalysisAgent processes an LLM response with a YAML `reasoning` field, THE DataAnalysisAgent SHALL extract and store the reasoning text in the Round_Data object for that round.
3. THE DataAnalysisAgent SHALL append each completed Round_Data object to a list stored on the SessionData instance, preserving insertion order.
4. IF the LLM response does not contain a parseable `reasoning` field, THEN THE DataAnalysisAgent SHALL store an empty string as the reasoning text in the Round_Data object.
### Requirement 2: Structured Status API Response
**User Story:** As a frontend developer, I want the status API to return structured round data, so that I can render execution cards in real time.
#### Acceptance Criteria
1. WHEN the frontend polls `GET /api/status`, THE Status_API SHALL return a JSON response containing a `rounds` array of Round_Data objects in addition to the existing fields (`is_running`, `has_report`, `progress_percentage`, `current_round`, `max_rounds`, `status_message`).
2. WHEN a new analysis round completes between two polling intervals, THE Status_API SHALL include the newly completed Round_Data object in the `rounds` array on the next poll response.
3. THE Status_API SHALL continue to return the `log` field containing raw log text for backward compatibility.
### Requirement 3: Execution Process Tab UI
**User Story:** As a user, I want to see each analysis round as a collapsible card with reasoning, code, results, and data evidence, so that I can understand the step-by-step analysis process.
#### Acceptance Criteria
1. THE Dashboard SHALL display an "执行过程" (Execution Process) tab as the first tab, replacing the current "Live Log" tab.
2. WHEN the Execution_Process_Tab is active, THE Dashboard SHALL render one Round_Card for each entry in the `rounds` array returned by the Status_API.
3. THE Round_Card SHALL default to a collapsed state showing only the round number and a one-line execution result summary.
4. WHEN a user clicks on a collapsed Round_Card, THE Dashboard SHALL expand the card to reveal: AI reasoning text, generated code (in a collapsible sub-section), execution result summary, data evidence section (labeled "本轮数据案例"), and raw log output (in a collapsible sub-section).
5. WHEN a new Round_Data object appears in the polling response, THE Dashboard SHALL append a new Round_Card to the Execution_Process_Tab without removing or re-rendering existing cards.
6. WHILE analysis is running, THE Dashboard SHALL auto-scroll the Execution_Process_Tab to keep the latest Round_Card visible.
### Requirement 4: Data Evidence Capture
**User Story:** As a user, I want to see the specific data rows that support each analytical conclusion, so that I can verify claims made by the AI agent.
#### Acceptance Criteria
1. WHEN the CodeExecutor executes code that produces a DataFrame result, THE CodeExecutor SHALL capture up to 10 representative rows from that DataFrame as the data evidence for the current round.
2. THE CodeExecutor SHALL serialize data evidence rows as a list of dictionaries (one dictionary per row, keys being column names) and include the list in the execution result returned to the DataAnalysisAgent.
3. IF the code execution does not produce a DataFrame result, THEN THE CodeExecutor SHALL return an empty list as the data evidence.
4. THE DataAnalysisAgent SHALL include the data evidence list in the Round_Data object for the corresponding round.
### Requirement 5: DataFrame Auto-Detection and Export
**User Story:** As a user, I want intermediate DataFrames created during analysis to be automatically saved as files, so that I can browse and download them from the Data Files tab.
#### Acceptance Criteria
1. WHEN code execution completes, THE CodeExecutor SHALL compare the set of DataFrame variables in the IPython namespace before and after execution to detect newly created DataFrames.
2. WHEN a new DataFrame variable is detected, THE CodeExecutor SHALL export the DataFrame to the session output directory as a CSV file named `{variable_name}.csv`.
3. IF a file with the same name already exists in the session output directory, THEN THE CodeExecutor SHALL append a numeric suffix (e.g., `_1`, `_2`) to avoid overwriting.
4. THE CodeExecutor SHALL record metadata for each auto-exported file: variable name, filename, row count, column count, and column names.
5. WHEN auto-export completes, THE CodeExecutor SHALL include the exported file metadata in the execution result returned to the DataAnalysisAgent.
### Requirement 6: Prompt Guidance for Intermediate File Saving
**User Story:** As a user, I want the LLM to proactively save intermediate analysis results as files, so that important intermediate datasets are available for review.
#### Acceptance Criteria
1. THE system prompt (`prompts.py`) SHALL include instructions directing the LLM to save intermediate analysis results (filtered subsets, aggregation tables, clustering results) as CSV or XLSX files in the `session_output_dir`.
2. THE system prompt SHALL instruct the LLM to print a standardized marker line after saving each file, in the format: `[DATA_FILE_SAVED] filename: {name}, rows: {count}, description: {desc}`.
3. WHEN the CodeExecutor detects a `[DATA_FILE_SAVED]` marker in the execution output, THE CodeExecutor SHALL parse the marker and record the file metadata (filename, row count, description).
### Requirement 7: Data Files API
**User Story:** As a frontend developer, I want API endpoints to list, preview, and download intermediate data files, so that the Data Files tab can display and serve them.
#### Acceptance Criteria
1. WHEN the frontend requests `GET /api/data-files?session_id={id}`, THE Data_Files_API SHALL return a JSON array of file entries, each containing: filename, description, row count, column count, and file size in bytes.
2. WHEN the frontend requests `GET /api/data-files/preview?session_id={id}&filename={name}`, THE Data_Files_API SHALL return a JSON object containing: column names (list of strings), and up to 5 data rows (list of dictionaries).
3. WHEN the frontend requests `GET /api/data-files/download?session_id={id}&filename={name}`, THE Data_Files_API SHALL return the file as a downloadable attachment with the appropriate MIME type (`text/csv` for CSV, `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` for XLSX).
4. IF the requested file does not exist, THEN THE Data_Files_API SHALL return HTTP 404 with a descriptive error message.
### Requirement 8: Data Files Tab UI
**User Story:** As a user, I want to browse intermediate data files produced during analysis, preview their contents, and download them individually.
#### Acceptance Criteria
1. THE Dashboard SHALL display a "数据文件" (Data Files) tab as the second tab.
2. WHEN the Data_Files_Tab is active, THE Dashboard SHALL fetch the file list from `GET /api/data-files` and render each file as a card showing: filename, description, and row count.
3. WHEN a user clicks on a file card, THE Dashboard SHALL fetch the preview from `GET /api/data-files/preview` and display a table showing column headers and up to 5 data rows.
4. WHEN a user clicks the download button on a file card, THE Dashboard SHALL initiate a file download via `GET /api/data-files/download`.
5. WHILE analysis is running, THE Dashboard SHALL refresh the file list on each polling cycle to show newly created files.
### Requirement 9: Gallery Removal and Inline Images in Report
**User Story:** As a user, I want images displayed inline within report paragraphs instead of in a separate Gallery tab, so that visual evidence is presented in context.
#### Acceptance Criteria
1. THE Dashboard SHALL remove the "Gallery" tab from the tab bar.
2. THE Dashboard SHALL remove the gallery carousel UI (carousel container, navigation buttons, image info panel) from the HTML.
3. THE Report_Tab SHALL render images inline within report paragraphs using standard Markdown image syntax (`![alt](url)`), as already supported by the existing `marked.js` rendering.
4. THE `switchTab` function in `script.js` SHALL handle only the three new tab identifiers: `execution`, `datafiles`, and `report`.
5. THE frontend SHALL remove all gallery-related JavaScript functions (`loadGallery`, `renderGalleryImage`, `prevImage`, `nextImage`) and associated state variables (`galleryImages`, `currentImageIndex`).
### Requirement 10: Supporting Data Button in Report
**User Story:** As a user, I want report paragraphs that make data-driven claims to have a "查看支撑数据" button, so that I can view the evidence data that supports each conclusion.
#### Acceptance Criteria
1. WHEN the Report_Tab renders a paragraph of type `text` that has associated data evidence, THE Dashboard SHALL display a "查看支撑数据" (View Supporting Data) button below the paragraph content.
2. WHEN a user clicks the "查看支撑数据" button, THE Dashboard SHALL display a popover or modal showing the associated data evidence rows in a table format.
3. THE `GET /api/report` response SHALL include a `supporting_data` mapping (keyed by paragraph ID) containing the data evidence rows relevant to each paragraph.
4. IF a paragraph has no associated data evidence, THEN THE Dashboard SHALL not display the "查看支撑数据" button for that paragraph.
### Requirement 11: Report-to-Evidence Linking in Backend
**User Story:** As a backend developer, I want the system to associate data evidence from execution rounds with report paragraphs, so that the frontend can display supporting data buttons.
#### Acceptance Criteria
1. WHEN generating the final report, THE DataAnalysisAgent SHALL pass the collected data evidence from all rounds to the report generation prompt.
2. THE final report generation prompt SHALL instruct the LLM to annotate report paragraphs with round references (e.g., `<!-- evidence:round_3 -->`) when a paragraph's content is derived from a specific analysis round.
3. WHEN the `GET /api/report` endpoint parses the report, THE backend SHALL extract evidence annotations and build a `supporting_data` mapping by looking up the referenced round's data evidence from the SessionData.
4. IF a paragraph contains no evidence annotation, THEN THE backend SHALL exclude that paragraph from the `supporting_data` mapping.
### Requirement 12: Session Data Model Extension
**User Story:** As a backend developer, I want the SessionData model to store structured round data and data file metadata, so that the new API endpoints can serve this information.
#### Acceptance Criteria
1. THE SessionData class SHALL include a `rounds` attribute (list of Round_Data dictionaries) to store structured data for each completed analysis round.
2. THE SessionData class SHALL include a `data_files` attribute (list of file metadata dictionaries) to store information about intermediate data files.
3. WHEN a new data file is detected (via auto-detection or prompt-guided saving), THE DataAnalysisAgent SHALL append the file metadata to the SessionData `data_files` list.
4. THE SessionData class SHALL persist the `rounds` and `data_files` attributes to the session's `results.json` file upon analysis completion.

102
.kiro/specs/analysis-dashboard-redesign/tasks.md Normal file
View File

@@ -0,0 +1,102 @@
# Tasks: Analysis Dashboard Redesign
## Phase 1: Backend Data Model + API Changes (Foundation)
- [x] 1. Extend SessionData model
- [x] 1.1 Add `rounds: List[Dict]` attribute to `SessionData.__init__()` in `web/main.py`, initialized to empty list
- [x] 1.2 Add `data_files: List[Dict]` attribute to `SessionData.__init__()` in `web/main.py`, initialized to empty list
- [x] 1.3 Update `_reconstruct_session()` to load `rounds` and `data_files` from `results.json` when reconstructing historical sessions
- [x] 1.4 Update `run_analysis_task()` to persist `session.rounds` and `session.data_files` to `results.json` on analysis completion
- [x] 2. Update Status API response
- [x] 2.1 Add `rounds` field to `GET /api/status` response dict, returning `session.rounds`
- [x] 2.2 Verify backward compatibility: ensure `log`, `is_running`, `has_report`, `progress_percentage`, `current_round`, `max_rounds`, `status_message` fields remain unchanged
- [x] 3. Add Data Files API endpoints
- [x] 3.1 Implement `GET /api/data-files` endpoint: return `session.data_files` merged with fallback directory scan for CSV/XLSX files, each entry containing filename, description, rows, cols, size_bytes
- [x] 3.2 Implement `GET /api/data-files/preview` endpoint: read CSV/XLSX via pandas, return `{columns: [...], rows: [...first 5 rows as dicts...]}`; return 404 if file not found
- [x] 3.3 Implement `GET /api/data-files/download` endpoint: return `FileResponse` with correct MIME type (`text/csv` or `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`); return 404 if file not found
- [x] 4. Enhance Report API for evidence linking
- [x] 4.1 Implement `_extract_evidence_annotations(paragraphs, session)` function: parse `<!-- evidence:round_N -->` comments from paragraph content, look up `session.rounds[N-1].evidence_rows`, build `supporting_data` mapping keyed by paragraph ID
- [x] 4.2 Update `GET /api/report` to include `supporting_data` mapping in response JSON
## Phase 2: CodeExecutor Enhancements
- [x] 5. Add evidence capture to CodeExecutor
- [x] 5.1 In `execute_code()`, after successful execution, check if `result.result` is a DataFrame; if so, capture `result.result.head(10).to_dict(orient='records')` as `evidence_rows`; wrap in try/except returning empty list on failure
- [x] 5.2 Also check the last-assigned DataFrame variable in the namespace as a fallback evidence source when `result.result` is not a DataFrame
- [x] 5.3 Include `evidence_rows` key in the returned result dict
- [x] 6. Add DataFrame auto-detection and export
- [x] 6.1 Before `shell.run_cell(code)`, snapshot DataFrame variables: `{name: id(obj) for name, obj in shell.user_ns.items() if isinstance(obj, pd.DataFrame)}`
- [x] 6.2 After execution, compare snapshots to detect new or changed DataFrame variables
- [x] 6.3 For each new DataFrame, export to `{output_dir}/{var_name}.csv` with numeric suffix deduplication if file exists
- [x] 6.4 Record metadata for each export: `{variable_name, filename, rows, cols, columns}` in `auto_exported_files` list
- [x] 6.5 Include `auto_exported_files` key in the returned result dict
- [x] 7. Add DATA_FILE_SAVED marker parsing
- [x] 7.1 After execution, scan `captured.stdout` for lines matching `[DATA_FILE_SAVED] filename: {name}, rows: {count}, description: {desc}`
- [x] 7.2 Parse each marker line and record `{filename, rows, description}` in `prompt_saved_files` list
- [x] 7.3 Include `prompt_saved_files` key in the returned result dict
## Phase 3: Agent Changes
- [x] 8. Structured Round_Data construction in DataAnalysisAgent
- [x] 8.1 Add `_summarize_result(result)` method: produce one-line summary from execution result (e.g., "执行成功，输出 DataFrame (150行×8列)" or "执行失败: {error}")
- [x] 8.2 In `_handle_generate_code()`, construct `round_data` dict with fields: round, reasoning (from `yaml_data.get("reasoning", "")`), code, result_summary, evidence_rows, raw_log, auto_exported_files, prompt_saved_files
- [x] 8.3 After constructing round_data, append it to `SessionData.rounds` (via progress callback or direct reference)
- [x] 8.4 Merge file metadata from `auto_exported_files` and `prompt_saved_files` into `SessionData.data_files`
- [x] 9. Update system prompts
- [x] 9.1 Add intermediate data saving instructions to `data_analysis_system_prompt` in `prompts.py`: instruct LLM to save intermediate results and print `[DATA_FILE_SAVED]` marker
- [x] 9.2 Add evidence annotation instructions to `final_report_system_prompt` in `prompts.py`: instruct LLM to add `<!-- evidence:round_N -->` comments to report paragraphs
- [x] 9.3 Update `_build_final_report_prompt()` in `data_analysis_agent.py` to include collected evidence data from all rounds in the prompt context
## Phase 4: Frontend Tab Restructuring
- [x] 10. HTML restructuring
- [x] 10.1 In `index.html`, replace tab labels: "Live Log" → "执行过程", add "数据文件" tab, keep "Report"; remove "Gallery" tab
- [x] 10.2 Replace the `logsTab` div content with an Execution Process container (`executionTab`) containing a scrollable round-cards wrapper
- [x] 10.3 Add a `datafilesTab` div with a file-cards grid container and a preview panel area
- [x] 10.4 Remove the Gallery tab HTML: carousel container, navigation buttons, image info panel
- [x] 11. JavaScript: Execution Process Tab
- [x] 11.1 Add `lastRenderedRound` state variable and `renderRoundCards(rounds)` function: compare `rounds.length` with `lastRenderedRound`, create and append new Round_Card DOM elements for new entries only
- [x] 11.2 Implement Round_Card HTML generation: collapsed state shows round number + result_summary; expanded state shows reasoning, code (collapsible), result_summary, evidence table ("本轮数据案例"), raw log (collapsible)
- [x] 11.3 Add click handler for Round_Card toggle (collapse/expand)
- [x] 11.4 Add auto-scroll logic: when analysis is running, scroll Execution Process container to bottom after appending new cards
- [x] 12. JavaScript: Data Files Tab
- [x] 12.1 Implement `loadDataFiles()`: fetch `GET /api/data-files`, render file cards showing filename, description, row count
- [x] 12.2 Implement `previewDataFile(filename)`: fetch `GET /api/data-files/preview`, render a table with column headers and up to 5 rows
- [x] 12.3 Implement `downloadDataFile(filename)`: trigger download via `GET /api/data-files/download`
- [x] 12.4 In `startPolling()`, call `loadDataFiles()` on each polling cycle when Data Files tab is active or when analysis is running
- [x] 13. JavaScript: Gallery removal and tab updates
- [x] 13.1 Remove gallery functions: `loadGallery`, `renderGalleryImage`, `prevImage`, `nextImage` and state variables `galleryImages`, `currentImageIndex`
- [x] 13.2 Update `switchTab()` to handle `execution`, `datafiles`, `report` identifiers instead of `logs`, `report`, `gallery`
- [x] 13.3 Update `startPolling()` to call `renderRoundCards()` with `data.rounds` on each polling cycle
- [x] 14. JavaScript: Supporting data in Report
- [x] 14.1 Update `loadReport()` to store `supporting_data` mapping from API response
- [x] 14.2 Update `renderParagraphReport()` to add "查看支撑数据" button below paragraphs that have entries in `supporting_data`
- [x] 14.3 Implement `showSupportingData(paraId)`: display a popover/modal with evidence rows rendered as a table
- [x] 15. CSS updates
- [x] 15.1 Add `.round-card`, `.round-card-header`, `.round-card-body`, `.round-card-collapsed`, `.round-card-expanded` styles
- [x] 15.2 Add `.data-file-card`, `.data-preview-table` styles
- [x] 15.3 Add `.supporting-data-btn`, `.supporting-data-popover` styles
- [x] 15.4 Remove `.carousel-*` styles (carousel-container, carousel-slide, carousel-btn, image-info, image-title, image-desc)
## Phase 5: Property-Based Tests
- [x] 16. Write property-based tests
- [x] 16.1 ~PBT~ Property 1: Round_Data structural completeness — generate random execution results, verify all required fields present with correct types and insertion order preserved
- [x] 16.2 ~PBT~ Property 2: Evidence capture bounded — generate random DataFrames (0-10000 rows, 1-50 cols), verify evidence_rows length <= 10 and each row dict has correct keys
- [x] 16.3 ~PBT~ Property 3: Filename deduplication — generate sequences of same-name exports (1-20), verify all filenames unique
- [x] 16.4 ~PBT~ Property 4: Auto-export metadata completeness — generate random DataFrames, verify metadata contains variable_name, filename, rows, cols, columns with correct values
- [x] 16.5 ~PBT~ Property 5: DATA_FILE_SAVED marker parsing round-trip — generate random filenames/rows/descriptions, verify parse(format(x)) == x
- [x] 16.6 ~PBT~ Property 6: Data file preview bounded rows — generate random CSVs (0-10000 rows), verify preview returns at most 5 rows with correct column names
- [x] 16.7 ~PBT~ Property 7: Evidence annotation parsing — generate random annotated Markdown, verify correct round extraction and non-annotated paragraph exclusion
- [x] 16.8 ~PBT~ Property 8: SessionData JSON round-trip — generate random rounds/data_files, verify serialize then deserialize produces equal data