---
name: alibabacloud-maxframe-coding
description: >
    This skill should be used when the user asks to "create MaxFrame program", "process data with MaxCompute", "write data to MaxCompute table", "perform DataFrame operations", "use Tensor operations", "implement machine learning workflows", "test MaxFrame code", "debug MaxFrame job", "run MaxFrame code", "iterate on MaxFrame code", "debug UDF locally", "local debug MaxFrame", or mentions MaxCompute/ODPS. This skill covers three main scenarios: writing MaxFrame code, remote debugging using interactive mode, and local debugging with IDE support.
license: MIT
compatibility: >
    Requires Python 3.7+ (recommended: Python 3.11+), maxframe package, and ODPS/MaxCompute access
metadata:
  domain: coding
  owner: maxframe-team
  contact: xiaming.cxm@alibaba-inc.com,minghaohu.hmh@alibaba-inc.com
---

# MaxFrame Coding - Create, Test, Debug, and Iterate

## What This Skill Can Do

This skill is designed for **creating, testing, debugging, and iteratively developing MaxFrame programs**. It provides end-to-end support for the entire development workflow. It can help you:

### Creating MaxFrame Programs
- **Create MaxFrame jobs** from scratch or modify existing ones
- **Design data processing pipelines** using pandas-compatible APIs
- **Implement data transformations** (filtering, grouping, merging, aggregating)
- **Write data to MaxCompute tables** including DLF external tables
- **Perform DataFrame operations** for tabular data processing
- **Use Tensor operations** for numerical computing and feature engineering
- **Implement machine learning workflows** using scikit-learn-like APIs
- **Select appropriate MaxFrame operators** for specific tasks
- **Handle complex data structures** including Arrow types and nested data
- **Process data in batches** using `.mf.apply_chunk()` for large datasets

### Testing and Debugging (Remote Debug Mode)
- **Execute MaxFrame code** and monitor execution progress
- **Test and debug MaxFrame code** iteratively based on results
- **Print logview URLs** for detailed execution diagnostics
- **Handle errors comprehensively** with logging and troubleshooting guidance
- **Discover SDK interfaces** from Python site-packages
- **Configure environment variables** properly using dotenv
- **Manage sessions** with proper cleanup patterns
- **Analyze common error patterns** (authentication, table access, execution timeouts, type mismatches, SQL errors)

### Local Debugging
- **Debug UDF functions locally** with `debug=True` or `debug="local"` parameter
- **Use IDE breakpoints** for step-by-step debugging in PyCharm, VSCode, or DataWorks Notebook
- **Debug offline** without connecting to MaxCompute cluster
- **Test with sample data** from memory, local files, or MaxCompute tables
- **Iterate quickly** on apply/apply_chunk functions without remote submission
- **Validate data transformation logic** with immediate feedback

## What This Skill Cannot Do

This skill is **NOT** designed for:

- **Performance analysis** - Analyzing job execution time, bottlenecks, or optimization recommendations
- **Deep troubleshooting** - Advanced debugging of complex service-side runtime issues requiring deep MaxCompute internals knowledge
- **Cluster management** - Managing MaxCompute cluster resources or configuration
- **Performance tuning** - Optimizing query plans, adjusting parallelism, or engine selection
- **Cost analysis** - Calculating MaxCompute resource costs or quota management

For performance analysis, advanced troubleshooting, or optimization needs, use appropriate MaxFrame monitoring tools or consult MaxCompute documentation.

## Three Main Development Scenarios

This skill supports three primary development scenarios, each optimized for different stages of the MaxFrame development lifecycle.

### Scenario 1: Writing MaxFrame Code

**When to use:**
- Creating new data processing pipelines
- Implementing business logic and data transformations
- Writing production-ready MaxFrame applications

**What it provides:**
- Complete workflow for creating MaxFrame programs
- Guidance on operator selection and validation
- Best practices for reading, processing, and writing data
- Code structure and execution patterns

**See:** "Common Workflow" section below for detailed instructions.

### Scenario 2: Remote Debug Mode (Interactive)

**When to use:**
- Testing with production-scale data on MaxCompute
- Need to verify execution on actual cluster
- Investigating runtime issues that require logview URLs
- Debugging distributed execution problems

**What it provides:**
- Execute code on MaxCompute cluster
- Monitor execution progress in real-time
- Access logview URLs for detailed diagnostics
- Handle remote execution errors comprehensively

**See:** "Interactive Coding, Testing, and Debugging" section for detailed instructions.

### Scenario 3: Local Debug Mode

**When to use:**
- Developing UDF functions (`apply`, `apply_chunk`)
- Need IDE breakpoints and step-by-step debugging
- Want to debug offline without network access
- Working on complex logic that requires iterative testing

**What it provides:**
- Zero-configuration local debugging with `debug=True`
- Full IDE support (PyCharm, VSCode, DataWorks Notebook)
- Offline debugging without MaxCompute connection
- Fast iteration on UDF functions

**See:** `references/local_debug.md` for comprehensive local debugging guide.

## Choosing the Right Scenario

**Use Writing MaxFrame Code when:**
- Starting a new MaxFrame project
- Implementing complete data pipelines
- Writing production-ready code

**Use Remote Debug Mode when:**
- Testing with actual cluster resources
- Debugging issues that require distributed execution
- Need logview URLs for diagnostics
- Validating performance at scale

**Use Local Debug Mode when:**
- Developing UDF functions iteratively
- Need IDE breakpoints for step-by-step debugging
- Working offline without network access
- Validating transformation logic quickly

**Typical Workflow:**
1. Write initial code → 2. Debug locally with IDE → 3. Test remotely on cluster → 4. Deploy to production

## Overview

This skill enables writing programs using the MaxFrame SDK from an end-user perspective. MaxFrame provides pandas-compatible APIs for large-scale distributed data processing on MaxCompute, allowing seamless parallelization of pandas code with minimal changes.

## Core Principles

### Use Public APIs Only

Always use APIs exposed by the MaxFrame SDK. Avoid internal or undocumented functions. Key public modules:

- `maxframe.dataframe` - DataFrame operations (pandas-compatible)
- `maxframe.tensor` - Tensor operations (NumPy-like)
- `maxframe.learn` - Machine learning (scikit-learn-like)
- `maxframe.session` - Session management
- `maxframe.udf` - User-defined functions
- `maxframe.config` - Configuration options

### Lazy Execution

MaxFrame uses lazy execution. Operations build a computation graph and only execute when `.execute()` is called. Always call `.execute()` to trigger computation.

### Operator Selection

For finding, validating, and recommending MaxFrame operators, use the **operator-selector agent**. The agent provides:
- Operator discovery based on task descriptions
- Operator existence validation
- Optimized operator recommendations
- Usage examples without loading large documentation

The operator-selector agent is available at `../../agents/operator-selector.md` or uses the `scripts/lookup_operator.py` script to search and retrieve operator information.

Use a subagent to run operator selection tasks by referencing this agent. Ask questions like:
- "What operators should I use for time series resampling?"
- "Does MaxFrame have an apply_chunk operator?"
- "Show me examples of using the merge operator"

Also you can use mcp context7 for retrieve more infomration.

---

## Common Workflow

### 1. Setup Session

```python
import os
import maxframe.dataframe as md
from maxframe.session import new_session
from odps import ODPS

# Auto-detect from environment (preferred in DataWorks/MaxCompute Notebook)
session = new_session()

# Or explicit ODPS connection
# o = ODPS(
#     access_id=os.getenv("ODPS_ACCESS_ID"),
#     secret_access_key=os.getenv("ODPS_ACCESS_KEY"),
#     project=os.getenv("ODPS_PROJECT"),
#     endpoint=os.getenv("ODPS_ENDPOINT"),
# )
# session = new_session(o)
```

### 2. Read Data

```python
# Read from MaxCompute table
df = md.read_odps_table("table_name")

# Read from MaxCompute table, with one unique column of the table
df = md.read_odps_table("table_name", index_col="xx")

# With column selection
df = md.read_odps_table("table_name", columns=['id', 'value', 'timestamp'])

# With partition filter
df = md.read_odps_table("table_name", partition='ds=2024-01-01')

# Read from SQL query (with filters)
df = md.read_odps_query("SELECT * FROM table WHERE date >= '2024-01-01' AND status = 'active'")
```

**Note**: If the user does not provide an input table name, construct a pandas DataFrame based on the job requirements and load it using `md.read_pandas()`:

```python
import pandas as pd
import numpy as np

# Construct sample data based on job characteristics
# Example: For a time series analysis job
example_pd_df = pd.DataFrame({
    'timestamp': pd.date_range('2026-01-01', periods=1000, freq='H'),
    'metric_name': np.random.choice(['cpu', 'memory', 'disk'], 1000),
    'value': np.random.randn(1000) * 10 + 50,
    'host_id': np.random.choice(['host1', 'host2', 'host3'], 1000)
})

# Load into MaxFrame
df = md.read_pandas(example_pd_df)
```

**Key guidelines for constructing sample data**:
- Match the data types and structure to the actual job requirements
- Use realistic value ranges appropriate for the domain
- Include sufficient rows to demonstrate the processing logic (typically 100-1000 rows)
- Ensure column names are descriptive and match the processing operations

### 3. Select Operators (Confirm Before Coding)

**STOP - Confirm with user before writing processing code.**

For operator selection and validation, use one of these approaches:

**Option A: Use Operator-Selector Script**
```bash
# Search for operators by task description
python scripts/lookup_operator.py search "time series resampling"

# Validate if a specific operator exists
python scripts/lookup_operator.py validate apply_chunk

# Get detailed operator information
python scripts/lookup_operator.py info groupby
```

**Option B: Consult Operator-Selector Reference**
- Reference: `./references/operator-selector.md`
- Use a subagent to run operator selection tasks

**Confirmation Checklist:**
- [ ] Identified all required operations for the task
- [ ] Found appropriate MaxFrame operators for each operation
- [ ] Presented options to user with explanations
- [ ] Received user confirmation or preference

**Example confirmation dialog:**
> "For your data aggregation task, I've identified these options:
> 1. `groupby().agg()` - Standard pandas-compatible approach
> 2. `mf.apply_chunk()` - For custom aggregation with large datasets
> 
> Which approach do you prefer, or would you like me to explore other options?"

### 4. Process Data

Once operators are confirmed, implement the data processing:

```python
# Filter
filtered = df[df['column'] > 10]

# GroupBy and aggregate
result = df.groupby('category').agg({'value': 'sum'})

# Standard pandas operations
df['new_col'] = df['col1'] + df['col2']
df_sorted = df.sort_values('column')
df_merged = df1.merge(df2, on='key')

# Custom function in batches (for large datasets)
def process_batch(chunk):
    return chunk * 2

result = df.mf.apply_chunk(
    process_batch,
    batch_rows=1024,
    output_type='dataframe'
)
```

### 5. Write Data

```python
# Write to MaxCompute table
md.to_odps_table(df, "output_table", overwrite=True).execute()

# Write to DLF external table (if enabled)
from maxframe import options
options.sql.settings = {
    "odps.maxframe.resolve_dlf_tables": "true"
}
md.to_odps_table(df, "dlf_table").execute()
```

### 6. Execute and Cleanup

```python
# Execute operations (required for lazy execution)
result.execute()

# Destroy session when done
session.destroy()
```

### 7. Compile and verify

Use `py_compile` to test your generated job script.

**Recommended pattern for production code** (always cleanup on success or failure):

```python
try:
    # Execute operations (required for lazy execution)
    result.execute()
finally:
    # Destroy session when done (always runs, even on error)
    session.destroy()
```

## Installation

For detailed installation instructions and dependency requirements, refer to `references/installation.md`.

## Key Modules

### DataFrame Operations

pandas-compatible APIs for data manipulation (filtering, grouping, merging, transformations). Use for tabular data like transaction logs and user profiles. See `references/key_modules.md` for details.

### Tensor Operations

NumPy-like operations for numerical computing (linear algebra, statistical operations). Use for matrix/array data in scientific computing or feature engineering. See `references/key_modules.md` for details.

### Machine Learning

scikit-learn-like ML capabilities including preprocessing and distributed model training (LightGBM, XGBoost). Use for predictive modeling on large-scale datasets. See `references/key_modules.md` for details.


## Important Notes

- **Session Management**: Create a session before operations and destroy it when done
- **Configuration**: Use `options` to configure behavior like progress display, chunk size, and SQL settings
- **DLF Support**: Enable `odps.maxframe.resolve_dlf_tables` in SQL settings for DLF external tables
- **Batch Processing**: Use `.mf.apply_chunk()` for efficient batch processing of large datasets
- **Lazy Execution**: Call `.execute()` to trigger computation after building the computation graph
- **Error Handling**: Always wrap critical operations in try-except blocks for production code
- **Resource Cleanup**: Explicitly destroy sessions and clean up intermediate results

---

## Interactive Coding, Testing, and Debugging (Remote Debug Mode)

This section covers remote debug mode for executing and debugging MaxFrame programs on the MaxCompute cluster. For local debugging with IDE support, see `references/local_debug.md`.

**When to use remote debug mode:**
- Testing with production-scale data
- Verifying execution on actual cluster
- Need logview URLs for diagnostics
- Debugging distributed execution issues

### Workflow Pattern

1. **Create Code** - Generate MaxFrame program using the coding workflow above
2. **Execute and Test** - Run the code and monitor execution
3. **Analyze Results** - Check outputs, capture errors, use logview for diagnostics
4. **Iterate and Fix** - Refine code based on results and repeat

### Environment Configuration

Use environment variables to manage MaxCompute connection credentials:

**Commonly used environment variables:**
- `ODPS_PROJECT` - MaxCompute project name
- `ODPS_ACCESS_ID` - Access key ID for authentication
- `ODPS_ACCESS_KEY` - Access key secret for authentication
- `ODPS_ENDPOINT` - MaxCompute service endpoint URL

**Load environment variables programmatically:**
```python
import dotenv
import os

# Load environment variables (do NOT read .env files directly)
dotenv.load_dotenv()

# Access environment variables
project = os.getenv("ODPS_PROJECT")
access_id = os.getenv("ODPS_ACCESS_ID")
```

### Printing Logview URLs

The logview URL provides detailed execution logs, task information, and error details from MaxCompute:

```python
session = new_session(o)

try:
    # Print logview address after session creation
    print(f"Logview: {session.get_logview_address()}")

    # Your operations
    df = md.read_odps_table("table_name")
    result = df.groupby('col').sum()
    result.execute()

    # Print logview after execution
    logview_url = session.get_logview_address()
    if logview_url:
        print(f"Execution Logview: {logview_url}")

finally:
    session.destroy()
```

### Comprehensive Error Handling

```python
import logging
import traceback

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

try:
    logger.info(f"Session created. Logview: {session.get_logview_address()}")

    # Your operations
    df = md.read_odps_table("table_name")
    result = df.groupby('col').sum()

    try:
        result.execute()
        logger.info("Execution completed successfully")

        logview_url = session.get_logview_address()
        if logview_url:
            logger.info(f"Final Logview: {logview_url}")

    except Exception as exec_error:
        logger.error(f"Execution failed: {str(exec_error)}")
        logger.error("Traceback:")
        traceback.print_exc()

        # Always print logview on error for diagnostics
        logview_url = session.get_logview_address()
        if logview_url:
            logger.error(f"Error Logview: {logview_url}")
            print(f"\n{'='*60}")
            print(f"ERROR: Execution failed")
            print(f"Logview URL: {logview_url}")
            print(f"{'='*60}\n")

        raise

except Exception as e:
    logger.error(f"Session error: {str(e)}")
    traceback.print_exc()
    raise

finally:
    try:
        session.destroy()
        logger.info("Session destroyed")
    except Exception as cleanup_error:
        logger.warning(f"Error during cleanup: {cleanup_error}")
```

### Common Error Patterns

#### 1. Authentication and Connection Errors
**Patterns:** "Invalid access key", "Authentication failed", "Connection refused"

**Solutions:**
- Verify environment variables are loaded correctly
- Check that `ODPS_ACCESS_ID` and `ODPS_ACCESS_KEY` are correct and not expired
- Ensure `ODPS_ENDPOINT` matches your region
- Verify `ODPS_PROJECT` name is accurate

#### 2. Table and Resource Not Found Errors
**Patterns:** "Table not found", "NoSuchTable", "Partition not found"

**Solutions:**
- Verify table name spelling and case sensitivity
- Check if table exists in the project
- Verify you have read permissions on the table
- For partitioned tables, check partition specification format

#### 3. Execution and Timeout Errors
**Patterns:** "Timeout", "Execution timeout", "Quota exceeded"

**Solutions:**
- Check logview URL for detailed execution status
- Review query complexity and optimize
- Consider using batch processing with `apply_chunk`
- Check MaxCompute resource quotas and limits

#### 4. Type and Schema Mismatch Errors
**Patterns:** "TypeError", "ValueError", "Column type mismatch"

**Solutions:**
- Check DataFrame dtypes: `df.dtypes` or `df.info()`
- Convert types explicitly: `df['col'] = df['col'].astype('int64')`
- Handle nullable types appropriately: `Int64`, `Float64`, `boolean`
- Check for None/NaN values before operations

#### 5. SQL and Query Errors
**Patterns:** "SQL syntax error", "Invalid SQL", "Column not found"

**Solutions:**
- Review the generated SQL query in logview
- Check column names for typos and case sensitivity
- Verify column names exist in the DataFrame
- Check for reserved keyword conflicts in column names

For detailed interactive coding, testing, and debugging guidance, see `references/interactive_coding.md`.

## Resources

### Agents

- **operator-selector** (at `../../agents/operator-selector.md`): Find, validate, and recommend MaxFrame operators. Use a subagent to run operator selection tasks to avoid loading large documentation in main context. This agent uses the `scripts/lookup_operator.py` script to search and retrieve operator information.

### Example Code

- **Working Examples**: `examples/` - 10 sample files covering groupby, DLF tables, complex structures, document similarity, GPU processing, and more.

### Reference Documentation

- **Local Debug Guide**: `references/local_debug.md` - Local debug mode for UDF development with IDE support, zero-configuration offline debugging
- **MaxFrame Context Guide**: `references/maxframe_context.md` - Comprehensive guide covering features, workflows, and best practices
- **Key Modules Reference**: `references/key_modules.md` - Reference for key modules with common operations
- **Operator Selection Rules**: `references/operator-selection-rules.md` - Rules for selecting optimal MaxFrame operators
- **Operator Selector Guide**: `references/operator-selector.md` - Agent guide with examples for finding, validating, and recommending MaxFrame operators
- **Interactive Coding Guide**: `references/interactive_coding.md` - Comprehensive guide for interactive coding, testing, debugging, SDK discovery, and error handling
- **MaxFrame Client Documentation**: `references/maxframe-client-docs/` - Generated API documentation (900+ files)
