# MaxFrame Client Library - Comprehensive Context Guide

## Table of Contents

- [Overview](#overview)
- [Installation](#installation)
- [Getting Started](#getting-started)
  - [Session Management](#1-session-management)
- [DataFrame Operations](#dataframe-operations)
  - [Data Input/Output](#2-data-inputoutput)
  - [Execution Model](#3-execution-model)
  - [DataFrame Basics](#4-dataframe-basics)
  - [Data Transformation](#5-data-transformation)
  - [Window Functions](#6-window-functions)
  - [Missing Data Handling](#7-missing-data-handling)
  - [Advanced DataFrame Features](#8-advanced-dataframe-features)
  - [MaxFrame-Specific Extensions](#9-maxframe-specific-extensions)
- [Tensor Operations (NumPy-like)](#tensor-operations-numpy-like)
  - [Tensor Basics](#10-tensor-basics)
- [Machine Learning (Learn Module)](#machine-learning-learn-module)
  - [Learn Basics](#11-learn-basics)
- [Configuration and Options](#configuration-and-options)
  - [Global Options](#12-global-options)
- [Remote Execution](#remote-execution)
  - [Remote Functions](#13-remote-functions)
- [User-Defined Functions (UDFs)](#user-defined-functions-udfs)
  - [UDF Capabilities](#14-udf-capabilities)
- [Best Practices](#best-practices)
  - [Performance Optimization](#15-performance-optimization)
  - [Common Patterns](#16-common-patterns)
- [Error Handling](#error-handling)
  - [Common Issues](#17-common-issues)
- [Complete Example](#complete-example)
  - [End-to-End Example](#18-end-to-end-example)
- [API Compatibility](#api-compatibility)
  - [Pandas API Support](#19-pandas-api-support)
  - [MaxFrame-Specific Patterns and Workarounds](#20-maxframe-specific-patterns-and-workarounds)
- [Summary](#summary)
- [Quick Reference](#quick-reference)
- [Additional API Usage Examples from Benchmarks](#additional-api-usage-examples-from-benchmarks)

---

## Overview

**MaxFrame** is a distributed computational framework developed by Alibaba Cloud that provides pandas-compatible APIs for large-scale data processing on MaxCompute. It enables Python developers to parallelize their pandas code seamlessly in a distributed environment with minimal code changes.

### Key Features
- **Pandas API Compatibility**: Drop-in replacement for pandas with familiar APIs
- **Distributed Computing**: Automatic parallelization on MaxCompute clusters
- **Lazy Execution**: Optimized computation graphs for better performance
- **Multiple Module Support**: DataFrame, Tensor (NumPy-like), and Learn (scikit-learn-like)
- **Seamless Integration**: Works with MaxCompute tables and ODPS ecosystem

### Supported Python Version
- **Recommended**: Python 3.7+ (Python 3.7 recommended for all functionalities)
- Higher Python versions supported with ongoing development

---

## Installation

### Basic Installation

Install via pip:

```bash
pip install maxframe
```

### Install Latest Beta Version

```bash
pip install --pre maxframe
```

### Install from Source

```bash
pip install git+https://github.com/aliyun/alibabacloud-odps-maxframe-client.git
```

---

## Getting Started

### 1. Session Management

#### Creating a Session

A session is required to execute MaxFrame operations. It connects to MaxCompute and manages the execution lifecycle.

**Basic Session Creation:**

```python
import maxframe.dataframe as md
from maxframe.session import new_session
from odps import ODPS
import os

# Create ODPS connection
o = ODPS(
    os.getenv('ODPS_ACCESS_ID'),
    os.getenv('ODPS_ACCESS_KEY'),
    project='your-project-name',
    endpoint='your-endpoint',
)

# Create MaxFrame session
session = new_session(o)
```

**Session with ODPS Object:**

```python
# Directly pass ODPS object
session = new_session(o, default=True)
```

**Session Auto-Detection:**

```python
# In environments like DataWorks or MaxCompute Notebook
# ODPS credentials are automatically available
from maxframe import new_session

session = new_session()  # Auto-detects ODPS from environment
```

#### Session Operations

```python
# Set as default session
session.as_default()

# Get default session
from maxframe import get_default_session
current_session = get_default_session()

# Destroy session when done
session.destroy()

# Or use context manager
with new_session(o) as session:
    # Your operations here
    pass
```

#### Session Configuration

```python
# Session with custom parameters
session = new_session(
    o,
    session_id='custom-session-id',
    default=True,
    backend='maxframe'
)
```

---

## DataFrame Operations

### 2. Data Input/Output

#### Reading from MaxCompute Tables

**Basic Table Read:**

```python
import maxframe.dataframe as md

# Read entire table
df = md.read_odps_table("table_name")
```

**Reading Partitioned Tables:**

```python
# Read specific partitions
df = md.read_odps_table(
    "partitioned_table",
    partitions=["pt1=20240119,pt2=10", "pt1=20240119,pt2=11"]
)

# Include partition columns in result
df = md.read_odps_table(
    "partitioned_table",
    partitions=["pt1=20240119,pt2=10"],
    append_partitions=True
)
```

**With Index Specification:**

```python
# Specify columns as index
df = md.read_odps_table(
    "table_name",
    index_col=["idx_col1", "idx_col2"]
)
```

#### Reading from SQL Queries

```python
# Execute SQL and get DataFrame
df = md.read_odps_query(
    "SELECT a.shop_name AS ashop, b.shop_name AS bshop "
    "FROM sale_detail_jt a RIGHT OUTER JOIN sale_detail b "
    "ON a.shop_name=b.shop_name"
)
```

#### Writing to MaxCompute Tables

**Basic Write:**

```python
# Write DataFrame to table
md.to_odps_table(df, "output_table_name").execute()

# Without index
df.to_odps_table("output_table_name", index=False).execute()
```

**Writing to Partitioned Tables:**

```python
# Static partition
df.to_odps_table(
    "partitioned_table",
    partition="pt=20240121,h=12"
).execute()

# Dynamic partitioning (columns determine partitions)
df.to_odps_table(
    "partitioned_table",
    partition_col=["pt_col", "h_col"]
).execute()
```

#### Working with Pandas Objects

**Convert from Pandas:**

```python
import pandas as pd
import maxframe.dataframe as md

# Convert pandas DataFrame to MaxFrame DataFrame
pd_df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
md_df = md.read_pandas(pd_df)
# Or
md_df = md.from_pandas(pd_df)
```

**Convert to Pandas:**

```python
# Download data to local pandas DataFrame
pd_df = md_df.to_pandas()
```

#### Reading CSV and Parquet

```python
# Read CSV
df = md.read_csv("path/to/file.csv")

# Read Parquet
df = md.read_parquet("path/to/file.parquet")
```

---

### 3. Execution Model

#### Lazy Execution

MaxFrame uses lazy execution for optimization. Operations are not executed until explicitly requested.

```python
# This creates a computation graph but doesn't execute
df = md.read_odps_table("source_table")
filtered_df = df[df['A'] > 10]
result_df = filtered_df.groupby('B').sum()

# Nothing has been executed yet
```

#### Triggering Execution

**Explicit Execution:**

```python
# Execute and return the result
result_df.execute()
```

**View Results:**

```python
# Preview first/last rows (auto-executes)
result_df.head(10).execute()

# In interactive environments (Jupyter, IPython)
result_df.execute()  # Shows preview automatically
```

**Fetch Full Data:**

```python
# Execute and fetch all data
data = result_df.execute().fetch()
```

#### Asynchronous Execution

```python
# Non-blocking execution
execution_info = df.head(3).execute(wait=False)

# Check progress
progress = execution_info.progress()

# Wait for completion
execution_info.result()
```

#### Progress Tracking

```python
# Show progress bar during execution
df.execute(show_progress=True)

# Auto-detect terminal capabilities
df.execute(show_progress='auto')
```

---

### 4. DataFrame Basics

#### Creating DataFrames

**From Scratch:**

```python
import maxframe.dataframe as md
import pandas as pd

# Create from dictionary
df = md.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# From pandas
pd_df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
df = md.read_pandas(pd_df)

# Create Series
s = md.Series([1, 2, 3, 4], name='my_series')
```

**From Records:**

```python
# Create from records
records = [('Alice', 25, 'NY'), ('Bob', 30, 'CA')]
df = md.from_records(records, columns=['Name', 'Age', 'State'])
```

#### Basic DataFrame Operations

**Viewing Data:**

```python
# View structure
df.dtypes      # Column types
df.shape       # Dimensions
df.columns     # Column names
df.index       # Index

# View data
df.head(5).execute()
df.tail(10).execute()
```

**Selecting Columns:**

```python
# Single column
df['column_name']

# Multiple columns
df[['col1', 'col2', 'col3']]

# Drop columns
df.drop(columns=['col_to_drop'])
```

**Filtering Rows:**

```python
# Boolean indexing
df[df['age'] > 25]
df[df['city'] == 'New York']

# Multiple conditions
df[(df['age'] > 25) & (df['salary'] > 50000)]

# Between
df[df['age'].between(20, 30)]
```

**Adding/Modifying Columns:**

```python
# Create new column
df['new_col'] = df['col1'] + df['col2']

# Conditional assignment
df['category'] = df['age'] > 30
```

---

### 5. Data Transformation

#### Sorting

```python
# Sort by single column
df.sort_values('column_name')

# Sort by multiple columns
df.sort_values(['col1', 'col2'], ascending=[True, False])

# Sort by index
df.sort_index()
```

#### Aggregation Functions

```python
# Single aggregations
df['column'].sum()
df['column'].mean()
df['column'].min()
df['column'].max()
df['column'].std()
df['column'].count()

# DataFrame-wide
df.sum()
df.mean()
```

#### GroupBy Operations

**Basic GroupBy:**

```python
# Group by single column
grouped = df.groupby('category')

# Aggregate
result = grouped['value'].sum()
result = grouped.agg({'value': 'sum', 'count': 'mean'})
```

**Multiple Aggregations:**

```python
# Named aggregations
from maxframe.dataframe import NamedAgg

result = df.groupby('category').agg(
    total_sum=NamedAgg(column='value', aggfunc='sum'),
    avg_value=NamedAgg(column='value', aggfunc='mean'),
    count=NamedAgg(column='id', aggfunc='count')
)
```

**GroupBy with Multiple Columns:**

```python
# Group by multiple columns
result = df.groupby(['category', 'subcategory']).sum()
```

**Transformation:**

```python
# Apply transformation within groups
df.groupby('category')['value'].transform(lambda x: (x - x.mean()) / x.std())
```

**Apply Custom Functions:**

```python
# Apply custom function to groups
def custom_agg(group):
    return group['value'].max() - group['value'].min()

result = df.groupby('category').apply(custom_agg)
```

#### Merging and Joining

**Merge DataFrames:**

```python
# Inner join
merged = df1.merge(df2, on='key_column')

# Left join
merged = df1.merge(df2, on='key_column', how='left')

# Multiple keys
merged = df1.merge(df2, left_on='id', right_on='user_id')
```

**Join:**

```python
# Join on index
result = df1.join(df2)

# Join on specific columns
result = df1.join(df2, on='key')
```

**Concatenate:**

```python
# Vertical concatenation
result = md.concat([df1, df2, df3])

# Horizontal concatenation
result = md.concat([df1, df2], axis=1)
```

---

### 6. Window Functions

#### Rolling Window

**Basic Rolling:**

```python
# Rolling sum with window size 3
df['rolling_sum'] = df['value'].rolling(window=3).sum()

# Rolling mean
df['rolling_mean'] = df['value'].rolling(window=5).mean()
```

**With Parameters:**

```python
# Rolling with minimum periods
df['value'].rolling(window=3, min_periods=1).mean()

# Centered window
df['value'].rolling(window=3, center=True).mean()
```

**Time-based Rolling:**

```python
# Rolling over time period
df_timeseries.rolling('2D').sum()
```

**Supported Aggregations:**

```python
rolling_window = df['value'].rolling(window=5)

rolling_window.sum()
rolling_window.mean()
rolling_window.std()
rolling_window.var()
rolling_window.min()
rolling_window.max()
rolling_window.count()
```

#### Expanding Window

```python
# Expanding sum (cumulative)
df['expanding_sum'] = df['value'].expanding().sum()

# With minimum periods
df['expanding_mean'] = df['value'].expanding(min_periods=2).mean()
```

**Supported Operations:**

```python
expanding = df['value'].expanding()

expanding.sum()
expanding.mean()
expanding.std()
expanding.min()
expanding.max()
expanding.count()
```

#### GroupBy with Window Functions

**Rolling within Groups:**

```python
# Rolling by group
result = df.groupby('category')['value'].rolling(window=3).mean()
```

**Expanding within Groups:**

```python
# Expanding by group
result = df.groupby('category')['value'].expanding().sum()
```

---

### 7. Missing Data Handling

```python
# Check for missing values
df.isna()
df.isnull()

# Check for non-missing values
df.notna()
df.notnull()

# Fill missing values
df.fillna(0)
df.fillna(method='ffill')  # Forward fill
df.fillna(method='bfill')  # Backward fill

# Drop missing values
df.dropna()                  # Drop rows with any NaN
df.dropna(how='all')         # Drop rows where all values are NaN
df.dropna(subset=['col1'])   # Drop rows where specified columns have NaN
```

---

### 8. Advanced DataFrame Features

#### Pivot Operations

**Pivot Table:**

```python
# Create pivot table
pivot = md.pivot_table(
    df,
    values='value',
    index='category',
    columns='date',
    aggfunc='sum'
)
```

**Pivot:**

```python
# Reshape data
pivoted = df.pivot(index='id', columns='variable', values='value')
```

**Melt:**

```python
# Unpivot data
melted = md.melt(df, id_vars=['id'], value_vars=['col1', 'col2'])
```

#### Apply Functions

```python
# Apply function to each element
df['col'].apply(lambda x: x * 2)

# Apply to DataFrame
df.apply(lambda row: row['A'] + row['B'], axis=1)
```

#### String Operations

```python
# String accessor
df['text'].str.upper()
df['text'].str.lower()
df['text'].str.contains('pattern')
df['text'].str.replace('old', 'new')
df['text'].str.split(',')
df['text'].str.strip()
```

#### DateTime Operations

```python
# Convert to datetime
df['date'] = md.to_datetime(df['date_string'])

# DateTime accessor
df['date'].dt.year
df['date'].dt.month
df['date'].dt.day
df['date'].dt.dayofweek
df['date'].dt.hour
```

**Create Date Range:**

```python
# Generate date range
dates = md.date_range(start='2024-01-01', end='2024-12-31', freq='D')
```

#### Value Operations

```python
# Value counts
df['category'].value_counts()

# Unique values
df['column'].unique()

# Number of unique values
df['column'].nunique()

# Get dummies (one-hot encoding)
dummies = md.get_dummies(df, columns=['category'])
```

#### Binning

```python
# Cut into bins
df['age_group'] = md.cut(df['age'], bins=[0, 18, 30, 50, 100], 
                         labels=['child', 'young', 'middle', 'senior'])

# Quantile-based binning
df['quartile'] = md.qcut(df['value'], q=4)
```

---

### 9. MaxFrame-Specific Extensions

MaxFrame provides special extensions via the `.mf` accessor for advanced distributed operations.

#### apply_chunk

Apply functions on chunks/batches of data for better performance.

**DataFrame apply_chunk:**

```python
# Process DataFrame in batches
def process_batch(df_chunk):
    # Process chunk of DataFrame
    return df_chunk * 2

result = df.mf.apply_chunk(
    process_batch,
    batch_rows=1000,
    output_type='dataframe',
    dtypes=df.dtypes
)
```

**With Custom Parameters:**

```python
def custom_processing(df_chunk, multiplier, offset):
    return df_chunk * multiplier + offset

result = df.mf.apply_chunk(
    custom_processing,
    args=(10,),
    offset=5,
    batch_rows=500,
    output_type='dataframe',
    dtypes=df.dtypes
)
```

**Series apply_chunk:**

```python
def process_series_batch(series_chunk):
    return series_chunk ** 2

result = df['value'].mf.apply_chunk(
    process_series_batch,
    batch_rows=1000,
    output_type='series',
    dtype='float64'
)
```

**Machine Learning Inference Example:**

```python
from maxframe.udf import with_python_requirements

@with_python_requirements("scikit-learn")
def predict_batch(df_chunk):
    predictions = model.predict(df_chunk[['feature1', 'feature2']])
    return pd.Series(predictions, index=df_chunk.index)

predictions = df.mf.apply_chunk(
    predict_batch,
    batch_rows=100,
    output_type='series',
    dtype='float64',
    name='predictions'
)
```

#### MapReduce Pattern

```python
# Custom MapReduce operation
def mapper(df_chunk):
    # Map operation
    return df_chunk.groupby('key')['value'].sum()

def reducer(df_group):
    # Reduce operation
    return df_group.sum()

result = df.mf.map_reduce(
    mapper=mapper,
    reducer=reducer,
    group_cols=['key']
)
```

#### Rebalance Data

```python
# Rebalance data distribution across workers
balanced_df = df.mf.rebalance()
```

#### Extract Key-Value

```python
# Extract keys from map/dict columns
df['keys'] = df['map_column'].mf.extract_kv('keys')
df['values'] = df['map_column'].mf.extract_kv('values')
```

---

## Tensor Operations (NumPy-like)

### 10. Tensor Basics

MaxFrame provides NumPy-compatible tensor operations for distributed array computing.

```python
import maxframe.tensor as mt

# Create tensors
arr = mt.array([1, 2, 3, 4, 5])
zeros = mt.zeros((3, 4))
ones = mt.ones((2, 3))
random = mt.random.rand(5, 5)

# From DataFrame
tensor = mt.from_dataframe(df)
```

#### Tensor Operations

```python
# Arithmetic
result = arr1 + arr2
result = arr1 * arr2

# Linear algebra
result = mt.dot(matrix1, matrix2)
result = mt.matmul(a, b)

# Aggregations
total = arr.sum()
average = arr.mean()
maximum = arr.max()

# Reshape
reshaped = arr.reshape((2, 3))

# Indexing
subset = arr[0:5]
subset = arr[arr > 0]
```

---

## Machine Learning (Learn Module)

### 11. Learn Basics

MaxFrame provides scikit-learn-compatible APIs for distributed machine learning.

```python
import maxframe.learn as ml
```

#### Preprocessing

```python
from maxframe.learn.preprocessing import StandardScaler, LabelEncoder

# Scale features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Encode labels
encoder = LabelEncoder()
y_encoded = encoder.fit_transform(y)
```

#### Model Training

```python
from maxframe.learn.linear_model import LinearRegression

# Train model
model = LinearRegression()
model.fit(X_train, y_train)

# Predict
predictions = model.predict(X_test)
```

#### Metrics

```python
from maxframe.learn.metrics import accuracy_score, mean_squared_error

# Evaluate
accuracy = accuracy_score(y_true, y_pred)
mse = mean_squared_error(y_true, y_pred)
```

---

## Configuration and Options

### 12. Global Options

MaxFrame provides a configuration system for customizing behavior.

```python
from maxframe import options

# Execution mode
options.execution_mode = 'trigger'  # or 'eager'

# Progress display
options.show_progress = True  # or False, 'auto'

# Chunk size
options.chunk_size = 1024

# Session configuration
options.session.max_alive_seconds = 3600
options.session.max_idle_seconds = 600
options.session.quota_name = 'your_quota'

# DataFrame backend
options.dataframe.dtype_backend = 'numpy'  # or 'pyarrow'

# Retry configuration
options.retry_times = 4
options.retry_delay = 0.1
```

#### Option Context

```python
from maxframe.config import option_context

# Temporary options
with option_context({'show_progress': True, 'chunk_size': 2048}):
    # Operations with custom options
    df.execute()
```

#### Common Options

| Option | Default | Description |
|--------|---------|-------------|
| `show_progress` | `'auto'` | Show progress bar during execution |
| `execution_mode` | `'trigger'` | Execution mode: 'trigger' or 'eager' |
| `chunk_size` | `None` | Size of data chunks |
| `session.max_alive_seconds` | `259200` | Maximum session lifetime (3 days) |
| `session.max_idle_seconds` | `3600` | Maximum idle time (1 hour) |
| `session.quota_name` | `None` | MaxCompute quota to use |
| `dataframe.dtype_backend` | `'numpy'` | Backend for dtypes: 'numpy' or 'pyarrow' |
| `retry_times` | `4` | Number of retry attempts |

---

## Remote Execution

### 13. Remote Functions

Execute custom Python code remotely on MaxCompute.

```python
from maxframe import remote

# Define remote function
@remote.spawn
def my_function(x, y):
    return x + y

# Execute remotely
result = my_function(10, 20)
result.execute()
```

**Run Scripts Remotely:**

```python
from maxframe.remote import run_script

# Execute Python script on cluster
result = run_script(
    script_path='path/to/script.py',
    args=['arg1', 'arg2']
)
```

---

## User-Defined Functions (UDFs)

### 14. UDF Capabilities

MaxFrame supports custom Python functions with dependency management.

```python
from maxframe.udf import with_python_requirements

# Function with dependencies
@with_python_requirements("numpy", "scipy")
def process_data(df):
    import numpy as np
    return np.sqrt(df['value'])

# Apply UDF
result = df['value'].apply(process_data)
```

---

## Best Practices

### 15. Performance Optimization

1. **Use Lazy Execution**: Build computation graphs before execution
2. **Batch Operations**: Use `apply_chunk` for batch processing
3. **Appropriate Chunk Size**: Configure based on data size
4. **Minimize Data Movement**: Read and write to MaxCompute tables directly
5. **Use Built-in Functions**: Prefer MaxFrame functions over custom UDFs
6. **Partitioning**: Read only necessary partitions from tables

### 16. Common Patterns

**ETL Pipeline:**

```python
# Read -> Transform -> Write
df = md.read_odps_table("source_table", partitions=["pt=20240101"])
transformed = df[df['status'] == 'active'].groupby('category').agg({'value': 'sum'})
transformed.to_odps_table("target_table", partition="pt=20240101").execute()
```

**Data Quality Check:**

```python
# Check for nulls
null_counts = df.isnull().sum()

# Check duplicates
duplicates = df.duplicated().sum()

# Value ranges
summary = df.describe()
```

**Incremental Processing:**

```python
# Process new data only
new_data = md.read_odps_table(
    "source_table",
    partitions=[f"dt={today}"]
)

processed = new_data.groupby('key').sum()
processed.to_odps_table(
    "result_table",
    partition=f"dt={today}"
).execute()
```

---

## Error Handling

### 17. Common Issues

**Session Timeout:**

```python
# Increase session timeout
from maxframe import options
options.session.max_alive_seconds = 7200  # 2 hours
```

**Memory Issues:**

```python
# Reduce chunk size
options.chunk_size = 512
```

**Connection Issues:**

```python
# Increase retry attempts
options.retry_times = 10
options.retry_delay = 1.0
```

---

## Complete Example

### 18. End-to-End Example

```python
import os
import maxframe.dataframe as md
from maxframe import new_session, options
from odps import ODPS

# Setup
o = ODPS(
    os.getenv('ALIBABA_CLOUD_ACCESS_KEY_ID'),
    os.getenv('ALIBABA_CLOUD_ACCESS_KEY_SECRET'),
    project='my_project',
    endpoint='http://service.odps.aliyun.com/api'
)

# Configure options
options.show_progress = True
options.session.quota_name = 'my_quota'

# Create session
with new_session(o) as session:
    # Read data
    users = md.read_odps_table('users')
    orders = md.read_odps_table('orders', partitions=['dt=20240101'])
    
    # Transform
    # Filter active users
    active_users = users[users['status'] == 'active']
    
    # Join with orders
    user_orders = orders.merge(
        active_users[['user_id', 'name', 'email']],
        on='user_id'
    )
    
    # Aggregate
    daily_summary = user_orders.groupby('user_id').agg({
        'order_id': 'count',
        'amount': ['sum', 'mean']
    })
    # Flatten column names
    daily_summary.columns = ['order_count', 'total_amount', 'avg_amount']
    
    # Filter high-value users
    high_value = daily_summary[daily_summary['total_amount'] > 1000]
    
    # Write results
    high_value.to_odps_table(
        'high_value_users',
        partition='dt=20240101'
    ).execute()
    
    print("Processing complete!")
```

---

## API Compatibility

### 19. Pandas API Support

MaxFrame aims for high compatibility with pandas APIs. Most common pandas operations are supported:

**Fully Supported:**
- DataFrame/Series creation and basic operations
- Indexing and selection (loc, iloc, boolean indexing)
- GroupBy operations and aggregations
- Merge, join, concat operations
- String and datetime accessors
- Missing data handling
- Sorting and ranking
- Mathematical operations (mean, median, std, var, min, max, sum)
- Array operations (rank, unique, duplicated, factorize)
- Eval and query expressions
- Rolling and expanding window operations
- Categorical operations

**Partially Supported:**
- Some advanced groupby operations (nth, interpolation not available)
- Complex multi-index operations
- Certain time series operations
- Eval/query with custom parser/engine parameters

**Not Supported:**
- Operations requiring full data in memory on client
- Some specialized statistical functions
- Direct matplotlib integration (use `.plot()` methods)
- Pandas internal APIs (pd.core.algorithms, pd.core.nanops)
- Numexpr/Numba configuration (MaxFrame uses its own engine)
- Parser/engine parameters for eval (only default 'maxframe' parser)

Refer to the official documentation for detailed API compatibility matrices.

---

### 20. MaxFrame-Specific Patterns and Workarounds

#### Index Creation Pattern

MaxFrame's `md.Index` wraps pandas Index objects rather than creating them directly. When you need specific Index types (RangeIndex, CategoricalIndex, DatetimeIndex, MultiIndex), create them with pandas first, then wrap with `md.Index()`:

```python
import pandas as pd
import maxframe.dataframe as md

# Create pandas Index objects first
pd_range_index = pd.RangeIndex(0, 10000, 1, name="range_index")
pd_categorical_index = pd.CategoricalIndex(['A', 'B', 'C'] * 100, name="cat_idx")
pd_datetime_index = pd.DatetimeIndex(pd.date_range('2024-01-01', periods=100))
pd_multi_index = pd.MultiIndex.from_tuples([('A', 1), ('A', 2), ('B', 1)])

# Wrap with MaxFrame Index
mf_range_index = md.Index(pd_range_index)
mf_categorical_index = md.Index(pd_categorical_index)
mf_datetime_index = md.Index(pd_datetime_index)
mf_multi_index = md.Index(pd_multi_index)

# Use in DataFrame
df = md.DataFrame({'value': range(100)}, index=mf_range_index)
```

#### Eval and Query Limitations

MaxFrame supports `eval()` and `query()` methods but with some limitations:

```python
# ✅ Supported: DataFrame.eval() for column assignments
df.eval("new_col = col1 + col2")
df.eval("""
col3 = col1 * col2
col4 = col1 / col2
""")

# ✅ Supported: query() for boolean filtering
df.query("age > 25 and salary > 50000")

# ✅ Supported: local_dict parameter
x = 10
md.eval("df.A + x", local_dict={"df": df, "x": x})

# ✅ Supported: target parameter
md.eval("new_col = df.A + df.B", target=df)

# ❌ Not Supported: parser parameter (raises NotImplementedError)
# df.eval("A + B", parser='python')  # Will fail

# ❌ Not Supported: engine parameter (raises NotImplementedError)
# df.eval("A + B", engine='numexpr')  # Will fail

# Default behavior: MaxFrame uses its own parser='maxframe'
```

#### GroupBy Operations

Most groupby operations work as expected, with a few exceptions:

```python
# ✅ Supported: Standard aggregations
df.groupby('category').mean()
df.groupby('category').agg({'value': ['sum', 'mean', 'std']})

# ✅ Supported: Apply operations (use skip_infer for complex functions)
df.groupby('category').apply(
    lambda x: md.Series({'max': x['value'].max(), 'min': x['value'].min()}),
    skip_infer=True
)

# ✅ Supported: Parameters
df.groupby('category').mean(numeric_only=True)
df.groupby('category').sum(min_count=1)
df.groupby('category').std(ddof=0)

# ❌ Not Supported: nth() method
# df.groupby('category').nth(0)  # Not available

# ❌ Not Supported: quantile with interpolation parameter
# df.groupby('category').quantile(0.5, interpolation='linear')  # Not available
```

#### Statistical Operations

MaxFrame handles NaN values automatically in statistical operations:

```python
# All these operations handle NaN automatically (skipna=True by default)
df['column'].mean()      # Equivalent to pandas nanmean
df['column'].std()       # Equivalent to pandas nanstd
df['column'].var()       # Equivalent to pandas nanvar
df['column'].sum()       # Equivalent to pandas nansum
df['column'].median()    # Equivalent to pandas nanmedian

# Correlation and covariance
df.corr()               # Handles NaN automatically
df.cov()                # Handles NaN automatically

# No need to use pandas internal APIs like pd.core.nanops
# MaxFrame's public API methods handle NaN values correctly
```

#### Apply Operations with skip_infer

For complex apply operations, use `skip_infer=True` to improve performance:

```python
# For complex transformations that return Series
result = df.groupby('category').apply(
    lambda x: md.Series({
        'total': x['value'].sum(),
        'avg': x['value'].mean(),
        'count': len(x)
    }),
    skip_infer=True  # Skip type inference for better performance
)

# For Series apply with custom functions
result = df['column'].apply(
    lambda x: complex_function(x),
    skip_infer=True
)
```

#### Lazy Execution Best Practices

Always remember to call `.execute()` on MaxFrame objects:

```python
# ❌ Wrong: Result is not computed
result = df[df['age'] > 25].groupby('city').mean()
print(result)  # Shows computation graph, not data

# ✅ Correct: Execute to get results
result = df[df['age'] > 25].groupby('city').mean()
result = result.execute()  # Now result contains actual data
print(result)  # Shows the data

# ✅ Chain operations before executing
result = (df[df['age'] > 25]
          .groupby('city')
          .agg({'salary': 'mean', 'count': 'size'})
          .execute())
```

#### Working with Different Data Types

```python
# ✅ Supported: Basic dtype conversions
df['int_col'].astype('float64')
df['string_col'].astype('category')
df['int_col'].astype('string')

# ✅ Supported: Categorical operations
df['cat_col'].cat.categories
df['cat_col'].cat.codes
df['cat_col'].cat.add_categories(['new_cat'])
df['cat_col'].cat.remove_categories(['old_cat'])

# ⚠️ Limited: Nullable dtypes (Int64, Float64, boolean, string)
# These work but may have limited support compared to pandas
df['col'].astype('Int64')
df['col'].astype('Float64')
df['col'].astype('boolean')
```

#### Array Operations

MaxFrame supports common array operations through its public API:

```python
# ✅ Supported: Rank operations
df['column'].rank(method='average')
df['column'].rank(method='min')

# ✅ Supported: Unique and duplicated
df['column'].unique()
df['column'].duplicated()
df.drop_duplicates()

# ✅ Supported: Factorize
codes, uniques = md.factorize(df['column'])

# ❌ Don't use: Internal pandas APIs
# from pandas.core.algorithms import rank  # Don't use internal APIs
# Use MaxFrame's public methods instead
```

#### Performance Tips

1. **Batch Operations**: Use `.mf.apply_chunk()` for processing large datasets in batches
2. **Skip Infer**: Use `skip_infer=True` in apply operations when you know the output type
3. **Minimize Executions**: Build computation graphs and execute once rather than multiple times
4. **Use Built-in Functions**: MaxFrame's built-in aggregations are optimized for distributed execution
5. **Avoid Internal APIs**: Always use MaxFrame's public API methods

```python
# ✅ Good: Single execution of complex pipeline
result = (df
          .query("age > 25")
          .groupby('category')
          .agg({'value': ['sum', 'mean', 'std']})
          .execute())

# ❌ Bad: Multiple executions
filtered = df.query("age > 25").execute()
grouped = filtered.groupby('category').execute()
result = grouped.agg({'value': 'sum'}).execute()
```

---

## Summary

MaxFrame provides a powerful, pandas-compatible interface for distributed data processing on MaxCompute. Key takeaways:

1. **Easy Migration**: Minimal code changes from pandas
2. **Scalability**: Automatic distributed execution
3. **Familiar APIs**: Use pandas, NumPy, scikit-learn patterns
4. **Lazy Execution**: Optimized computation graphs
5. **MaxCompute Integration**: Seamless table read/write
6. **Extensibility**: Custom UDFs and extensions via `.mf` accessor

For advanced features and detailed API documentation, refer to the official MaxFrame documentation at https://maxframe.readthedocs.io

---

## Quick Reference

### Common Imports

```python
import maxframe.dataframe as md
import maxframe.tensor as mt
import maxframe.learn as ml
from maxframe import new_session, options, execute, fetch
from odps import ODPS
```

### Essential Commands

```python
# Session
session = new_session(odps_object)
session.destroy()

# Read
df = md.read_odps_table("table")
df = md.read_pandas(pd_df)

# Write
df.to_odps_table("table").execute()

# Execute
df.execute()
df.execute().fetch()

# Options
options.show_progress = True
options.chunk_size = 1024
```

---

## Additional API Usage Examples from Benchmarks

### Index Operations
```python
# Creating different types of indexes
import pandas as pd

# RangeIndex
pd_range_index = pd.RangeIndex(0, 10000, 1, name="range_index")
range_index = md.Index(pd_range_index)

# CategoricalIndex
categories = ["A", "B", "C"]
cat_data = ["A", "B", "C", "A", "B"]  # example data
pd_cat_index = pd.CategoricalIndex(cat_data, categories=categories, name="cat_index")
cat_index = md.Index(pd_cat_index)

# DatetimeIndex
dates = pd.date_range("2020-01-01", periods=1000, freq="D")
pd_datetime_index = pd.DatetimeIndex(dates, name="datetime_index")
datetime_index = md.Index(pd_datetime_index)

# MultiIndex
level1 = ["A", "B"] * 500
level2 = list(range(1000))
pd_multi_index = pd.MultiIndex.from_arrays([level1, level2], names=["first", "second"])
multi_index = md.Index(pd_multi_index)

# Common index operations
idx_unique = index.unique()
idx_sorted = index.sort_values()
idx_dropped = index.drop_duplicates()  # New API discovered in benchmarks
idx_to_frame = index.to_frame()       # New API discovered in benchmarks

# Index to dataframe conversion
df_from_index = index.to_frame(name="custom_name")
```

### Series Operations
```python
# Series creation
series = md.Series([1, 2, 3, 4, 5], name="values")

# Basic operations
result = series + 10
result = result * 2
result = result / 5

# Aggregations
mean_val = series.mean()
sum_val = series.sum()
min_val = series.min()
max_val = series.max()

# Statistical operations
median_val = series.median()
std_val = series.std()
var_val = series.var()
q25 = series.quantile(0.25)
q75 = series.quantile(0.75)

# Apply and map
applied = series.apply(lambda x: x ** 2)
mapped = series.map({1: "one", 2: "two", 3: "three"})

# String operations on series
string_series = md.Series(["hello", "world", "test"])
upper_strings = string_series.str.upper()
string_lengths = string_series.str.len()

# Value operations
value_counts = series.value_counts()
unique_vals = series.unique()

# Series to dataframe conversion
series_df = series.to_frame()
named_series_df = series.to_frame(name="custom_name")

# Groupby operations on series
grouped = series.groupby(level=0)
grouped_mean = grouped.mean()

# Rolling and expanding operations
rolling_mean = series.rolling(window=10).mean()
expanding_sum = series.expanding().sum()

# Sorting
sorted_series = series.sort_values()
sorted_by_index = series.sort_index()

# Missing value operations
missing_mask = series.isna()
not_missing_mask = series.notna()
filled_series = series.fillna(0)
dropped_na = series.dropna()
```

### DataFrame Operations
```python
# Create a sample dataframe
df = md.DataFrame({
    "int_col": [1, 2, 3, 4, 5],
    "float_col": [1.1, 2.2, 3.3, 4.4, 5.5],
    "category_col": ["A", "B", "A", "C", "B"],
    "group_col": ["Group1", "Group1", "Group2", "Group2", "Group1"]
})

# Describe operations
description = df.describe()
series_description = df["int_col"].describe()

# Apply operations
df_result = df.apply(lambda x: x["int_col"] * x["float_col"], axis=1)

# GroupBy operations
grouped = df.groupby("group_col")
agg_result = grouped.agg({"int_col": ["mean", "sum", "count"]})
transform_result = grouped["int_col"].transform(lambda x: x - x.mean())

# Merge operations
df2 = md.DataFrame({
    "int_col": [1, 2, 3],
    "additional_col": [10, 20, 30]
})
merged = md.merge(df, df2, on="int_col", how="inner")
left_joined = md.merge(df, df2, on="int_col", how="left")

# Head and tail operations
head_result = df.head(10)
tail_result = df.tail(10)

# Sample operations
sampled = df.sample(n=100, random_state=42)

# Query operations
filtered = df.query("int_col > 2 and float_col < 4.0")
complex_filtered = df.query("int_col > 30 and float_col > 80000 and score > 70")

# Cross-tabulation and pivot tables
crosstab_result = md.crosstab(df["category_col"], df["group_col"])
pivot_result = md.pivot_table(
    df,
    values="float_col",
    index="category_col",
    columns="group_col",
    aggfunc="mean"
)

# Melt operations
melted = md.melt(df, id_vars=["group_col"], value_vars=["int_col", "float_col"])
```

### Utility Functions
```python
# Binning operations
ages = md.Series([25, 35, 45, 55, 65])
age_groups = md.cut(ages, bins=3)  # Equal-width bins
age_groups = md.cut(ages, bins=[0, 30, 50, 100], labels=["Young", "Adult", "Senior"])

# Quantile-based binning
salaries = md.Series([30000, 50000, 70000, 90000, 120000])
salary_quartiles = md.qcut(salaries, q=4)  # Quartiles
salary_quartiles = md.qcut(salaries, q=4, labels=["Low", "Medium", "High", "Very High"])

# One-hot encoding
df_with_categorical = md.DataFrame({"department": ["IT", "HR", "Sales", "IT"]})
dummies = md.get_dummies(df_with_categorical, columns=["department"])
dummies_prefix = md.get_dummies(df_with_categorical, columns=["department"], prefix="dept")
dummies_dropped = md.get_dummies(df_with_categorical, columns=["department"], drop_first=True)

# Type conversion
string_numbers = md.Series(["100", "200.5", "300"])
numeric_series = md.to_numeric(string_numbers, errors="coerce")  # Invalid values become NaN
integer_series = md.to_numeric(string_numbers, downcast="integer")

# Expression evaluation
eval_df = md.DataFrame({
    "a": [1, 2, 3],
    "b": [4, 5, 6],
    "c": [7, 8, 9],
    "d": [10, 11, 12]
})
result_df = eval_df.eval("result = a + b")
eval_df = eval_df.eval("result = (a + b) * c / d")
flagged_df = eval_df.eval("flag = a > b")
multi_assign_df = eval_df.eval("""
x = a + b
y = c * d
""")

# Record operations
records = [{"id": i, "value": i * 10, "name": f"item_{i}"} for i in range(5)]
records_df = md.DataFrame.from_records(records)
dict_df = md.DataFrame.from_dict({
    "id": [1, 2, 3],
    "value": [10, 20, 30],
    "name": ["item_1", "item_2", "item_3"]
})

# Array utilities
unique_vals = md.unique(df["category_col"])
codes, uniques = md.factorize(df["category_col"])

# Missing value functions
missing_mask = md.isna(df["int_col"])
not_missing_mask = md.notna(df["int_col"])
```

### Advanced Operations
```python
# Categorical operations
cat_series = df["category_col"].astype("category")
categories = cat_series.cat.categories
codes = cat_series.cat.codes
is_ordered = cat_series.cat.ordered
cat_with_new = cat_series.cat.add_categories(["F", "G"])
cat_removed = cat_series.cat.remove_categories(["C"])

# String accessor operations (series)
text_series = md.Series(["Hello World", "MaxFrame Test", "Data Processing"])
lower_text = text_series.str.lower()
upper_text = text_series.str.upper()
text_len = text_series.str.len()
contains_pattern = text_series.str.contains("Hello")
replaced_text = text_series.str.replace("Hello", "Hi")
split_text = text_series.str.split(" ")
string_slice = text_series.str.slice(0, 5)

# Datetime accessor operations
dates = ["2021-01-01", "2021-02-15", "2021-03-20"]
datetime_series = md.to_datetime(md.Series(dates))
years = datetime_series.dt.year
months = datetime_series.dt.month
days = datetime_series.dt.day
weekdays = datetime_series.dt.weekday
quarters = datetime_series.dt.quarter

# Date range generation
date_range = md.date_range(start="2020-01-01", end="2024-12-31", freq="D")
monthly_range = md.date_range(start="2020-01-01", end="2024-12-31", freq="MS")
business_days = md.date_range(start="2020-01-01", end="2024-12-31", freq="B")

# Mathematical and statistical operations
series_with_nans = md.Series([1, 2, None, 4, 5])
mean_val = series_with_nans.mean()  # Automatically handles NaN
std_val = series_with_nans.std()    # Automatically handles NaN
sum_val = series_with_nans.sum()    # Automatically handles NaN

# Correlation and covariance
numeric_df = md.DataFrame({
    "A": [1, 2, 3, 4, 5],
    "B": [2, 4, 6, 8, 10],
    "C": [5, 3, 1, 4, 2]
})
corr_matrix = numeric_df.corr()
cov_matrix = numeric_df.cov()

# Skewness and kurtosis
skewness = numeric_df["A"].skew()
kurtosis = numeric_df["A"].kurtosis()
```

---

*This context document covers the essential aspects of MaxFrame from a user perspective. For the latest features and updates, always refer to the official documentation.*

