---
name: alibabacloud-maxframe-runtime-image-creator
description: Generate custom Docker images for MaxFrame DPE runtime with user-specified Python libraries. Use when user needs custom runtime environment with additional dependencies, requests custom Python packages for MaxFrame, or wants to create a personalized DPE image. Generates Dockerfile and build instructions for custom MaxCompute DPE runtime.
license: Apache-2.0
metadata:
  author: maxframe-team
  version: "1.0"
compatibility: Requires Docker for building images, access to public Docker registries
---

# Create Custom MaxFrame DPE Runtime Image

This skill helps you create a custom Docker image for MaxFrame DPE (Data Processing Engine) runtime with your specified Python libraries.

## When to Use This Skill

Use this skill when you need to:
- Add custom Python libraries to MaxFrame runtime environment
- Create a personalized DPE image with specific dependencies
- Install packages not available in the standard MaxFrame runtime
- Configure a custom runtime for your MaxCompute jobs

## Workflow

### 1. Select Python Versions (Interactive)

**When you invoke this skill, use the `AskUserQuestion` tool to ask the user which Python versions to include:**

Present these options:
- **All versions (3.7-3.12)** - Maximum compatibility (default)
- **Recent versions (3.10-3.12)** - Good balance, smaller size
- **Python 3.11 only** - Recommended for production
- **Custom selection** - Let user specify exact versions

After the user selects, call the generation script with the appropriate `--python-versions` parameter.

**If running the script directly** (outside of skill invocation):
- The script will prompt interactively via command line
- Users can specify via `--python-versions` argument

### 2. Gather Requirements

The skill will ask you to provide:
- **Required libraries**: List of Python packages to install (e.g., `transformers`, `torch`, `pandas`)
- **Version constraints**: Specific versions if needed (e.g., `transformers==4.30.0`, `torch>=2.0`)
- **Image tag**: Custom tag for your Docker image (default: `maxframe-dpe-custom:latest`)

### 3. Generate Dockerfile

The skill will generate a Dockerfile based on:
- **Public base image**: Ubuntu 22.04, or 24.04 (no internal registry dependency)
- **Miniforge**: Conda-forge distribution for better package management
- **Multi-environment setup**: Isolated conda environments for each Python version
- **User packages**: Installed in all selected Python environments

**Note**: MaxFrame SDK and pyodps are provided by the DPE environment and are not included in the custom runtime image.

### 4. Build Instructions

The skill provides:
- Docker build command
- Testing instructions for each Python environment
- Push instructions for your registry
- Usage instructions for MaxFrame sessions

## Multi-Python Version Support

### Python Environments

This skill creates **isolated conda environments** for each selected Python version, matching the official DPE runtime pattern:

| Environment | Python Version | Use Case |
|-------------|----------------|----------|
| `py37` | 3.7 | Legacy compatibility |
| `py38` | 3.8 | Stable, aging |
| `py39` | 3.9 | Good balance |
| `py310` | 3.10 | Modern, stable |
| `py311` | 3.11 | **Recommended default** - Best performance |
| `py312` | 3.12 | Latest features |

### Interactive Selection

When you invoke the skill without `--python-versions`, you'll see:

```
Select Python Versions for Custom MaxFrame DPE Runtime
======================================================================

Available Python versions: 3.7, 3.8, 3.9, 3.10, 3.11, 3.12

Default: ALL versions (3.7-3.12)

Options:
  - Press Enter to include all versions (recommended)
  - Specify versions: e.g., '3.10,3.11,3.12'
  - Single version: e.g., '3.10'

Enter Python versions (or press Enter for all):
```

### Command Line Usage

```bash
# Interactive selection (default to all)
python3 scripts/generate_dockerfile.py \
    --packages transformers torch \
    --output-dir ./my-runtime

# Specify Python versions
python3 scripts/generate_dockerfile.py \
    --packages transformers torch \
    --python-versions 3.10,3.11,3.12 \
    --output-dir ./my-runtime
```

### Image Size Considerations

| Python Versions | Image Size | Build Time |
|----------------|------------|------------|
| All (3.7-3.12) | 3-5 GB | 10-20 min |
| 3 versions | 1.5-2.5 GB | 5-10 min |
| Single version | 0.8-1.2 GB | 2-5 min |

**Recommendation**: Select only the Python versions you need to minimize image size and build time.

## Critical Requirements

### MF_PYTHON_EXECUTABLE Environment Variable

**This is a MUST-APPLY rule enforced by MaxFrame.**

The `MF_PYTHON_EXECUTABLE` environment variable must be set to point to the Python executable in the default conda environment. MaxFrame uses this variable to detect the Python executable at runtime.

**Generated Dockerfile will include:**
```dockerfile
ENV CONDA_DEFAULT_ENV=py311
ENV MF_PYTHON_EXECUTABLE=/py-runtime/envs/py311/bin/python
```

**Path pattern:** `/py-runtime/envs/<env_name>/bin/python`

**Default environment selection:**
- Python 3.11 is preferred (if available)
- Otherwise, uses the highest selected Python version

**Why this is critical:**
- MaxFrame requires this to locate the Python interpreter for job execution
- Incorrect configuration will cause runtime failures
- Must point to conda environment's Python, not system Python

## Building from Public Images

This skill builds custom DPE runtime images from scratch using public images, requiring **no access to internal registries**.

**Key Components:**
- **Base**: Ubuntu 22.04 (public Docker Hub image)
- **Conda**: Miniforge (conda-forge distribution, better than Miniconda)
- **System**: OSS utilities, common tools, build dependencies
- **User packages**: Installed via conda in all selected environments

**Note**: MaxFrame SDK and pyodps are provided by the DPE server environment and are not installed in the custom runtime image.

**Advantages:**
- ✅ No internal registry access needed
- ✅ Reproducible builds anywhere
- ✅ Full control over base image
- ✅ Public and accessible to all users

The custom runtime includes:
- Python 3.11 environment (or selected versions)
- User-specified packages via conda
- System dependencies and build tools

**Note**: MaxFrame SDK and pyodps are provided by the DPE server infrastructure at runtime, not included in the custom image.

## Best Practices

### Conda vs System Packages

**System packages** (installed in system Python):
- System dependencies and build tools
- OSS utilities and common tools

**Conda packages** (installed in user environment):
- User-requested Python libraries
- ML/AI frameworks (PyTorch, TensorFlow, etc.)
- Specialized scientific computing packages

This separation ensures:
- Stable base environment
- Easy customization without conflicts
- Proper permission handling
- Reproducible builds

**Note**: MaxFrame SDK and pyodps are provided by the DPE server at runtime.

### Image Size Optimization

Generated Dockerfiles follow best practices:
- Multi-stage builds when beneficial
- Layer caching optimization
- Minimal base image size
- Cleanup of temporary files

### Security Considerations

- Uses public official images only (no internal registries)
- No hardcoded credentials
- Follows Docker security best practices
- Minimal attack surface

## Example Usage

### Basic Example

User request: "Create a DPE runtime with transformers and datasets libraries"

Generated Dockerfile will include:
```dockerfile
FROM ubuntu:22.04

# Install Miniforge and create Python environment
# Install user libraries via conda
RUN conda install -y -c conda-forge \
    transformers \
    datasets \
    && conda clean -afy
```

**Note**: MaxFrame SDK and pyodps are provided by the DPE server environment.

### Advanced Example

User request: "Create a DPE runtime with PyTorch 2.0, transformers 4.30, and huggingface-hub"

Generated Dockerfile will include:
```dockerfile
FROM ubuntu:22.04

# Install Miniforge and create Python environment
# Install specific versions via conda
RUN conda install -y -c pytorch -c conda-forge \
    pytorch=2.0 \
    transformers=4.30 \
    huggingface-hub \
    && conda clean -afy

# Configure environment
ENV TRANSFORMERS_CACHE=/tmp/transformers_cache
```

## Output Files

The skill generates:

1. **Dockerfile**: Custom Docker image definition
2. **README.md**: Build and usage instructions
3. **requirements.txt** (optional): Pinned dependency list for reproducibility
4. **.dockerignore**: Optimized ignore patterns for build context

## Next Steps After Generation

1. **Build the image**:
   ```bash
   docker build -t your-registry/maxframe-dpe-custom:v1 .
   ```

2. **Test locally** (optional):
   ```bash
   docker run --rm your-registry/maxframe-dpe-custom:v1 python -c "import transformers; print(transformers.__version__)"
   ```

3. **Push to registry**:
   ```bash
   docker push your-registry/maxframe-dpe-custom:v1
   ```

4. **Use in MaxFrame**:
   ```python
   from maxframe.session import new_session

   session = new_session(
       odps=odps_connection,
       image="your-registry/maxframe-dpe-custom:v1"
   )
   ```

## Troubleshooting

### Common Issues

**Build fails with conda errors**:
- Check package names and versions
- Ensure conda channels are correct
- Try without version constraints first

**Import errors in custom image**:
- Verify package installation path
- Check PYTHONPATH configuration
- Ensure conda environment is activated

**Image size too large**:
- Review included packages
- Use multi-stage builds
- Clean conda cache: `conda clean -afy`

**OSS utilities not working**:
- System packages should be in system Python path
- Check OSS credentials configuration
- Verify network connectivity

## Related Resources

For more information, see:
- `references/base_image_details.md` - Details about the base DPE runtime image
- `references/conda_best_practices.md` - Conda usage guidelines
- `assets/Dockerfile.template` - Template for custom Dockerfiles

## Support

If you encounter issues:
1. Check the generated README.md for specific instructions
2. Verify Docker is installed and running
3. Ensure you have network access to pull base images
4. Check conda package availability in conda-forge channel