InternEvo

Environment Setup

• Environment Installation
  • Environment Preparation
  • Installation through pip
  • Installation through Source Code
  • Environment Image
  • NPU Environment Installation

Quickstart Guide

• Quickstart Guide
  • Installation
  • Data Preparation
  • Training Configuration
  • Start Training
  • Training Results
  • Load the training checkpoint and generate.
  • Long Text Generation

Model Setup

• Training Initialization
  • Argument Parsing
  • Model Initialization
  • Dataloader Initialization
  • Parallel Communication Initialization
  • Optimizer Initialization
  • Trainer Initialization

• Script of Starting Training
  • Configuration Parameter Parsing
  • Initialization process
  • Start Training Process

Parallel Training

• Parallel Mode and Principle
  • Tensor Parallel
  • Pipeline Parallel
  • Data Parallel
  • ZeRO1.5
  • 2D-Attention

Data Format

• Data Load and Procedure
  • Daterloader Loading Data
  • Achieve Data From Dataloader
  • During the Forward process, the data format is:

Mixed Precision

• Mixed Precision Training
  • Implementation Instructions
  • TF32 Training

Mixture-of-Experts

• Mixture-of-Experts
  • Parameter Settings
  • Model Training

Model Checkpointing

• Model Checkpointing
  • CheckpointManager
  • Model loading and saving path format conventions.
  • Asynchronous upload.
  • Snapshot Checkpoint
  • Checkpoint automatic recovery
  • Manual control of checkpoint storage

Profiler

• Performance Analysis Tool
  • Torch Profiler
  • Memory Profiler

Monitor

• Monitor and Alert
  • Monitoring
  • Alerting
  • Light Monitoring

Example

• Training Example Introduction
  • 7B Demo
  • 20B Demo

Q&A

• Q&A

Indices and tables

• Index

• Module Index

• Search Page