How do data, tensor, pipeline and expert parallelism combine?

The bottleneck

When one device cannot handle parameters, activations or compute, work must be distributed. DP partitions samples, TP tensors within layers, PP layers and EP experts. They can combine, but are not four independent multipliers.

The mechanism

TP frequently exchanges partial results within a layer and favors fast links. PP transfers activations between stages and fills pipelines with microbatches. DP synchronizes gradients or sharded states. EP dispatches and gathers routed tokens; in MoE, its layout often interacts with DP groups.

For a simple dense mesh: devices = DP × TP × PP

Examples and evidence

Example: 16 devices can form DP=2, TP=4, PP=2. Placing TP groups on fast local links is a starting hypothesis to benchmark. Do not blindly multiply by EP again. Megatron 2019 demonstrated intra-layer parallelism; the 2021 work studied combinations.

New constraints

PP has bubbles and stage imbalance, TP can be constrained by collectives, and EP by hot experts and all-to-all. Inspect memory, network topology, effective batch and fault recovery jointly rather than maximizing device count.