New HPC / AI servers based on the AMD Instinct MI300 series

LLAMA 2-70B Benchmark

This is a realistic inference benchmark for most use cases. AMD has a 40% latency advantage, which is logically explainable given the 60% bandwidth advantage over the H100. As the as yet unreleased H200 is close to the AMD GPU in terms of bandwidth, it is to be expected that the H200 will perform similarly. However, it is also to be expected that there will be a considerable price advantage in favour of the AMD architecture.

Bloom

In this benchmark, the Mi300X outperforms the H100 in terms of data throughput by a factor of 1.6. The benchmark is impressive, but possibly misleading. The model used for this benchmark is very large and a long input sequence was used. The system with the smaller memory (H100) is forced to work with a much smaller stack size as the KVCache takes up the entire memory capacity. The system with the larger memory (MI300) can use a larger stack size to utilise its computing power. While this is indeed a real advantage and the throughput-centred scenario is not theoretical but real, there are other scenarios where the performance gap is significantly smaller. *The stated values and categorisations refer to data published by AMD