wkdrb2100@gmail.com

The quote has two A100 lines. 40GB and 80GB. The Tensor cell is 312 TFLOP/s on both. You pick as if only capacity doubled. Read the NVIDIA datasheet down the column and the split is memory. 40GB is HBM2 at 1,555 GB/s. 80GB SXM is HBM2e at 2,039 GB/s. 80GB PCIe is 1,935 GB/s. Core … Read more

You seat the card and load the model. nvidia-smi reads GPU-Util 80%. It looks busy. You close the window. Put the datasheet TFLOP/s next to the tokens that actually come out and the numbers disagree. A 312 TFLOP/s card delivers tokens at a few hundredths of that peak. The question is what the 80% was … Read more