Optimization (3 blogmarks)

In the Inference Engineering episode from the latent.space podcast, they referred to NVFP4 a bunch of times. That’s a quantization method. Quantization is the process of optimizing an LLM’s parameters so that it can fit better and perform better on broader sets of hardware. Quantization reduces the memory footprint of a model by finding opportunities to use lower-precision floating point values without sacrificing too much fidelity.

NVFP4 is a 4-bit floating-point format for NVIDIA Blackwell GPUs. It reduces memory bandwidth and storage for inference while keeping accuracy close to higher-precision formats.

Unlike uniform INT4 quantization, NVFP4 keeps floating-point semantics with a shared exponent and a compact mantissa, which improves dynamic range. Blackwell Tensor Cores support mixed-precision execution across FP16, FP8, and FP4, so models can use FP4 for weights and activations while accumulating in higher precision (typically FP16).

I had no idea Ruby hashes had this much of a memory footprint (relative to struct).

It’s super impressive to see the small steady improvements the Ruby core team / contributors are able to squeeze out by doing hyper-specific, low-level optimizations.

Reading this post reminded me of some of the Aaron Patterson talks I’ve seen.

I don't have much practical use for this kind of under-the-hood perf optimization analysis in my day-to-day. However, I do find it interesting to see the bits of Ruby code one can use to pull out these kinds of memory footprint numbers -- primarily ObjectSpace.memsize_of.

Here is some of my own basic fiddling:

> require 'objspace'
=> true
> ObjectSpace.memsize_of(1)
=> 0
> ObjectSpace.memsize_of("Hello, World!")
=> 40
> ObjectSpace.memsize_of([1,2,3])
=> 40
> ObjectSpace.memsize_of({[1,2,3] => :abc})
=> 160

And here is a snippet of measuring code from the post:

require 'objspace'
puts "Ruby: #{RUBY_VERSION}"

11.times do |size|
  struct_class = size.zero? ? Object : Struct.new(*size.times.map { |i| :"m_#{i}" })
  struct = ObjectSpace.memsize_of(struct_class.new)
  hash = ObjectSpace.memsize_of(Hash[size.times.map { |i| [i, i] }])
  diff = (hash.to_f / struct).round(1)
  puts "size: #{size} \tstruct: #{struct} \thash: #{hash} \tdiff: #{diff}x"
end

A mini-game used to disguise an interactive learning resource on optimal path finding and The Traveling Salesman Problem.