You haven’t really specified anything about how you’re setting the font size, or what kind of API are you using, or what you’re trying to achieve, really.
Care to explain a bit better, and provide some more context?
Note that using Cairo directly is a bad idea. Your application will not support complex text, so it will not be possible to localize it properly in many locales.
To be clear: cairo supports simple text (Latin scripts, and I think maybe Cyrillic scripts?) but it will not work properly for complex text (Hebrew, Arabic, Indic scripts, CJKV, etc.) Whether that matters for your embedded application, I don’t know, but it would be unacceptable to use it in GNOME applications.
For cairo docs it is hard to understand what textsize exactly means, so maybe it would be good to ask on cairo mailing list for clarification. Text size is always difficult, there is the pointsize, which is 1/72 inch, there is pixel size, and there is em size, there is height of lower and upper case characters and extend of some characters below text base line. I hope this is well defined for Pango, as I have myself an app in early stage which will need exact text dimensions, i.e. a text with exactly 10mm in height. But I will care for that details much later (GitHub - StefanSalewski/SDT: A simple Nim GTK4 drawing and design tool)
When you are creating an embedded GUI, and you are using C or C++ as programming language, then you may also consider Blend2d. It is much faster than cairo and much more active.
I already use blend2d, but since i can’t get no satisfaction with the embedding techniques, i need to experiment starting again and again. So, this time, prototyping, i’m using cairo because:
it’s already in the system → zero setup
it maps fonts → no additional code for FontConfig required
it has this simple (toy) api → probably less code to measure the glyphs