IIRC the documentation around this is not great, but yes, size_allocate is called by containers to “assign” a size to its children. A container will call that method on its children every time it performs layout again. If your custom container has children then you will want calculate their sizes there, then call .size_allocate() on each of the children to give them a position. (But only if those children return should_layout() == true)
Also you will probably need to implement measure and maybe request_mode to perform the initial size negotiation, see Height-for-width Geometry Management for how that works. IMO if you are really curious about the details, the source for some of the adaptive containers in libadwaita is a good reference for what to do.
About Window and Box, there are actually two ways to implement the vfunc. One is to override it in the widget, the other is to add a layout manager. The layout manager takes precedence, so e.g. Box uses BoxLayout so it will always use the allocate implementation from that layout manager. Window does not have a layout manager but it is already a container, so probably you should not use that as a parent class if implementing a custom layout because anything you do can conflict with it. What you want to do is either implement a custom container widget (or implement a custom layout manager and add it to the widget) and then set it as the only child using window.set_child().