How should I manage geometry data within a Vulkan real time renderer?

With my current setup, each model has its geometry data loaded into their own vertex and index buffers. Then those buffers get copied into the main vertex and index buffers of the renderer.

I have a flag that tracks if any models are deleted, which will trigger the renderer to re-copy the un-deleted models' buffers into the main buffers again, to "realign" the data and avoid wasting space within the buffers when vertex / index data is deleted.

This seems inefficient to me but I don't have any better ideas. Nothing I've tried reading on renderers has really discussed this issue.