Real models as google/owlvit-base-patch16 have non-contiguous weights in their state dict. This is for testing purposes.
