Deep learning is a promising, ultra-fast approach for inverse design in nano-optics, but despite fast advancement of the field, the computational cost of dataset generation, as well as of the training procedure itself remains a major bottleneck. This is particularly inconvenient because new data need to be generated and a new network needs to be trained for any modification of the problem. We propose a technique that allows to train a single neural network on a broad range of design targets without any re-training. The key idea of our method is to enrich existing data with random "regions of interest" (ROI) labels. A model trained on such ROI-decorated data becomes capable to operate on a broad range of physical targets, while it learns to focus its design effort on a user-defined ROI, ignoring the rest of the physical domain. We demonstrate the method by training a tandem-network on the design of dielectric core-shell nano-spheres for electric and magnetic dipole and quadrupole scattering over a broad spectral range. The network learns to tailor very distinct, flexible design targets like scattering due to specific multipoles in narrow spectral windows. Varying the design problem does not require any re-training. Our approach is very general and can be directly used with existing datasets. It can be straightforwardly applied to other network architectures and problems.