If you control the domain then yes you could. But if I want to put a link on my website to some SVG hosted elsewhere and I want it to be safe for you to open that link in a new tab then there's not really a way for CSP to protect you the user from the host deploying a malicious SVG.
Like opening a PNG in a new tab is harmless but opening an SVG in a new tab is opening a pretty substantial can of worms.
If your threat model is “I don’t want the image I’m hotlinking to be replaced with something else when opened in a new tab”, then no image format is safe.
If you control the domain then yes you could. But if I want to put a link on my website to some SVG hosted elsewhere and I want it to be safe for you to open that link in a new tab then there's not really a way for CSP to protect you the user from the host deploying a malicious SVG.
Like opening a PNG in a new tab is harmless but opening an SVG in a new tab is opening a pretty substantial can of worms.
If your threat model is “I don’t want the image I’m hotlinking to be replaced with something else when opened in a new tab”, then no image format is safe.