So you can't use SSL with Varnish?
You seem to be right, but in general, Varnish is focused on HTTP acceleration and caching, so it's not a bug, but a feature. In order to be able to use SSL with Varnish you need to add Nginx to your stack. Clients make requests to Nginx and then Nginx makes a request to Varnish.
This is a good article, that will help you: https://www.digitalocean.com/community/tutorials/how-to-configure-varnish-cache-4-0-with-ssl-termination-on-ubuntu-14-04