Add support for wildcard hosts

[RealOrangeOne]

It's often necessary to support multiple hosts for a single Django application. This PR reuses the implementation for ALLOWED_HOSTS in allowing ROBOTS_ALLOW_HOSTS to specify multiple hostnames to allow.

This change is backwards compatible.

[j4mie]

Sorry for the massive delay in responding to this! If I understand correctly, this is to support a single Django application which serves content under multiple hostnames, and you want to allow robots access to some hostnames but disallow others?

To be honest I feel like that is probably slightly stretching the definition of "simple", and that you should probably just implement this in your application. Changing the core setting for the project makes me a bit twitchy, backwards-compat or not.

Let me know if I've misunderstood the use case..

[RealOrangeOne]

Nope, that's exactly it! Having 1 django app serve multiple hosts is quite common, whether they serve different content or the same.

I agree it stretches the definition slightly, but for a low-overhead and generally useful feature (I realise that's literally the definition of scope creep...).