From the stand up today

[14:01:27] <smooge> so this one i would like to evaluate with mobrien when he is back from PTO

TL;DR
In AWS a subnet can only be in one availability zone. Use multiple subnets

There is a bit of guesswork here but it looks like all of the copr instances are using vpc-0afefac8bae905972 and the subnet subnet-0995f6a466849f4c3 this is why all the instances are landing in us-east-1c.

There are 6 subnets in this VPC each one in a different AZ

• subnet-01d4e967ab5e78005 (us-east-1d)
• subnet-05437ac82d63b6ef5 (us-east-1f)
• subnet-07b0b3168a353e3ee (us-east-1e)
• subnet-08cadf5a14b530ac4 (us-east-1b)
• subnet-0995f6a466849f4c3(us-east-1c)
• subnet-09c74a3e6420a206b(us-east-1a)

The idea would be to spread the instances across the subnets.
If you use ansible you could use this filter to spread instances randomly across the subnets by providing it a list and letting it choose. https://docs.ansible.com/ansible/latest/plugins/lookup/random_choice.html

One thing to note is that not all AZ's are equal, there can be different instance availability between zones.

It is also difficult to determine which ones have this availability without trial and error as the naming is unique per account so us-east-1a in this account is likely a different AZ to us-east-1a in another account.

i believe AWS do this to load balance and avoid everyone just filling up the same az.

One other thing to note is that if you use something like a NAT gateway that will also be set up in a single AZ so even though instances from different AZ's use it if the AZ goes down it will cause a single point of failure. I don't really know your architecture to spot any other points where this may occur.

Thank you for the sum-up, @mobrien .

I agree, based on the docs it is clear that we have to specify either the subnet id or configure a default subnet id (there's no concept like "random default subnet" option):
https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RunInstances.html

We can pick random subnet by ansible pretty easily, thank you for the tip.

These parts should do what @mobrien suggested:

https://pagure.io/fedora-infra/ansible/blob/3a17e3698a32c5428a1fc12581750008c2206edb/f/roles/copr/backend/files/provision/spinup_aws_task.yml#_3-5

https://pagure.io/fedora-infra/ansible/blob/3a17e3698a32c5428a1fc12581750008c2206edb/f/roles/copr/backend/templates/provision/aws_cloud_vars.yml#_12-27

We can close this ticket I think.

That looks good to me @praiskup and should solve the issue