Issue #9462: Please add new CidrBlock to cluster/testing-farm - fedora-infrastructure

fedora-infrastructure

#9462 Please add new CidrBlock to cluster/testing-farm

Closed: Invalid 4 years ago by mobrien. Opened 4 years ago by astepano.

Hello,

Please add new CidrBlock to cluster/testing-farm VPC.
This VPC is actively used by fedora-ci for osci+testing-farms purposes.
Currently this VPC has:

aws ec2 describe-vpcs --vpc-ids vpc-0896aedab4753e76f
                    "CidrBlock": "10.123.0.0/24",

There are two EKS clusters: for osci + testing-farms.
Both EKS use the same VPC.
250 IPs for all resources in this VPC.
We need more IP.
Each VM, each container, each EKS node uses IP from this CDR.

Also, please add new-subnets based on new CidrBlock.
Current CidrBlock is divided to two sub-nets:

aws ec2 describe-subnets --filters "Name=subnet-id,Values=subnet-03089904253762f32,subnet-0b84fdcd88b5803c2"
            "CidrBlock": "10.123.0.0/25",
            "CidrBlock": "10.123.0.128/25",

This is very urgent, mainly because TFT + osci schedules more and more new tests for fedora.
Without IP we cannot requests new necessary compute-resources. Also this influences ELN process.

astepano commented 4 years ago

Add @msrb, @bookwar , @mvadkert

mvadkert commented 4 years ago

Yep, I can confirm this can start blocking us very soon

astepano commented 4 years ago

https://console.aws.amazon.com/vpc/home?region=us-east-1#subnets:search=testing-farm;sort=tag:Name

It is already blocking as, today we could not ask for new EKS node when load went high.

astepano commented 4 years ago

Hi, please take a look, this really blocks us:

  Normal   Scheduled               <unknown>           default-scheduler                     Successfully assigned jenkins/eln-periodic-5177-t8g1t-pxw8r-xl1l8 to ip-10-123-0-31.ec2.internal
  Warning  FailedCreatePodSandBox  67s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4ba36a1c6d64a795ef8766e21a3f43865ab84c5efd47f1ddbbe0a5d953da815b" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  65s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ffe9d9dab89e45d207d5d8551edc8a7edeeef3ffe075992a60db42827ec05a98" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  63s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f46e299122f3e26768d7de28f681fc0df36e73bcfbaffd400e47376e7ea71512" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  61s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f02ce2f07d85f017b61856b69c894e53160cfd64ec8d0bdd0166207707e6fdd9" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  59s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4adf2c0f58886cea1c2fa2cd2d44a3d601619fc0c3970dc684ae231826f2d85c" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  57s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "150ab3ece3fc93ee99bc5e1077cdae45608ad3b3f63fcbfa60f74fd9dc6aa8af" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  55s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "cd751cd9a90f82dc93b0cec0f1b293ce948a1e4a2829353b1e9d4b4eb38c8d4c" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  53s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "797c09471e97b2c23cb98297190ae747f30ac2efb35270bb19e048b5ee015ff0" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  51s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "07c323e6a0ee9a3001bee1881943515aed2c50e96eacb89b667bc328e5f99e0b" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Normal   SandboxChanged          44s (x12 over 66s)  kubelet, ip-10-123-0-31.ec2.internal  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  43s (x4 over 48s)   kubelet, ip-10-123-0-31.ec2.internal  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ec4340280b799fb0c906c1610a07ea6bbec680a255636a52142cbfad5e6e1554" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container

astepano commented 4 years ago

I do not have necessary privileges to do this:

aws ec2 associate-vpc-cidr-block \
    --vpc-id vpc-0896aedab4753e76f \
    --cidr-block 10.122.0.0/16

An error occurred (UnauthorizedOperation) when calling the AssociateVpcCidrBlock operation: You are not authorized to perform this operation.

mobrien commented 4 years ago

I have attached a secondary CIDR block of 10.122.0.0/16 to the VPC.

Let me know if this works. I will leave the ticket open until you are satisfied that it does

Edited 4 years ago by mobrien

mvadkert commented 4 years ago

\o/ ty!

mvadkert commented 4 years ago

hmm, I do not see it on us-east-1 attached to vpc-0896aedab4753e76f ... @mobrien

mvadkert commented 4 years ago

This link show still show 2 subnets:
https://console.aws.amazon.com/vpc/home?region=us-east-1#subnets:search=testing-farm;sort=tag:Name

And both have just one CIDR block

astepano commented 4 years ago

aws ec2 describe-vpcs --vpc-ids vpc-0896aedab4753e76f


           "CidrBlockAssociationSet": [
                {
                    "AssociationId": "vpc-cidr-assoc-02fdb52f66d418775",
                    "CidrBlock": "10.123.0.0/24",
                    "CidrBlockState": {
                        "State": "associated"
                    }
                },
                {
                    "AssociationId": "vpc-cidr-assoc-05a88d5d8957452c7",
                    "CidrBlock": "10.122.0.0/16",
                    "CidrBlockState": {
                        "State": "associated"
                    }
                }
            ],

Cidr is attached to VPC. But need to add sub-nets now.

astepano commented 4 years ago

I cannot create subnet:

aws ec2 create-subnet --vpc-id vpc-0896aedab4753e76f --cidr-block 10.122.1.0/17

An error occurred (UnauthorizedOperation) when calling the CreateSubnet operation: You are not authorized to perform this operation.

@mobrien could you please create 2 subnets too? Thank you very much for helping us.

mobrien commented 4 years ago

Sorry I should have created those with the VPC.

They are created now with the cidr blocks 10.122.2.0/24 & 10.122.1.0/24

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: aws, ci, high-gain, low-trouble, ops

4 years ago

Metadata Update from @mobrien:
- Issue assigned to mobrien

4 years ago

astepano commented 4 years ago

@mobrien please, create two more subnets.
The reason is:

https://github.com/aws/containers-roadmap/issues/170

EKS cluster config cannot be extended with new subnets ^^^

It is only possible (I very hope for this hack): when you tag a new net + there is a new version available. AWS offers new version 1.8. But, imagine: some infra outage, and then burst of new tasks. We have rabbitmq queues that will save all messages. At that time its very possible that OSCI+TFT will collide again. I will take this two subnets exclusively for OSCI EKS cluster. Please create two more subnets for TFT.

I am really sorry, but limitation on demand adding new subnets to EKS is very boring.

Edited 4 years ago by astepano

astepano commented 4 years ago

https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/eni-and-ip-target.md

astepano commented 4 years ago

Hack with tagging subnet + update k8s to new version didn't help.

Unfortunately it is necessary to recreate the cluster to update the subnets [1].

With ip-caching technology and unpredictable using IPs from the one-subnet ignoring completely the second subnet, it is not possible to make two EKS to work in with subnet where is 128 IPS.

Such un-happy outcome :-(

[1] https://github.com/aws/containers-roadmap/issues/170

astepano commented 4 years ago

I opened a new ticket to create a new VPC: https://pagure.io/fedora-infrastructure/issue/9466.
Unfortunately it is necessary to recreate the cluster to update the subnets .

mobrien commented 4 years ago

I will close this ticket in place of the new one created https://pagure.io/fedora-infrastructure/issue/9466

Metadata Update from @mobrien:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)

4 years ago