fedora-infrastructure

#9462 Please add new CidrBlock to cluster/testing-farm
Closed: Invalid by mobrien. Opened by astepano.

Closed: Invalid
Reopen Issue

Hello,

Please add new CidrBlock to cluster/testing-farm VPC.
This VPC is actively used by fedora-ci for osci+testing-farms purposes.
Currently this VPC has:

aws ec2 describe-vpcs --vpc-ids vpc-0896aedab4753e76f
                    "CidrBlock": "10.123.0.0/24",

There are two EKS clusters: for osci + testing-farms.
Both EKS use the same VPC.
250 IPs for all resources in this VPC.
We need more IP.
Each VM, each container, each EKS node uses IP from this CDR.

Also, please add new-subnets based on new CidrBlock.
Current CidrBlock is divided to two sub-nets:

aws ec2 describe-subnets --filters "Name=subnet-id,Values=subnet-03089904253762f32,subnet-0b84fdcd88b5803c2"
            "CidrBlock": "10.123.0.0/25",
            "CidrBlock": "10.123.0.128/25",

This is very urgent, mainly because TFT + osci schedules more and more new tests for fedora.
Without IP we cannot requests new necessary compute-resources. Also this influences ELN process.

Add @msrb, @bookwar , @mvadkert

Yep, I can confirm this can start blocking us very soon

https://console.aws.amazon.com/vpc/home?region=us-east-1#subnets:search=testing-farm;sort=tag:Name

It is already blocking as, today we could not ask for new EKS node when load went high.

Screenshot_from_2020-11-12_12-43-51.png

Hi, please take a look, this really blocks us: 

  Normal   Scheduled               <unknown>           default-scheduler                     Successfully assigned jenkins/eln-periodic-5177-t8g1t-pxw8r-xl1l8 to ip-10-123-0-31.ec2.internal
  Warning  FailedCreatePodSandBox  67s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4ba36a1c6d64a795ef8766e21a3f43865ab84c5efd47f1ddbbe0a5d953da815b" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  65s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ffe9d9dab89e45d207d5d8551edc8a7edeeef3ffe075992a60db42827ec05a98" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  63s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f46e299122f3e26768d7de28f681fc0df36e73bcfbaffd400e47376e7ea71512" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  61s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f02ce2f07d85f017b61856b69c894e53160cfd64ec8d0bdd0166207707e6fdd9" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  59s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4adf2c0f58886cea1c2fa2cd2d44a3d601619fc0c3970dc684ae231826f2d85c" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  57s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "150ab3ece3fc93ee99bc5e1077cdae45608ad3b3f63fcbfa60f74fd9dc6aa8af" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  55s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "cd751cd9a90f82dc93b0cec0f1b293ce948a1e4a2829353b1e9d4b4eb38c8d4c" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  53s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "797c09471e97b2c23cb98297190ae747f30ac2efb35270bb19e048b5ee015ff0" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  51s                 kubelet, ip-10-123-0-31.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "07c323e6a0ee9a3001bee1881943515aed2c50e96eacb89b667bc328e5f99e0b" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container
  Normal   SandboxChanged          44s (x12 over 66s)  kubelet, ip-10-123-0-31.ec2.internal  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  43s (x4 over 48s)   kubelet, ip-10-123-0-31.ec2.internal  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ec4340280b799fb0c906c1610a07ea6bbec680a255636a52142cbfad5e6e1554" network for pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8": networkPlugin cni failed to set up pod "eln-periodic-5177-t8g1t-pxw8r-xl1l8_jenkins" network: add cmd: failed to assign an IP address to container

I do not have necessary privileges to do this:

aws ec2 associate-vpc-cidr-block \
    --vpc-id vpc-0896aedab4753e76f \
    --cidr-block 10.122.0.0/16
An error occurred (UnauthorizedOperation) when calling the AssociateVpcCidrBlock operation: You are not authorized to perform this operation.

I have attached a secondary CIDR block of 10.122.0.0/16 to the VPC.

Let me know if this works. I will leave the ticket open until you are satisfied that it does
Edited by mobrien

\o/ ty!

hmm, I do not see it on us-east-1 attached to vpc-0896aedab4753e76f ... @mobrien

This link show still show 2 subnets:
https://console.aws.amazon.com/vpc/home?region=us-east-1#subnets:search=testing-farm;sort=tag:Name

And both have just one CIDR block

 
aws ec2 describe-vpcs --vpc-ids vpc-0896aedab4753e76f
           "CidrBlockAssociationSet": [
                {
                    "AssociationId": "vpc-cidr-assoc-02fdb52f66d418775",
                    "CidrBlock": "10.123.0.0/24",
                    "CidrBlockState": {
                        "State": "associated"
                    }
                },
                {
                    "AssociationId": "vpc-cidr-assoc-05a88d5d8957452c7",
                    "CidrBlock": "10.122.0.0/16",
                    "CidrBlockState": {
                        "State": "associated"
                    }
                }
            ],

Cidr is attached to VPC. But need to add sub-nets now.

Screenshot_from_2020-11-12_18-33-41.png

I cannot create subnet:

aws ec2 create-subnet --vpc-id vpc-0896aedab4753e76f --cidr-block 10.122.1.0/17
An error occurred (UnauthorizedOperation) when calling the CreateSubnet operation: You are not authorized to perform this operation.

@mobrien could you please create 2 subnets too? Thank you very much for helping us.

Sorry I should have created those with the VPC.

They are created now with the cidr blocks 10.122.2.0/24 & 10.122.1.0/24

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: aws, ci, high-gain, low-trouble, ops

Metadata Update from @mobrien:
- Issue assigned to mobrien

@mobrien please, create two more subnets.
The reason is:

https://github.com/aws/containers-roadmap/issues/170

EKS cluster config cannot be extended with new subnets ^^^

It is only possible (I very hope for this hack): when you tag a new net + there is a new version available. AWS offers new version 1.8. But, imagine: some infra outage, and then burst of new tasks. We have rabbitmq queues that will save all messages. At that time its very possible that OSCI+TFT will collide again. I will take this two subnets exclusively for OSCI EKS cluster. Please create two more subnets for TFT.

I am really sorry, but limitation on demand adding new subnets to EKS is very boring.
Edited by astepano

https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/eni-and-ip-target.md

Hack with tagging subnet + update k8s to new version didn't help.

Unfortunately it is necessary to recreate the cluster to update the subnets [1].

With ip-caching technology and unpredictable using IPs from the one-subnet ignoring completely the second subnet, it is not possible to make two EKS to work in with subnet where is 128 IPS.

Such un-happy outcome :-(

[1] https://github.com/aws/containers-roadmap/issues/170

I opened a new ticket to create a new VPC: https://pagure.io/fedora-infrastructure/issue/9466.
Unfortunately it is necessary to recreate the cluster to update the subnets .

I will close this ticket in place of the new one created https://pagure.io/fedora-infrastructure/issue/9466

Metadata Update from @mobrien:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)
Metadata

aws ci high-gain low-trouble ops

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.