#8109 Fedora Magazine seems to be down
Closed: Fixed 4 years ago by kevin. Opened 4 years ago by asamalik.

The Fedora Magazine doesn't load for me.

Based on a very quick IRC chat, it might be the proxies:

19:13 <asamalik> is the Fedora Magazine down? http://fedoramagazine.org
19:16 <sub_pop> It's hanging for me.
19:17 <jlanda> works on my side
19:18 <jlanda> seems that some of our proxies can't reach the actual backend server

Could someone please have a look at that?


Please try again now and see if you still get any issues?

The problem is that one of our proxies cannot reach it. I disabled it from dns yesterday, but I only disabled the ipv4 address, so I suspect you were hitting the ipv6 one. I just disabled that now.

Metadata Update from @kevin:
- Issue assigned to kevin
- Issue priority set to: Waiting on Assignee (was: Needs Review)

4 years ago

It works now! Thanks for looking into it!

@misc any idea why proxy08.fedoraproject.org would not be able to reach the magazine?

Nope, wpengine is hosted on GCP so I guess a issue between GCP and Colocation America Corp or something. I can try to hit their support, but I am not root there, and can't run any diagnostic or anything (like tcpdump).

Routing seems to work since I can ping from the proxy, so I hope that's not some weird IDS or something that got triggered from the use of the proxy.

So we have another user reporting on #fedora-devel, for proxy02. I just connected and curl seems to work ok.

19:13:57|  mhroncok> cverna: is fedoramagazine.org healthy?
19:14:22|  mhroncok> cverna: keeps rolling for me and a friend told me they've got error 500
19:21:52|  misc> mhh, people were reporting issue with a specific proxy
19:22:25|  misc> mhroncok: what proxy are you hitting ?
19:22:31|  misc> (like, what IP)
19:22:57|  mhroncok> misc: 2001:4178:2:1269::fed2

so, kinda wonder if chromium is not hitting proxy08. Or it could be a caching issue somewhere on DNS.

I've tried to clean chromium's dns cache. it didn't help right away, but in couple minutes, I can load it again.

So wpengine told that we have triggered a IDS, as I suspected, due to the same IP doing "bad" stuff (failed login, etc).

We can whitelist IP, but only 5 of them, for the whole account. So we have to move out of the proxy setup, which was planned anyway, but that mean transfering the ssl certificates and DNS, etc, etc.

Ok so after sleeping on the issue, I realized a few things:
- the blog for fedora magazine is using fedoramagazine.org, which is a dedicated domain name, so we can transfer the certificate.

But commblog, hosted at the same IP do not use a dedicated domain, but use a shared wildcard one, so if we want to transfer, we would need to get the certiicate (and private key) of the wildcard certificate. This would be somehow a security problem, so we need a new solution.

So we have 2 choice:
- accept that we have down time of 5m to 1h with that
- try to get a letsencrypt certificate using the DNS challenge, then transfer it to WPEngine, then change the DNS, and then we can do the regular http01 challenge

Since fedora magazine is easier and more urgent (cause people report problem, while no one told anything about commblog, despites having the same issue), I will work on fedora magazine first (likely this weekend).

I am going to contact folks for the commblog issue.

So, I decided to test the plan for Fedora magazine, and it should work. (even if it took longer than expected, cause I need to get support for adding a SSL cert). Please ping me if stuff broke, cause while, in a true devops fashion, it work on my laptop, doesn't mean it work for others.

This should be all done? The ip is moved and no longer hitting our proxies...

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

4 years ago

Login to comment on this ticket.

Metadata
Attachments 1
png
Attached 4 years ago View Comment