When the pod in OpenShift crashes there is currently no way for the maintainer of that OpenShift project to know (except for crashloops). It would be nice to have e-mail sent (ideally with logs) to maintainer each time the pod crashes and being restarted. For crashloops it should be enough to get just a summary with the log from first crash.
Not urgent, just a nice thing to have for openshift apps.
CC: @dkirwan
Metadata Update from @kevin: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops
It's possible to determine if a pod has crashed/restarted a number of times within a short period etc. We can create PrometheusRule objects which track such things. We can target specific triggers using the severity field [1].
Alertmanager within the cluster can be configured to do something custom when an alert fires with a custom severity, eg: the trigger can be configured to send email to an email list containing the admins of the openshift-app etc.
Lets not attach logs to emails, the alert emails can link receivers back to the openshift web console, where they must first authenticate before being able to read the logs etc.
This sounds like something that could be done. Do we have mailing list that contains owners of openshift-apps? Could we instead use e-mail that would be configured somewhere in OpenShift project directly?
Link instead of attached docs sounds fine, how long till the link will expire and the log will be hard to find? For restarting pod you don't usually have access to old logs, just the current and previous run, which doesn't help :-/
Yep think we have an easy way of configuring app owners groups to have an email list. Eg we use this feature for the openshift-sysadmin group currently to receieve all alerts on the clusters.
openshift-sysadmin
The link I'm thinking of would be just to the project. Users with the permissions to log in and access the pods would know which pod is having trouble via the original alert, then should be able to retrieve the logs from the crashing pods [1]. As to how long they stick around or we might lose the original error, yep could and does happen! There are many solutions we could look at here depending on how much resources we want to throw at it.
I think another list might be overkill here. Openshift knows the app owners by username, couldn't the alert just send all those usernames @fedoraproject.org ?
I'll have to double check if its possible to send multiple emails an alert at the same time, might be possible!
@dkirwan Did you have time to check this?
i think we have a way to do this, sending same alert for multiple receivers through prometheus .. @dkirwan @kevin right ?
sorry folks was on PTO missed all these pings. Yep definitely possible to setup multiple receivers, and it seems to be possible to send multiple email addresses in one receiver too [1].
Think you have the basic technical requirements figured out, but would be good to get a POC in place, and document the process that other app owners can then follow to begin implementing.
Basically then need:
for each app, we need to create a receiver which listens for that unique severity created previously for prometheus/alertmanager, and add the list of emails which expect alerts.
[1] https://github.com/davidkirwan/asset_monitoring/blob/master/kubernetes/prometheusrules.yaml#L19
@darknao worked on this and came up with some simple alerting rules that we can generically apply to all applications. ;)
This should alert/send emails to appowners (or to alert_users if that is set).
It should alert on cronjobs failing, pods crashing, etc. (See ./roles/openshift/project/templates/prometheusRules.yml ).
I am going to mail the infra list a heads up about this and then run playbooks for at least the projects we manage.
Thanks @darknao!
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.