fedora-infra / ansible

Clone
Source Code
GIT

#777 Clean up the deployment of the simple_message_to_bus script
Merged 3 years ago by nphilipp. Opened 3 years ago by nphilipp.
fedora-infra/ nphilipp/ansible main--simple-message-to-bus-cleanup  into  main

file modified
+10 -4 
@@ -1,13 +1,19 @@ 
 

- #!/bin/sh
 

- MSGTOPIC_PREFIX="${MSGTOPIC_PREFIX:-logging.stats}"
 

- topic="${MSGTOPIC_PREFIX}.$1"
 

+ #!/bin/bash
 

+ 

+ # To set a common prefix for all messages, do `export MSGTOPIC_PREFIX=...` in callers.
 

+ if [ -n "$MSGTOPIC_PREFIX" -a "${MSGTOPIC_PREFIX%.}" = "$MSGTOPIC_PREFIX" ]; then
 

+     MSGTOPIC_PREFIX="${MSGTOPIC_PREFIX}."
 

+ fi
 

+ topic="${MSGTOPIC_PREFIX}$1"
 

  shift
 

  sent_at="$(TZ=UTC date -Iseconds)"
 

  id="$(uuidgen -r)"
 

  body="{"
 

  sep=""
 

  

- for body_piece; do
 

+ # To preset items in the message body, do `export MSGBODY_PRESET="key1=value1 key2=value2"` before
 

+ # calling simple_message_to_bus. This doesn't work with spaces in the keys or values, sorry.
 

+ for body_piece in $MSGBODY_PRESET "$@"; do
 

      key_type="${body_piece%%=*}"
 

      key="${key_type%%:*}"
 

      type="${key_type#${key}}"

file added
+41 
@@ -0,0 +1,41 @@ 
 

+ #####################################################
 

+ # Ensure PATH in /etc/crontab contains /usr/local/bin
 

+ #####################################################
 

+ 

+ - name: check if PATH is set in crontab
 

+   lineinfile:
 

+     path: /etc/crontab
 

+     state: absent
 

+     regexp: '^PATH\s*='
 

+   check_mode: yes
 

+   changed_when: false
 

+   register: path_set_in_crontab
 

+ 

+ - name: add PATH if not set in crontab
 

+   lineinfile:
 

+     path: /etc/crontab
 

+     state: present
 

+     insertbefore: BOF
 

+     line: 'PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin'
 

+     validate: crontab -T %s
 

+   when: not path_set_in_crontab.found
 

+ 

+ - name: check if PATH in crontab contains /usr/local/bin
 

+   lineinfile:
 

+     path: /etc/crontab
 

+     state: absent
 

+     regexp: '^PATH\s*=\s*(.*:)?/usr/local/bin(:.*)?\s*'
 

+   check_mode: yes
 

+   changed_when: false
 

+   register: local_in_path_in_crontab
 

+   when: path_set_in_crontab.found
 

+ 

+ - name: append /usr/local/bin to PATH in crontab if missing
 

+   lineinfile:
 

+     path: /etc/crontab
 

+     state: present
 

+     backrefs: yes
 

+     regexp: '^(PATH\s*=\s*(?:.*\S)?)\s*$'
 

+     line: '\1:/usr/local/bin'
 

+     validate: crontab -T %s
 

+   when: path_set_in_crontab.found and not local_in_path_in_crontab.found

file modified
+6 
@@ -2,3 +2,9 @@ 
 

    copy: src=simple_message_to_bus dest=/usr/local/bin/ mode=0755
 

    tags:
 

      - fedora-messaging
 

+ 

+ - name: ensure PATH in crontab contains /usr/local/bin
 

+   import_tasks: crontab_path.yml
 

+   tags:
 

+     - fedora-messaging
 

+     - cron

file modified
+10 -9 
@@ -19,8 +19,9 @@ 
 

  # along with Fedora Project Infrastructure Ansible Repository.  If
 

  # not, see <http://www.gnu.org/licenses/>.
 

  

- RUN_ID="$(uuidgen -r)"
 

- simple_message_to_bus combinehttplogs.start run_id="$RUN_ID"
 

+ export MSGTOPIC_PREFIX=logging.stats
 

+ export MSGBODY_PRESET="loghost=$(hostname) run_id=$(uuidgen -r)"
 

+ simple_message_to_bus combinehttplogs.start
 

  

  # Some constants / standard paths
 

  LOGDIR=/var/log/hosts
 
@@ -82,9 +83,9 @@ 
 

  

  for FILE in ${FILES}; do
 

      TEMP=$(echo ${FILE} | sed 's/\.xz$//')
 

-     simple_message_to_bus combinehttplogs.logmerge.proxy.start run_id="$RUN_ID" log="$PROXYLOG" file="$FILE" target="$TARGET" temp="$TEMP"
 

+     simple_message_to_bus combinehttplogs.logmerge.proxy.start log="$PROXYLOG" file="$FILE" target="$TARGET" temp="$TEMP"
 

      perl ${LOGMERGE} ${PROXYLOG}/${FILE} > ${TARGET}/${TEMP}
 

-     simple_message_to_bus combinehttplogs.logmerge.proxy.finish run_id="$RUN_ID" log="$PROXYLOG" file="$FILE" target="$TARGET" temp="$TEMP" result="$?"
 

+     simple_message_to_bus combinehttplogs.logmerge.proxy.finish log="$PROXYLOG" file="$FILE" target="$TARGET" temp="$TEMP" result="$?"
 

  done
 

  

  ##
 
@@ -93,9 +94,9 @@ 
 

  

  for FILE in ${FILES}; do
 

      TEMP=$(echo ${FILE} | sed 's/\.xz$//')
 

-     simple_message_to_bus combinehttplogs.logmerge.download.start run_id="$RUN_ID" proxylog="$DL_LOG" file="$FILE" target="$TARGET" temp="$TEMP"
 

+     simple_message_to_bus combinehttplogs.logmerge.download.start proxylog="$DL_LOG" file="$FILE" target="$TARGET" temp="$TEMP"
 

      perl ${LOGMERGE} ${DL_LOG}/${FILE} > ${TARGET}/${TEMP}
 

-     simple_message_to_bus combinehttplogs.logmerge.download.finish run_id="$RUN_ID" proxylog="$DL_LOG" file="$FILE" target="$TARGET" temp="$TEMP"  result="$?"
 

+     simple_message_to_bus combinehttplogs.logmerge.download.finish proxylog="$DL_LOG" file="$FILE" target="$TARGET" temp="$TEMP"  result="$?"
 

  done
 

  

  ##
 
@@ -106,9 +107,9 @@ 
 

  

  for FILE in ${FILES}; do
 

      TEMP=$(echo ${FILE} | sed 's/\.xz$//')
 

-     simple_message_to_bus combinehttplogs.logmerge.people.start run_id="$RUN_ID" proxylog="$PEOPLE" file="$FILE" target="$TARGET" temp="$TEMP"
 

+     simple_message_to_bus combinehttplogs.logmerge.people.start proxylog="$PEOPLE" file="$FILE" target="$TARGET" temp="$TEMP"
 

      perl ${LOGMERGE} ${PEOPLE}/${FILE} > ${TARGET}/${TEMP}
 

-     simple_message_to_bus combinehttplogs.logmerge.people.finish run_id="$RUN_ID" proxylog="$PEOPLE" file="$FILE" target="$TARGET" temp="$TEMP" result="$?"
 

+     simple_message_to_bus combinehttplogs.logmerge.people.finish proxylog="$PEOPLE" file="$FILE" target="$TARGET" temp="$TEMP" result="$?"
 

  done
 

  

  # Now we link up the files into latest directory
 
@@ -125,4 +126,4 @@ 
 

      done
 

      popd &> /dev/null
 

  fi
 

- simple_message_to_bus combinehttplogs.finish run_id="$RUN_ID"
 

+ simple_message_to_bus combinehttplogs.finish

file modified
+7 -5 
@@ -25,8 +25,10 @@ 
 

  # we are 5 days behind.
 

  

  # We have dropped this down to 3 days on 2019-10-01
 

- RUN_ID="$(uuidgen -r)"
 

- simple_message_to_bus condense-mirrorlogs.start run_id="$RUN_ID"
 

+ 

+ export MSGTOPIC_PREFIX=logging.stats
 

+ export MSGBODY_PRESET="loghost=$(hostname) run_id=$(uuidgen -r)"
 

+ simple_message_to_bus condense-mirrorlogs.start
 

  

  let NUMDAYS=3
 

  let OLDDAYS=$(( $NUMDAYS+1 ))
 
@@ -73,9 +75,9 @@ 
 

  elif [[ -f ${WORKFILE} ]]; then 

      echo "The workfile for ${YEAR}/${MONTH}/${DAY} already existed."
 

  else
 

-     simple_message_to_bus condense-mirrorlogs.mirrorlist.start run_id="$RUN_ID" workfile="$WORKFILE" logfile="$LOGFILE"
 

+     simple_message_to_bus condense-mirrorlogs.mirrorlist.start workfile="$WORKFILE" logfile="$LOGFILE"
 

      ${LBIN}/mirrorlist.py -o ${WORKFILE} ${LOGFILE}; 

-     simple_message_to_bus condense-mirrorlogs.mirrorlist.finish run_id="$RUN_ID" workfile="$WORKFILE" logfile="$LOGFILE" result="$?"
 

+     simple_message_to_bus condense-mirrorlogs.mirrorlist.finish workfile="$WORKFILE" logfile="$LOGFILE" result="$?"
 

  fi
 

  

  # So the data isn't strictly across month boundries due to the end of
 
@@ -110,4 +112,4 @@ 
 

  

  # cleanup the temp data
 

  rm -rf ${TEMPDIR}
 

- simple_message_to_bus condense-mirrorlogs.finish run_id="$RUN_ID"
 

+ simple_message_to_bus condense-mirrorlogs.finish

file modified
+6 -5 
@@ -1,7 +1,8 @@ 
 

  #!/bin/bash
 

  

- RUN_ID="$(uuidgen -r)"
 

- simple_message_to_bus countme-update.start run_id="$RUN_ID"
 

+ export MSGTOPIC_PREFIX=logging.stats
 

+ export MSGBODY_PRESET="loghost=$(hostname) run_id=$(uuidgen -r)"
 

+ simple_message_to_bus countme-update.start
 

  

  # Where do we keep our local/internal data?
 

  LOCAL_DATA_DIR=/var/lib/countme
 
@@ -37,10 +38,10 @@ 
 

      if [ "$DRYRUN" ]; then 

        return 0
 

      else
 

-       simple_message_to_bus countme-update.command.start run_id="$RUN_ID" command="$@"
 

+       simple_message_to_bus countme-update.command.start command="$@"
 

        "$@"
 

        RESULT=$?
 

-       simple_message_to_bus countme-update.command.finish run_id="$RUN_ID" command="$@" result="$?"
 

+       simple_message_to_bus countme-update.command.finish command="$@" result="$?"
 

        return $RESULT
 

      fi
 

  }
 
@@ -120,5 +121,5 @@ 
 

  _run atomic_copy $TOTALS_DB $PUBLIC_TOTALS_DB
 

  _run atomic_copy $TOTALS_CSV $PUBLIC_TOTALS_CSV
 

  

- simple_message_to_bus countme-update.finish run_id="$RUN_ID"
 

+ simple_message_to_bus countme-update.finish

The simple_message_to_bus script as deployed on log01 has diverged from how it is defined in Ansible, probably due to hotfixing things. This PR aims to clean up the situation by doing this:

  • Ensure that PATH is set in /etc/crontab and contains /usr/local/bin
  • Remove default for message topic prefixes from the script, set this in calling scripts explicitly as MSGTOPIC_PREFIX
  • Let scripts using simple_message_to_bus set MSGBODY_PRESET to define items which should be present in all sent messages

This PR is based on top of #766 which should be merged first.

Metadata Update from @nphilipp:
- Pull-request tagged with: freeze-break-request
3 years ago

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

+1

+1 lgtm, zuul fail looks to be due to file rename

rebased onto b43e5920442ea1cae190dd7def43096d24920ed6
3 years ago

rebased onto b43e5920442ea1cae190dd7def43096d24920ed6
3 years ago

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

rebased onto 8e05661
3 years ago

Pull-Request has been merged by nphilipp
3 years ago

Build succeeded.
Metadata
None

freeze-break-request

Changes Summary 6
+10 -4
file changed
roles/fedora-messaging-utils/files/simple_message_to_bus
+41
file added
roles/fedora-messaging-utils/tasks/crontab_path.yml
+6 -0
file changed
roles/fedora-messaging-utils/tasks/main.yml
+10 -9
file changed
roles/web-data-analysis/files/combineHttpLogs.sh
+7 -5
file changed
roles/web-data-analysis/files/condense-mirrorlogs.sh
+6 -5
file changed
roles/web-data-analysis/files/countme-update.sh
Powered by Pagure 5.14.1
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.