Smooge tells me that the current system's "rawdb" output is threatening to overwhelm the system. Will originally estimated that growth of rawdb would be something like 10GB/year, but I think we underestimated Fedora growth — and particularly, EPEL usage.

The current system processes raw Apache log files into rawdb and then generates totals.db from that file. It would be possible to instead accumulate directly into the totals.db.

 for every line in the http log:
     if the line matches the data format we're expecting
          collapse date to just "weeknum"
          extract (os_name, os_variant, os_version, os_arch, sys_age, repo_tag, repo_arch)
          if line matching weeknum + extracted info isn't in the db
                 add it with hits = 1
          else if it is in the db
                  increment hits

The last part could be done with a sqlite "upsert".

The main complications, as I understand it are:

1. The intermediate "rawdb" files are faster to parse then going through the whole text logs again, if we need to do that.
2. The Log files rotate at midnight, but we might have stragglers off by as much as 241 seconds (according to comments in the code) which are in the next day's log.
   • This is important because technically the "countme" week could land twice from one system in that same four-minute window if it's the week boundary. So we can't just do "one file, one whole day". Rather, we need to keep each day "open" until the next day comes
   • ...and therefore each week

For the first, it might just be a matter of "well, that's too bad", if we don't have the space.

For the second, though, we could do this a different way.

1. Instead of logging raw lines, do the above pseudocode above to "intermediate.db" as the log files come in.
2. When we get the http log that contains the last possible minutes of the week (that is, the first day of the next week -- Monday's log file, with its possible few entries of the Sunday before):
   • process that
   • then, we can be sure that all the previous week entries are in (because there definitely won't be any from Sunday in Tuesday's log)
   • copy all of the entries matching previous weeknum to totals.db
   • verify totals.db
   • publish totals db
   • delete all rows from intermediate.db before the current week
   • so it'll now just contain the entries from this latest Monday
   • and vacuum the intermediate.db sqlite file (no more space problem!)

Am I missing anything here? It's very possible I am!