It so happened that our web server had not restarted for very very long. It kept a copy of the SSL certificate in memory when it started up a long time ago, unaware that letsencrypt had pulled the carpet underneath it and renewed the certificate.

The idea is to reload nginx after every successful renewal of certificate. It seems that a newer version of certbot is capable of handling hooks, so we upgraded the version of letsencrypt that was bundled with the distribution and added a renew-hook:

certbot renew -q --deploy-hook "/etc/init.d/nginx reload"

Tuj's dream of no manual intervention is shattered again. There are too many moving parts. We should have stuck with plain html + insecure http. This way it would be so much easier to keep everything working even after we are gone.