The story behind it

I was pointed to a vulnerable Django setup not too long ago, and had a closer look. Their setup would have allowed potential arbitrary remote code execution through this combination of configurations, with emphasis on combination:

1. They were using a Redis database that listens to the public internet for both a Django Celery Queue and a Django cache. With the right credentials, an attacker could talk to the database directly.
2. They had Redis credentials — user and password — in the CELERY_BROKER_URL Django setting (which is not uncommon with Celery).
3. Django's default SafeExceptionReporterFilter does not cleanse setting key CELERY_BROKER_URL, and so Django's debug mode reveals CELERY_BROKER_URL content including credentials on server errors.
4. They had Debug mode enabled by mistake and without knowing, more on how that was possible below.
5. They had a Django view that could be forced to crash to then reveal the debug error page.
6. Django would unpickle anything that an attacker puts into the Redis cache after takeover and thus run attacker code through pickle as known for years.

In isolation, none of these cause as big of a problem, but combined things get scary; looking at that same picture from the opposite side, fixing any single of these parts would have closed the attack vector as whole. The affected setup has part (4) fixed now, the related pull request is public. It's a good reminder that in Python both bool('False') and bool('0') evaluate to True even when it feels wrong in some way.

What about closing some of these open doors for everyone and by default, wouldn't that be nice?

Trying to close these open doors by default in Django

Regarding (2) and (3), the CELERY_BROKER_URL issue with SafeExceptionReporterFilter is something that I already reported over five years ago but it's easy to argue that never activating debug mode is the only true fix (and that's not wrong but also doesn't help the situation) and so it was closed as "wontfix" then, and again this year when I wanted to give it another shot with a pull request when it became clear again to be a killer in practice.

Regarding (4), the debug mode that was enabled by accident could have been prevented by Django limiting settings.DEBUG to instances of bool (or by disallowing string values like "off", "no", "0", "disabled", "false", "False" except the latter approach does not scale well beyond English if that matters). It can always be argued that we cannot protect all users from themselves and that this is beyond the line of user responsibility, but it would have saved that particular setup. The issue was closed as "wontfix".

Regarding (6), making use of pickle in caching secure by wrapping it with a layer of did-we-pickle-this-ourself-earlier protection will cost some performance (which is yet to be proven critical), but it would make a good secure default and something that only those users should turn off in order to re-gain the lost performance who understand their threat model well and whether it's really okay to own all of Django when the cache database gets owned. It was closed as "wontfix".

Because I had one more hardening issue to report that keeps coming up in the wild, I filed one more issue for the security-by-obscurity issue with /static/staticfiles.json where attackers can learn about your Django dependencies, their precise versions and get new ideas for targetted attacks from that, in particular with setups missing security updates (which is the true issue to be fixed in the setup, indeed). Almost everyone starts hiding that file once made aware of the implications but the issue was closed as "wontfix".

Four hardening issues closed as "wontfix" felt like an unfortunate pattern to me — I knew "wontfix" as an exception only, even outside of security — so I reached out to the Django security team via e-mail to be sure they were in support of these "wontfix"es and that this was not just one big misunderstanding. They are in support of it and me investing more time in discussions on the forums is their wished way forward, too. So not a misunderstanding.

I have decided to direct my time and energy elsewhere, to rather blog about it here in order to raise awareness about these issues before these doors are closed by default. I also have some hope (just a tiny bit) that maybe one of my readers — could be you — wants to be the force to advance these topics in the Django forums.

Bonus track

Regarding (5), the particular crash I found was interesting. EmailField comes with max_length=254 by default and so if you have API endpoints that ask for well-formed e-mail-addresses and store them into the database without length validation, passing a too-long-but-well-formed e-mail address may allow crashing view code (at database entrance) that looks perfectly healthy at first but doesn't do enough for validating objects.

Stay secure, and have a nice day!

Sebastian Pipping