Skip to main content

Hartwork Blog

Free Software, Music, Chinese Chess

DomainFactory WTF 2025

Or how DomainFactory — part of Host Europe and a GoDaddy company — takes turning against their customers to a whole new level with their 2025 forced e-mail migration to Microsoft Office 365, and breaks things left and right in the process…

Some background before we start

DomainFactory had an excellent reputation 20 years ago. I was one of those new customers that they won through an exciting offer for their 5th anniversary in 2005. Everything times five: 5 domains, 5 GB webspace, 5 databases and so on — good times, and good usability. I was a happy customer.

Then in 2015(?) the company announced to move their data centers out of Germany to Strasbourg in France. I can only guess that the amount of protest about that out-of-Germany move got DomainFactory to eventually only move some of their servers to Strasbourg and to offer hosting in Germany as an alternative, for a premium. So customers had been hosted in Germany for years and suddenly were charged a premium for what they already had for free before — cool! I'm pretty sure that's illegal but you tell me. I have not forgotten or forgiven that move and it was the point I stopped recommending DomainFactory to others. A similar thing happened again later when they stopped offering that 5x-everything deal to existing customers and force-migrated me to a different deal that was more expensive; nothing in that change was of my interest.

Now in 2025 things changed to a whole new level of WTF.

The forced migration of e-mail to Office 365 in March/April 2025 — what a disaster!

I will save you and me a timeline-like view of the story and instead just focus on the damage that DomainFactory has done:

  • Three days before the start of the migration, access to the mailbox section of the admin panel started to be denied altogether. So I had no way of seeing the list of mailboxes or deleting mailboxes so that their contained e-mails would not be copied to Microsoft. DomainFactory's note "[W]e will freeze your mailbox settings three days before the scheduled migration." (translated from German) in their announcement e-mail did not stand out, and was easy to miss unless you read every single word of that e-mail.

  • Two of three mailboxes affected by migration had >100 items skipped, and migration was stopped and not finished. DomainFactory did not inform or apologize or communicate plans about how to continue. How do I know? From my own search in the Exchange admin center they expose labeled "Advanced". That's the same mailboxes will start to be charged 0.99 EUR per month per mailbox (from April 18th for me) when they have been included for free for years. With corrupted e-mails (and a load of friction in migration) but now paid. And they did say that "No further action is required on your part before the migration." (translated from German) Except for doing backups or jumping ship maybe.

  • My forwarding alias used for my account with PayPal was deleted, putting my use of PayPal at risk; the user interface no longer allows the creation of cross-domain mail aliases which was the case for my forwarder with PayPal.

  • Customer support has closed 3 of 3 support requests created via One.Done! with a template reply and without resolving my problems. They were:

    • The fact that the PayPal forwarder alias was lost in the migration. That issue is unresolved. (DF-748556)
    • The fact that they blocked two domain transfers despite use of the correct Auth-Code. (DF-760730)
    • (The fact that all folders seemed gone after re-login to IMAP. It turned out to need a re-subscription to all folders on my end. Would have been easy to send a helpful response to. (DF-748751))

  • Waiting time for customer support by phone was 40 minutes (on April 3rd, afternoon). My total waiting time to speak with DomainFactory phone support (in the last two weeks) was roughly 40 + 25 + 25 + 7 = 97 minutes.

  • Sending e-mails with Thunderbird on Linux turned out impossible to get to work again after migration. I don't think it's by accident that their official help on Thunderbird says "Mac and Windows" but does not mention Linux. Yes, I did try to enable "app" SMTP auth but without any observable change in log-in failure.

  • Because address sebastian@pipping.org was realized as an alias, the new webmail at outlook.office365.com would not allow me to send e-mail as my primary address sebastian@pipping.org. So sending as sebastian@pipping.org did not work in any way. For each address, the first attempt to log in via webmail would also fail with a weird error that went away after I re-tried with "flagging enabled".

  • At no point did I approve Microsoft's terms of service, yet they now have a copy of all my e-mails. No idea in which jurisdiction or data center. Also, if you had asked me: I wanted zero Microsoft in my e-mail infrastructure — zero. By now access to productivity.secureserver.net is blocked to me unless I accept Microsoft terms: the closing button disappeared for me on April 15th.

  • Deleting of one mailbox ...@hartwork.org (that happened to be the last postbox on that domain) killed all aliases for hartwork.org as a side effect (and e-mail to them was rejected). These aliases magically re-appeared when I dared experiment with the restore deleted users feature and could hardly believe my eyes when that in fact restored aliases unrelated to the mailbox but on the same domain.

  • (One night the whole admin panel was impossible to log into when I was about to adjust settings.)

  • All server-side filters are gone since the migration. Previously, certain mail headers would move e-mails to — say — mailing list bug-make into a dedicated folder, likely based on technology procmail. All gone, everything goes straight to flooding INBOX now.

  • At the time of this writing, I have already lost more than a full week of time and effort in cleaning up this mess. Thanks DomainFactory!

My migration to INWX for domains and DNS and to Manitu Webhosting for e-mail is ongoing still — at 95% maybe —, it takes time. Having recent backups from the use of offlineimap helped a ton. Many thanks to that project!

I did look at providers that target e-mail hosting specifically and in Germany before I understood that I should look for webhosting that includes e-mail instead, both in terms of pricing and in terms of flexibility with creation of new IMAP mailboxes over time. I ruled out…

  • goneo because they insist on also taking over the nameserver for custom domains.
  • mail.de because they do not seem to support serving custom domains as an MX server and need one account per IMAP mailbox.
  • mailbox.org because — longer story (..) — multiple IMAP accounts are not fun to manage and rather expensive with them.
  • Posteo because they decided to not support custom domains (and seem to need one account per IMAP mailbox).
  • Tuta because they deliberately do not support IMAP.

Let me add a few pictures from the crime scene:

Pictures from the crime scene

Exchange admin center: Migration details

Exchange admin center: Migration details

Webmail log-in error

Webmail log-in error

Microsoft terms that I (still) have not accepted

Note that the closing button in the top right disappeared for me on April 15th.

Microsoft terms not accepted

DomainFactory customer satisfaction guarantee

Customer satisfaction guarantee

How to configure s-nail/s-mailx/mailx for use with Manitu

In context of Sending e-mail on successful SSH login I needed to configure s-nail 14(.9.25) (also known as s-mailx or command mailx) to send e-mail using hoster Manitu.

The working result for file ~/.mailrc was this:

set v15-compat
set mta=smtps://EMAIL:PASSWORD@mail.manitu.de:465 \
  smtp-auth=login

Without the set v15-compat I would get this error:

s-nail: New-style URL used without *v15-compat* being set                                                                                                               
/root/dead.letter 4/88                                                                                                                                                  
s-nail: ... message not sent

It is worth nothing that both parameters EMAIL and PASSWORD will need to have special characters percent-encoded — e.g. @ becomes %40 — to not violate URI syntax; see section "3.2.1. User Information" of RFC 3986 for details if curious.

Here is one way how to percent-encode a string on the command line:

# echo -n 'user@host.invalid' | python3 -c 'from urllib.parse import quote; import sys; print(quote(sys.stdin.read(), safe=""))'
user%40host.invalid
    ^^^

Please note that passwords should be passed through standard input rather than the command line, to not end up in shell history and to not be visible to other users of the system.

Did my post help? Let me know!

How much security is in long-term support?

Everyone loves the idea of long-term support — LTS for short — where a vendor promises you to provide security updates for a few years: little friction and strong security. Does long-term support live up to that dream? It does not. What makes me say that?

The dream island named "long-term support" knows five types of monsters:

Meet monster type 1: "Out of scope"

Long-term support comes with a scope. For example, in Ubuntu LTS repositories main and restricted are in scope and universe is out of scope.

When I asked the Ubuntu Security team in March 2020 about the four unfixed CVEs in liburiparser1 0.8.4-1 of Ubuntu 18.04.4 LTS that all had fixes available upstream they were not responding "We just fixed it" or even "We will fix it"; instead they were explaining to me that because uriparser was in universe it is "community maintained", i.e. not part the LTS deal. To Canonical that may make sense, but it was an eye-opener for me at the time.

It is worth a side note here that Debian does not make two-classes distinctions like that in Debian stable, which is one reason I started recommending Debian over Ubuntu.

To summarize: Many vulnerable packages are considered out of the scope of long-term support.

Meet monster type 2: "Never labeled as security upstream"

Not every project labels all security fixes as security fixes and goes forward registering CVEs for them. That's a mistake because without a security label, bugfixes do not get backported. So they remain unfixed — oops! Why are many security fixes not labeled as security fixes?

First, requesting CVEs is work, e.g. you need to carefully fill out a form for each CVE request answering questions like:

  • What is the impact?
  • Which versions of the software are affected?
  • How would the vulnerability be attacked?

Second, if you are the one that caused the vulnerability, it takes a mindset that does not consider admitting to a vulnerability as shameful. Not everyone is in a place where admitting to mistakes feels safe.

Third, not labeling something as a vulnerability can reduce the number of eyes on it and so one could hope, that fewer attackers find out about the vulnerability. But that's security by obscurity, which does not help against attackers that are serious about a target, and takes away options from everyone else.

To summarize: What was never labeled as a security fix in the first place will not be backported by long-term support; it will be known to attackers but not be fixed.

Meet monster type 3: "Backporting too complex"

Many security fixes are small and feasible to backport but some are not. libexpat has had a few of these; one example is my fix to billion laughs attacks for libexpat 2.4.0 (CVE-2013-0340) where the fix was not backported to RHEL 8 despite being affected, likely because of complexity. In Red Hat's own words from a FAQ answer about will-not-fix decisions:

A "will not fix" status means that a fix for an affected product version is not planned or not possible due to complexity, which may create additional risk.

To summarize: If a security fix is considered too complex to backport, long-term support will be left vulnerable.

Meet monster type 4: "Backport arrived too late"

Unless a coordinated disclosure effort (sometimes called "responsible disclosure") is delaying the publication of a vulnerability and the related upstream fixes until all backporting work has been completed, backported patches arrive later than patches to the latest release of the software. That time window can be used by attackers to attack long-term support systems in particular.

It should be noted that coordinated disclosure is work, in particular when many parties are involved. It seems fair to say that coordinated disclosure is often limited to the most severe of vulnerabilities or when someone offers to do the coordination work on behalf of the security researcher. CERT/CC VINCE may be able to help security researchers with tiring coordination and communication.

To summarize: The delay of backporting creates additional opportunities for attackers (that do not exist with rolling release approaches).

Meet monster type 5: "Criticality threshold not met"

In some contexts, there are thresholds for a minimum CVSS score that need to be met — e.g. CVSS >=7.0 — for an obligation to backport (or update). So any vulnerability classified as CVSS 5.5, in that context, would come with a free pass to ignore. A very bad idea in my book, in particular, given how often CVSS scores are off — either too high or too low — in evaluation.

I'm not sure what Red Hat's precise cutoff point is for MinGW packages, but they do have a (confirmed) policy to only fix vulnerabilities considered critical in MinGW packages, and it shows; mingw-expat in RHEL 8 is missing multiple security fixes.

It is worth noting, that when security researchers are not sure whether remote code execution is possible — a common case —, they should mention that they cannot rule out code execution.

To summarize: A vulnerability may be considered not critical enough to fix by vendors, and evaluation of criticality is complex and unreliable.

Is any of that surprising?

If your answer is "yes" I would be curious to hear about the consequences you are drawing.

Sebastian Pipping

Recursion kills: The story behind CVE-2024-8176 / Expat 2.7.0 released, includes security fixes

For readers new to Expat:

libexpat is a fast streaming XML parser. Alongside libxml2, Expat is one of the most widely used software libre XML parsers written in C, specifically C99. It is cross-platform and licensed under the MIT license.

Expat 2.7.0 has been released earlier today. I will make this a more detailed post than usual because in many ways there is more to tell about this release than the average libexpat release: there is a story this time.

What is in release 2.7.0?

The key motivation for cutting a release now is to get the fix to a long-standing vulnerability out to users: I will get to that vulnerability — CVE-2024-8176 — in detail in a moment. First, what else is in this release?

There are also fixes to the two official build systems as usual, as well as improvements to the documentation.

There is a new fuzzer xml_lpm_fuzzer by Mark Brand that OSS-Fuzz has already started to include with their daily continuous fuzzing; the fuzzer is based on Clang's libFuzzer and Google's libprotobuf-mutator (LPM) that applies a variant of coverage-guided fuzzing called structured fuzzing. A side job of integrating that new fuzzer was making dependency libprotobuf-mutator support the versions of Protobuf that are shipped by Ubuntu 24.04, 22.04 and 20.04: my related work upstream is available to everyone.

Another interesting sideshow of this release is the (harmless) TOCTTOU issue that was uncovered by static analysis in a benchmarking helper tool shipped next to core libexpat. If you have not heard of that class of race condition vulnerability but are curious, the related pull request could be of interest: it is textbook TOCTTOU in a real-world example.

One other thing that is new in this release is that Windows binaries are now built by GitHub Actions rather than AppVeyor and not just 32bit but also 64bit. I have added 64bit binaries post-release to the previous release Expat 2.6.4 already on January 21st, but only now it is becoming a regular part of the release process.

The vulnerability report

So what is that long-standing vulnerability about? In July 2022 — roughly two and a half years ago — Jann Horn of Google Project Zero and Spectre/Meltdown fame reached out to me via e-mail with a finding in libexpat, including an idea for a fix.

What he found can be thought of as "the linear version of billion laughs" — a linear chain (of so-called general entities) rather than a tree — like this:

<!DOCTYPE doc [
  <!ENTITY g0 ''>
  <!ENTITY g1 '&g0;'>
  <!ENTITY g2 '&g1;'>
]>
<doc>&g2;</doc>

Except not with two (or three) levels, but thousands. Why would a chain of thousands of entity references be a problem to libexpat? Because of recursion, because of recursive C function calls: each call to a function increases the stack, and if functions are calling each other recursively, and attacker-controlled input can influence the number of recursive calls, then with the right input, attackers can force the stack to overflow into the heap: stack overflow, segmentation fault, denial of service. It depends on the stack size of the target machine how many levels of nesting it takes for this to hit: 23,000 levels of nesting would be enough to hit on one machine, but not another.

The education that introduces or leads people towards recursion should come with a warning; recursion is not just beautiful, a thinking tool and allowing for often simpler solutions — it also has a dark side to it: a big inherent security problem. The article The Power of 10: Rules for Developing Safety-Critical Code warned about the use of recursion in 2006, but Expat development already started in 1997.

Already in that initial e-mail, Jann shared what he considered the fix — avoiding (or resolving) recursion — and there was a proof-of-concept patch attached that showcased how that could be done in general. Unlike other Project Zero findings, there would be no 90-days-deadline for this issue, because — while stack clashing was considered and is a theoretical possibility — denial of service was considered to be the realistic impact. It should be noted that this risk assessment comes without any guarantees.

The vulnerability process

Two things became apparent to me:

  1. It seemed likely that this vulnerability had multiple "faces" or variants, and that the only true fix would indeed be to effectively remove all remaining recursion from Expat. It is not the first time that recursion has been an issue in C software, or even libexpat in particular: Samanta Navarro resolved vulnerable recursion in a different place in libexpat code in February 2022 already. Thanks again!
  2. That it would be a pile of work, not a good match to my unpaid voluntary role in Expat as an addition to my unrelated-to-Expat day job, and not without risk without a partner at detail level on the topic. My prior work on fixing billion laughs for Expat 2.4.0 made me expect this to be similar, but bigger.

And with that expectation, the issue started aging without a fix, and in some sense, I felt paralyzed about the topic and kept procrastinating about it for a long time. Every now and then the topic came up with my friend, journalist and security researcher Hanno Böck whom I had shared the issue with. He was arguing that even without a fix, the issue should be made public at some point.

One reason why I was objecting to publication without a fix was that it was clear that in lack of a cheap clean fix, vendors and distributions would start applying quick hacks that would produce false positives (i.e. rejecting well-formed benign XML misclassified as an attack), leave half of the issue unfixed, and leave the ecosystem with a potentially heterogeneous state of downstream patches where — say — in openSUSE a file would be rejected but in Debian it would parse fine — or the other way around: a great mess.

I eventually concluded that the vulnerability could not keep sitting in my inbox unfixed for another year, that it needed a fix before publication to not cause a mess, and that I had to take action.

Reaching out to companies for help

In early 2024, I started considering ways of finding help more, and added a call for help banner to the change log that was included with Expat 2.6.2. I started drafting an e-mail that I would send out to companies known to use libexpat in hardware. I had started maintaining a (by no means complete) public list of companies using Expat in hardware that now came in handy.

On April 14th, 2024 I started finding looking for security contacts for companies on that list. For some, it was easy to find and for others, I gave up eventually; for some, I am still not sure whether I got the right address or whether they are ghosting me as part of an ostrich policy. I wish more companies would start serving /.well-known/security.txt; finding vulnerability report contacts is still actual work in 2025 and should not be.

So then I mailed to circa 40 companies using a template, like this:

Hello ${company},


this e-mail is about ${company} product IT security.
Are you the right contact for that?  If not please forward
it to the responsible contact within ${company} — thank you!

On the security matter:

It has come to my attention that ${company} products and
business rely on libexpat or the "Expat" XML parser library,
e.g. product ${product} is using libexpat according to
document [1].  I am contacting you as the maintainer of
libexpat and its most active contributor for the last 8
years, as can be seen at [2]; I am reaching out to you today
to raise awareness that:

- All but the latest release of libexpat (2.6.2) have
  security issues known to the public, so every product
  using older versions of libexpat can be attacked through
  vulnerable versions of libexpat.

- Both automated fuzzing [3] and reports from security
  researchers keep uncovering vulnerabilities in libexpat,
  so it needs a process of updating the copy of libexpat
  that you bundle and ship with your products, if not
  already present.

- My time on libexpat is unfunded and limited, and there is
  no one but me to constantly work on libexpat security and
  to also progress on bigger lower priority tasks in
  libexpat.

- There is a non-public complex-to-fix security issue in
  libexpat that I have not been able to fix alone in my
  spare time for months now, that some attackers may have
  managed to find themselves and be actively exploiting
  today.

I need partners in fixing that vulnerability.
Can ${company} be a partner in fixing that vulnerability,
so that your products using libexpat will be secure to use
in the future?

I am looking forward to your reply, best



Sebastian Pipping

Maintainer of libexpat


[1] ${product_open_source_copyright_pdf_url}
[2] https://github.com/libexpat/libexpat/graphs/contributors
[3] https://en.wikipedia.org/wiki/Fuzzing

Replies are coming in

The responses I got from companies were all over the map:

  • My "favorite" reply was "We cannot understand what you want from us" when everyone else had understood me just fine. Nice!

  • Though that competes with the reply "A patch will be released after the fall." when they had not received any details from me. Okay!

  • There was arguing that the example product that I had mentioned was no longer receiving updates (rather than addressing their affected other products that are not end-of-life and continue to use libexpat).

  • I was asked to prove a concrete attack on the company's products (which would not scale, need access to the actual product, etc).

  • That they "do not have sufficient resources to assist you on this matter even if libexpat is employed in some of .....'s products" came back a few times.

It was interesting and fun in some sense, and not fun in another.

Next stop: confidentiality

What came next was that I asked companies to sign a simple freeform NDA with me. Companies were not prepared for that. Why was I asking for an NDA and TLP:RED? To (1) make sure that who got the details would need to collaborate on a true fix and not just monkey-patch their own setups and (2) to avoid the scenario of heterogeneous trouble fixes that I mentioned before that would have been likely in case of a leak before there was a true fix.

Some discussions failed at NDA stage already, while others survived and continued to video calls with me explaining Jann's findings in detail.

It is worth noting that I knew going in that many vulnerability reward programs exclude the whole class of denial of service and so I tied sharing the expected impact to signing an NDA to reduce the chances of everyone discarding it "Oh 'just' denial of service, we'll pass".

The eventual team and security work

Simplifying a bit, I found two main partner companies in this: Siemens and a company that would not like to be named, let's call them "Unnamed Company". Siemens started development towards a candidate fix, and Unnamed Company started evaluating options of what other companies they could pay to help for them, which got Linutronix and also Red Hat involved.

Siemens took the builder role while Linutronix, Red Hat and I provided quality assurance of various kinds. While we did not work day and night, it is fair to say that we have been working on the issue since May 2024 — for about 10 months.

The three faces of the vulnerability

It did indeed turn out that the vulnerability has multiple — three — faces:

1. General entities in character data

<!DOCTYPE doc [
  <!ENTITY g0 ''>
  <!ENTITY g1 '&g0;'>
  <!ENTITY g2 '&g1;'>
]>
<doc>&g2;</doc>

2. General entities in attribute values

<!DOCTYPE doc [
  <!ENTITY g0 ''>
  <!ENTITY g1 '&g0;'>
  <!ENTITY g2 '&g1;'>
]>
<doc key='&g2;'/>

3. Parameter entities

<!DOCTYPE doc [
  <!ENTITY % p0 ''>
  <!ENTITY % p1 '&#37;p0;'>
  <!ENTITY % p2 '&#37;p1;'>
  <!ENTITY % define_g0 "<!ENTITY g0 '&#37;p2;'>">
  %define_g0;
]>
<doc/>

The third variant "Parameter entities" reuses ideas from my 2021 exploit for vulnerability Parameter Laughs (CVE-2021-3541): It used the same mechanism of delayed interpretation.

There are three related parameterized attack payload generators in Python available; use should be straighforward:

# python3 payload3.py --help | head -n1
usage: payload3.py [-h] [count]

# python3 payload3.py 4
<!DOCTYPE doc [
  <!ENTITY % p0 ''>
  <!ENTITY % p1 '&#37;p0;'>
  <!ENTITY % p2 '&#37;p1;'>
  <!ENTITY % p3 '&#37;p2;'>
  <!ENTITY % p4 '&#37;p3;'>
  <!ENTITY % define_g0 "<!ENTITY g0 '&#37;p4;'>">
  %define_g0;
]>
<doc/>

# python3 payload3.py | xmlwf -p -r /dev/stdin
Segmentation fault

Please use attack payload responsibly!

Conclusions and gratitude

It is no overstatement to say that without Berkay Eren Ürün — the main author of the fix — and his manager Dr. Thomas Pröll at Siemens there would be no fix today: a big and personal "thank you!" from me.

Thanks to Unnamed Company, to Linutronix, to Red Hat for your help making this plane fly!

Thanks to Jann Horn for his whitehat research and the demo patch that lead the path to a fix!

Thanks to everyone who contributed to this release of Expat!

And please tell your friends:

Please leave recursion to math and keep it out of (in particular C) software: it kills and will kill again.
Kind regards from libexpat, see CVE-2022-25313 and CVE-2024-8176 for proof.

For more details about this release, please check out the change log.

If you maintain Expat packaging or a bundled copy of Expat or a pinned version of Expat somewhere, please update to 2.7.0. Thank you!

Sebastian Pipping

Most IT companies fail to serve security.txt for RFC 9116 in 2025

I happen to maintain a public list of companies using libexpat in hardware, though not complete by any means. Last time I tried mass-mailing companies about a security issue in April 2024. Finding the right contact for security was non-trivial and even failed in some cases. E.g. for Humax Digital I eventually gave up.

It is needless to say that if your security contacts are too hard to find, that says something about how urgently you want to fix security issues (or not).

So I felt like re-checking how many of these 50 companies are serving /.well-known/security.txt (or the significantly less common /security.txt) a la RFC 9116 in 2025.

The sad answer is: 39 out of the 50 companies I tested do not, i.e. 78%. Here's who and where exactly I tested:

If you work at a company that does not serve /.well-known/security.txt yet, please fix it or share a link to https://securitytxt.org/ with a co-worker or management of yours so they can — thank you!