One of the comments:

It’s always been my intuition that null-hypothesis significance testing is even more inappropriate in the tech industry than academia. The traditional framework considers an asymmetric standard of evidence between the null and alternative hypothesis, but oftentimes in industry settings there’s no natural choice for a “null hypothesis” at all. There’s no reason NOT to chase noise when all alternatives are costless or equivalently costly. But it’s surprisingly hard to convince people of this even at big-name “independent thinker” firms. They end up tying themselves in knots trying to justify some arbitrary choice as a “null hypothesis” just to reshape the problem into something traditional methodology can handle–presumably because so long as they’re going with the state-of-the-art in science at the universities, nobody can blame them for how they’ve done things.