Pay-per-output? AI firms blindsided by beefed up robots.txt instructions