But at that point why have hydrogen-based or ammonia-based energy infrastructure? Why not use the excess electricity to make fertilizer and use that to grow crops?

Are the factories that make hydrogen so much cheaper than the factories that make fertilizer that when you build enough factories to make use of all the excess electricity (meaning they're idle 80% of the time), the cost of having those factories make fertilizer instead of hydrogen is greater than the cost of building and maintaining the entire infrastructure for using hydrogen or ammonia as energy storage/fuel?