You can go back but it's very difficult. Only the biggest nerds can do it, with great dedication and time. That process is called reverse engineering.
For a very simple example, suppose I wrote some code to add how many apples Jack and Jill have together. The source code might look like
jackApples = 3
jillApples = 4
numApples = jackApples + jillApples
But the computer doesn't care about Jack, or Jill, or apples for that matter. It only cares about numbers. So when the compiler puts it into ones and zeros all those useful names get dropped. And when I decompile the binary (what we call those ones and zeros) what I get back might look more like
var1 = 3
var2 = 4
var3 = var1 + var2
And if I want to change how many apples Jill has it's a whole process of trial and error to figure out which variable is Jill's number of apples.
Now expand that to thousands or millions of lines of code and you begin to see why nerds want source code instead of binaries.