Gravitational potential energy can, when g is constant, be defined as:

U = mgy

When g is constant you are free to set the zero for potential energy. Define the zero at some convenient level, and measure all vertical distances and potential energies from there.

The critical number is not the value of the potential energy, it's the change in potential energy. ΔU is the same no matter where you define the zero.