Where was the Big Bang....?

It is well known that the big bang created the universe from a single point and ever since the universe has been expanding in all directions. Now presumably the expansion is in straight lines away from the initial bang (ok with some variances for gravitational interaction) so therefore why cant Astronomers actually say at waht point in the sky the big bang took place. We have a complete 360 degree spherical view of the entire universe, so we must be able to look in the direction of the big bang...so why dont we know it?