Ron,
What kind of "formulas" do you want?
Navigation on the surface of the Earth is no more complicated than applications of spherical trigonometry. A book with MORE than you ever will want to know about this stuff is "Textbook on Spherical Astronomy" by Smart.
Basically, if you imagine a triangle drawn on the surface of the Earth, then you can describe the angles between adjacent sides (i.e. where the sides meet) as A, B, and C (other than being on the surface of the sphere, these angles are the same as the interior angles of a plane triangle - i.e. the angles you would measure with a protractor). Opposite each of these interior angles is a portion of a great circle, of arc length respectively a, b, and c (where a, b, and c are opposite the relevant angles, and the angular value is the angle that the portion of the great circle subtends as viewed from the center (inside) of the sphere.
Then, the relations between angles and sides are
sin a sin b sin c
------- = ------- = ------- (the "sine" formulae),
sin A sin B sin C
and cos a = (cos b)*(cos c) + (sin b)*(sin c)*(cos A)
(the "cosine" formulae, with appropriate permutations).
Some practice in envisioning and drawing spherical triangles relevant to the problem at hand is necessary. For starters, imagine a spherical triangle defined by the North Pole, a point on the equator at longitude zero, and a point on the equator at longitude 90 degrees. Then, the interior angles will all be 90 degrees and the lengths of the sides will also all be 90 degrees.
For navigation in general, you probably know your current location (in latitude and longitude) and your destination (also in latitude and longitude) and want to know what direction you need to go in order to reach your destination in the quickest possible manner (ignoring windage, current, etc. - i.e. what is the heading of your destination) and how far away is your destination.
The side of the triangle between you and your destination is then the interesting one. Its length (in angular measure, converted to linear measure via the radian formula) and direction (i.e. angle between it and due north) are what you need to calculate.
To do this, draw a spherical triangle with corners at the north pole, your current location, and your destination. Then, the interior angle at the top (A) will be the difference on longitudes of your current location and destination, and two of the sides (b and c) will have angular lengths that are 90 degrees minus the latitudes.
From the cosine formula above, you can then find the length (a) of the third side: you know A and b and c. Convert a from angular measure to linear distance (the distance to your destination) using the radius of the Earth as (approximately!) 6370 km.
Once you know a, you can find your desired heading by finding the angle b (assuming you are now at the point which is the junction of sides a and c.
Be careful when taking inverse trig functions!
These instructions are specifically a bit incomplete in that I hope you will draw some triangles and derive the relevant results!
Note that you can also accomplish much of this with vector products. For instance, the dot product (done with unit vectors) will give you the angle between two points on the Earth: convert your lat, lon (of both current position and destination) to x,y,z coordinates
x = cos(lat) * cos(lon)
y = cos(lat) * sin(lon)
z = sin(lat)
and take the dot product to get the cosine of the angular separation. use the radius of the Earth, as above, to then get the distance to your destination.
All of this assumes a truly spherical Earth, which ain't quite the case, but close enough unless you are trying to land ballistic missiles on the bad guys a continent away (in which case you may miss by 20-40 kilometers)!!