Most wrong answers in aviation arithmetic do not come from hard formulas. They come from a correct formula fed the wrong kind of number: a speed in the wrong unit, a direction measured from the wrong north, a wind pointed the wrong way. The result looks plausible, which is what makes it dangerous to trust.
The cases below each show the size of the error with numbers run through the calculators on this site. All the inputs are round illustrative values, not data for any aircraft or airport.
Units that look alike
Knots and miles per hour. A speed of 120 mph is 104.3 kt. Over 300 NM, the flight time calculator gives 2 h 30 min at 120 kt but 2 h 53 min at 120 mph. Reading one unit as the other moves the answer by 23 minutes.
Nautical and statute miles. A distance of 300 statute miles is 260.7 NM, and 300 NM is 345.2 statute miles or 555.6 km. The three units are compared in nautical miles vs statute miles.
Feet and meters. An altitude of 3,000 m is 9,842.52 ft, more than three times the bare number. Elevations and obstacle heights appear in either unit depending on the chart and the country.
The fix is the same each time: write the unit next to every number and convert once, with exact factors, using the aviation unit conversion calculator.
Directions measured from the wrong reference
True and magnetic north. Courses measured from a chart’s meridians are true; a compass reads magnetic. The two differ by the local magnetic variation. For a true heading of 090° with 10° of westerly variation, the true heading and magnetic heading calculator gives a magnetic heading of 100°. Apply the variation with the wrong sign, as if it were easterly, and the answer is 080°: a 20° error, twice the variation itself. The sign convention is explained in true heading vs magnetic heading.
Wind from and wind to. Aviation wind direction is always the direction the wind blows from. On a track of 360° with a 20 kt wind from 360°, the headwind and tailwind calculator shows a 20.0 kt headwind. Treat the same figure as the direction the wind is going to, which amounts to entering 180°, and it becomes a 20.0 kt tailwind. At a true airspeed of 110 kt that is the difference between a ground speed of 90.0 kt and 130.0 kt.
Altimeter settings in two unit systems
Altimeter settings are given in inches of mercury in some countries and hectopascals in others. Standard pressure is 29.92 inHg or 1013.25 hPa, and the digits invite confusion: “992” could be read as 29.92 inHg or as 992 hPa.
They are very different pressures. A setting of 992 hPa is 29.29 inHg. For a field at 1,000 ft, the pressure altitude calculator returns 1,001 ft with 29.92 inHg and 1,585 ft with 992 hPa, a gap of almost 600 ft from the same three digits. The altimeter pressure conversion calculator converts between the two, and aviation altimeter settings explained covers what the setting does.
Airspeed where ground speed belongs
Time, fuel for a distance and climb gradient all depend on speed over the ground, not speed through the air. Using true airspeed in their place quietly assumes there is no wind.
| Calculation | With airspeed (110 kt or 90 kt) | With ground speed in a 20 kt headwind |
|---|---|---|
| Time for 200 NM | 1 h 49 min at 110 kt | 2 h 13 min at 90 kt |
| Trip fuel for 200 NM at 8 US gal/h | 14.5 US gal | 17.8 US gal |
| Gradient at 500 ft/min | 333 ft/NM at 90 kt | 429 ft/NM at 70 kt |
The first two rows err on the optimistic side in a headwind: the flight takes 24 minutes longer and uses 3.3 US gal more than the no-wind figure. The gradient row errs the other way in a headwind, and a tailwind reverses all three. The ground speed calculator solves the wind triangle so that the right speed goes into the fuel required calculator and the climb gradient calculator. See understanding ground speed in aviation for the reasoning.
Fuel errors: mixed units and early rounding
Volume and mass. Fuel is measured in gallons or liters by some gauges and in pounds or kilograms by others. Endurance is fuel divided by fuel flow, and both must be in the same unit. With 40 US gal on board and a flow of 8 US gal/h, the fuel endurance calculator gives 5 h 00 min. Enter the fuel as 240 lb and leave the flow as 8, and the result is 30 h 00 min. Converting between volume and mass needs the density of the actual fuel at its actual temperature, which is why the calculators on this site never do it for you.
Rounding too early. Suppose 21.5 US gal were used in 2 h 30 min. The fuel consumption calculator gives 8.60 US gal/h. With 40 US gal usable, that flow gives 4 h 39 min of endurance. Round the flow to 9 first and the answer becomes 4 h 27 min. Twelve minutes have disappeared into a rounding step. Carry full precision through the working and round only the final answer.
Rules of thumb used outside their range
Mental shortcuts are approximations that work near the conditions they were tuned for.
- 1,000 ft per inHg. The exact relationship is a power law. With a setting of 29.63 inHg at sea level the rule gives 290 ft against an exact 270 ft. At 28.00 inHg the rule gives 1,920 ft against 1,825 ft, an error of 95 ft that keeps growing as pressure falls further from standard.
- 120 ft per °C above ISA. For a 5,000 ft field at 35 °C with a standard setting, the density altitude calculator gives 8,330 ft from the full formula and 8,590 ft from the rule, a 260 ft difference at about 30 °C above standard.
Use a rule of thumb to catch gross errors in a calculated answer, and the exact formula for the answer itself.
What a correct calculation still cannot tell you
Avoiding these mistakes makes the arithmetic right. It does not make the inputs right. Winds change along a route and with altitude, fuel flow varies with power setting and phase of flight, variation shifts with place and time, and none of the formulas here knows anything about a particular aircraft’s performance or limits.
These calculators are educational. Real flight planning uses current official charts and weather information, performance and fuel figures from the aircraft flight manual (POH/AFM), and the reserves and procedures set by the applicable regulations and the operator.
