Calculation Methodology

This page explains how AviationCalc produces its numbers, so that you can judge each result for yourself and reproduce it by hand.

Where the formulas come from

The calculators use standard textbook relationships. Each calculator page lists the public references that describe its formula. Across the site these are the FAA Pilot’s Handbook of Aeronautical Knowledge, Airplane Flying Handbook and Aeronautical Information Manual; NASA Glenn Research Center’s Beginner’s Guide to Aeronautics; the U.S. Standard Atmosphere, 1976 and the ICAO standard atmosphere; U.S. National Weather Service formula pages for pressure and density altitude; and NIST Special Publication 811 for units. These organizations have not reviewed, approved or endorsed AviationCalc.

Unit conversion conventions

Every input is converted to one internal unit before the formula is applied, and the result is converted back for display. The factors are exact definitions where they exist:

Because 29.92 inHg is itself a rounded figure, it converts to 1013.21 hPa, not exactly 1013.25 hPa. A pressure altitude worked from 29.92 inHg therefore differs from the elevation by about one foot.

Fuel volume and fuel mass are never converted into each other. That conversion needs the density of the actual fuel at its actual temperature, which the site does not know.

Standard atmosphere assumptions

Atmospheric calculations use the International Standard Atmosphere for dry air, from 610 m below sea level to 20 km:

• Sea level: 288.15 K (15 °C), 101,325 Pa (1013.25 hPa), 1.225 kg/m³, speed of sound 340.294 m/s (661.48 kt).
• Temperature falls 6.5 K per kilometer up to 11 km, then stays at 216.65 K up to 20 km.
• Gas constant for dry air 287.05287 J/(kg·K), ratio of specific heats 1.4, standard gravity 9.80665 m/s².
• Altitudes in these formulas are geopotential altitudes.

Humidity is ignored except in the air density calculator, where an optional relative humidity is applied with a Magnus-type approximation for saturation vapor pressure, valid from −40 °C to 60 °C.

Conventions for direction and sign

• Wind direction is the direction the wind blows from.
• Angle differences are wrapped into the range −180° to 180°, and headings into 001° to 360°, so calculations across north behave correctly.
• A positive crosswind is from the right; a negative headwind is a tailwind; a positive wind correction angle is to the right.
• Easterly variation and deviation are positive: true = magnetic + easterly variation, and magnetic = compass + easterly deviation.

Rounding practices

Calculations are carried out in double-precision floating point and rounded only for display. Speeds are shown to 0.1 kt, altitudes to the nearest foot, headings to the nearest degree, Mach numbers to three decimals, and open-ended unit conversions to seven significant figures. The steps under “Show the working” display rounded intermediate values, so re-doing the arithmetic from those displayed values can differ from the final result in the last digit.

Input validation

Every field is checked before anything is calculated. Missing values, text that is not a number, values outside the physically meaningful range for that field, and combinations that have no solution (for example a crosswind larger than the true airspeed in a wind triangle, or a result at or above Mach 1 in the subsonic airspeed formula) are refused with a message beside the field. Division by zero cannot occur because zero is rejected wherever a value is used as a divisor.

Validation procedures

1. Each calculator has at least five automated test cases covering normal values, boundary values, invalid input and unit conversions. There are more than 240 cases in total.
2. Expected values for anchor cases come from hand calculation, published standard-atmosphere tables and exact unit definitions.
3. Every valid case is also computed by a second implementation written separately in another programming language, using different formulations where possible: vector algebra for the wind triangle, imperial units for the aerodynamic and turn equations, pressure ratios for the airspeed chain, and a different great-circle formula. The two must agree to one part in a million.
4. The worked example on each calculator page is generated by running the calculator, so it always matches the live result.

These checks confirm that the arithmetic matches the stated formulas. They are not a certification of any kind.

Known limitations

• The real atmosphere is not the standard atmosphere. Temperature, pressure and humidity vary from the model at every altitude.
• Simple models. Fuel, range and climb calculators assume constant speed and constant fuel flow. They ignore climb, descent, weight change and changing winds.
• Spherical Earth. Great-circle distances use a mean radius of 6,371.0088 km and can differ from ellipsoidal distances by up to about 0.5%.
• Subsonic airspeed formula. The calibrated-to-true airspeed conversion is the subsonic compressible relation and is refused at Mach 1 or above.
• Ideal gas. Density and the speed of sound use ideal-gas relations with a constant ratio of specific heats.
• No aircraft data. Coefficients, corrections, stall speeds, fuel flows and thrust are whatever you enter. The site holds no performance data for any aircraft and applies no limits or reserves.

For these reasons results are for learning only. See the Disclaimer.

Reporting an error

If you believe a result is wrong, please use the contact form and choose “Report a calculation or content error”. Include the calculator, every value and unit you entered, the result shown and the result you expected, with your source or working if you have it. Confirmed errors are corrected, and a new test case is added to the automated suite.