Cost basis is the number a gain or loss is measured from. For shares you bought, it is the purchase price plus any commission. For shares you received as pay, it is the value you were taxed on when you got them: the vest-date price for RSUs, the purchase-date price for ESPP shares, and the market value at exercise for NSOs. Sell for more than the basis and you have a gain; sell for less and you have a loss.
Getting basis right prevents double taxation. Say 100 RSUs vest at $80, and $8,000 is reported as wages on your W-2. Your basis is $8,000. If you sell at $85, the gain is $500, not $8,500. The problem is that brokers sometimes report a basis of $0 for stock compensation, and if you copy that onto your return you pay income tax on the $8,000 twice.
Basis also depends on which shares you sell. If you bought at $50, $70, and $90 and sell one lot, specific identification lets you choose the $90 lot to minimize the gain, or the $50 lot to maximize it for offsetting a loss. The default at most brokers is first in, first out.
The common mistakes are accepting the broker's basis without checking it against your W-2 for equity compensation, and losing track of basis on old holdings or inherited shares (which reset to the value on the date of death). Keeping the vest and purchase confirmations makes tax season far easier.