Undistorting a cheap fisheye webcam without losing the corners

The standard pinhole model falls apart beyond about 120° field of view. cv2.fisheye handles it much better, but the default output crops away a surprising amount of the image.

Calibrate with the fisheye model

flags = (cv2.fisheye.CALIB_RECOMPUTE_EXTRINSIC
         | cv2.fisheye.CALIB_CHECK_COND
         | cv2.fisheye.CALIB_FIX_SKEW)
rms, K, D, rvecs, tvecs = cv2.fisheye.calibrate(
    objpoints, imgpoints, gray.shape[::-1], None, None, flags=flags,
    criteria=(cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 1e-6))

Use at least 25 views of the board, and make sure some of them put the board into the corners of the frame. That is where the distortion lives.

Keep the corners

The trick is the balance parameter of estimateNewCameraMatrixForUndistortRectify: 0 crops to valid pixels only, 1 keeps everything.

newK = cv2.fisheye.estimateNewCameraMatrixForUndistortRectify(
    K, D, (w, h), np.eye(3), balance=0.6)
map1, map2 = cv2.fisheye.initUndistortRectifyMap(
    K, D, np.eye(3), newK, (w, h), cv2.CV_16SC2)
out = cv2.remap(frame, map1, map2, cv2.INTER_LINEAR)

Compute the maps once and reuse them: remap alone runs at well over 100 fps on 720p.