TFG|镜头校准

语法糖
, ,

摄像机(Camera)也是渲染过程中不可缺少的一部分，这个实验将介绍摄像机如何。本文为官方提供的入门项目Camera calibration的阅读笔记，代码均摘自源文件中。

摄像机的参数分为内参和外参。内参是与相机自身特性相关的参数，比如相机的焦距、像素大小等。外参是在世界坐标系中的参数，比如相机的位置、旋转方向等。

内参
为什么叫内参呢，这个是因为这些参数是只有相机来决定的，不会因为外界环境而改变。 相机的内参是：1/dx、1/dy、r、u0、v0、f
opencv中的内参是4个，分别为fx、fy、u0、v0。其实opencv中的fx也就是F*Sx，其中F是焦距上面的f，Sx是像素/每毫米也就是上面的1/dx。
dx和dy表示x方向和y方向的一个像素分别占多少个单位，是反映现实中的图像物理坐标关系与像素坐标系转换的关键（我理解的是可以反映像元密度）。
u0，v0代表图像的中心像素坐标和图像原点像素坐标之间相差的横向和纵向像素数。

Camera calibration

摄像机将一个3D物体映射到2D平面上，这一过程同现实世界中的相机一样，可调节的内参非常多，如镜头变形(lens distortion)、感光(ISO)、焦距(focal length)以及曝光时间(exposure time)等。

在这个算法中我们选取两个内参作为要预测的值：

  • 主点(Principal point)：光学中心(optical center)在图像上的投影，理想情况下与图像中心重合。
  • 焦距：光学中心距离图像面的长度，这个参数可以控制缩放。

通过获取相机参数，我们可以做更多有意思的事，比如重构3D模型。该实验中我们用一个简单的矩形进行演示。

导入相关包

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np

#################################
# Imports the necessary modules #
#################################

from tensorflow_graphics.math.optimizer import levenberg_marquardt
from tensorflow_graphics.rendering.camera import perspective  

tf.enable_eager_execution()

创建渲染器

我们的目标是输入相机参数，渲染一张包含一个正方形（4个顶点）的图像。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
def render_rectangle(rectangle_vertices, focal, principal_point, image_dimensions):
  """Renders a rectangle on the image plane.

  Args:
    rectangle_vertices: the four 3D corners of a rectangle.
    focal: the focal lengths of a projective camera.
    principal_point: the position of the principal point on the image plane.
    image_dimensions: the dimensions (in pixels) of the image.

  Returns:
    A 2d image of the 3D rectangle.
  """
  image = np.zeros((int(image_dimensions[0]), int(image_dimensions[1]), 3))#初始化一张全空白图像(w,h,[r,g,b])
  vertices_2d = perspective.project(rectangle_vertices, focal, principal_point)
  vertices_2d_np = vertices_2d.numpy()
  top_left_corner = np.maximum(vertices_2d_np[0, :], (0, 0)).astype(int)
  bottom_right_corner = np.minimum(
      vertices_2d_np[1, :],
      (image_dimensions[1] - 1, image_dimensions[0] - 1)).astype(int)
  for x in range(top_left_corner[0], bottom_right_corner[0] + 1):
    for y in range(top_left_corner[1], bottom_right_corner[1] + 1):
      c1 = float(bottom_right_corner[0] + 1 -
                 x) / float(bottom_right_corner[0] + 1 - top_left_corner[0])
      c2 = float(bottom_right_corner[1] + 1 -
                 y) / float(bottom_right_corner[1] + 1 - top_left_corner[1])
      image[y, x] = (c1, c2, 1)
  return image

下面我们定义一个初始的内参作为对照，并将它渲染出来。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# Sets up the vertices of the rectangle.
rectangle_depth = 1000.0
rectangle_vertices = np.array(
    ((-150.0, -75.0, rectangle_depth), (150.0, 75.0, rectangle_depth)))

# Sets up the size of the image plane.
image_width = 400
image_height = 300
image_dimensions = np.array((image_height, image_width), dtype=np.float64)

# Sets the horizontal and vertical focal length to be the same. The focal length
# picked yields a field of view around 50degrees.
focal_lengths = np.array((image_height, image_height), dtype=np.float64)
# Sets the principal point at the image center.
ideal_principal_point = np.array(
    (image_width, image_height), dtype=np.float64) / 2.0

# Let's see what our scene looks like using the intrinsic paramters defined above.
render = render_rectangle(rectangle_vertices, focal_lengths, ideal_principal_point,
                          image_dimensions)
_ = plt.imshow(render)

同样的，我们可以随机生成其他状态下的渲染结果：

1
2
3
4
5
6
7
8
9
10
11
12
## 这里可以进行自定义或用随机生成
focal_length_x = 100  #@param { min: 100.0, max: 500.0 }
focal_length_y = 500  #@param { min: 100.0, max: 500.0 }
optical_center_x = 386  #@param { min: 0.0, max: 400.0 }
optical_center_y = 152  #@param { min: 0.0, max: 300.0 }
## 通过该函数生成对应图片
render = render_rectangle(
    rectangle_vertices,
    np.array((focal_length_x, focal_length_y), dtype=np.float64),
    np.array((optical_center_x, optical_center_y), dtype=np.float64),
    image_dimensions)
_ = plt.imshow(render)

拟合开始

先定义几个函数来简化我们的操作：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
## 显示出原图案、预测图案、两者之差
def plot_optimization_step(observation, prediction):
  plt.figure(figsize=(20, 10))
  ax = plt.subplot("131")
  ax.set_title("Observation")
  _ = ax.imshow(observation)
  ax = plt.subplot("132")
  ax.set_title("Prediction using estimated intrinsics")
  _ = ax.imshow(prediction)
  ax = plt.subplot("133")
  ax.set_title("Difference image")
  _ = ax.imshow(np.abs(observation - prediction))
  plt.show()

## 输出误差
def print_errors(focal_error, center_error):
  print("Error focal length %f" % (focal_error,))
  print("Err principal point %f" % (center_error,))

Let’s now define the values of the intrinsic parameters we are looking for, and an initial guess of the values of these parameters.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
def build_parameters():
  # Constructs the intrinsic parameters we wish to recover.
  real_focal_lengths = focal_lengths * np.random.uniform(0.8, 1.2, size=(2,))
  real_principal_point = ideal_principal_point + (np.random.random(2) -
                                                  0.5) * image_width / 5.0

  # Initializes the first estimate of the intrinsic parameters.
  estimate_focal_lengths = tf.Variable(real_focal_lengths +
                                       (np.random.random(2) - 0.5) *
                                       image_width)
  estimate_principal_point = tf.Variable(real_principal_point +
                                         (np.random.random(2) - 0.5) *
                                         image_width / 4)
  return real_focal_lengths, real_principal_point, estimate_focal_lengths, estimate_principal_point

As described earlier, one can compare how the 3D object would look using the current estimate of the intrinsic parameters, can compare that to the actual observation. The following function captures a distance beween these two images which we will seek to minimize.

1
2
3
4
5
6
7
def residuals(estimate_focal_lengths, estimate_principal_point):
  vertices_2d_gt = perspective.project(rectangle_vertices, real_focal_lengths,
                                       real_principal_point)
  vertices_2d_observed = perspective.project(rectangle_vertices,
                                             estimate_focal_lengths,
                                             estimate_principal_point)
  return vertices_2d_gt - vertices_2d_observed

All the pieces are now in place to solve the problem; let’s give it a go!

Note: the residuals are minimized using Levenberg-Marquardt, which is particularly indicated for this problem. First order optimizers (e.g. Adam or gradient descent) could also be used.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# Samples intrinsic parameters to recover and an initial solution.
real_focal_lengths, real_principal_point, estimate_focal_lengths, estimate_principal_point = build_parameters(
)

# Contructs the observed image.
observation = render_rectangle(rectangle_vertices, real_focal_lengths,
                               real_principal_point, image_dimensions)

# Displays the initial solution.
print("Initial configuration:")
print_errors(
    np.linalg.norm(estimate_focal_lengths - real_focal_lengths),
    np.linalg.norm(estimate_principal_point - real_principal_point))
image = render_rectangle(rectangle_vertices, estimate_focal_lengths,
                         estimate_principal_point, image_dimensions)
plot_optimization_step(observation, image) # 显示最初始时的结果

# Optimization.
_, (estimate_focal_lengths,
    estimate_principal_point) = levenberg_marquardt.minimize(
        residuals, (estimate_focal_lengths, estimate_principal_point), 1)

print("Predicted configuration:")
print_errors(
    np.linalg.norm(estimate_focal_lengths - real_focal_lengths),
    np.linalg.norm(estimate_principal_point - real_principal_point))
image = render_rectangle(rectangle_vertices, estimate_focal_lengths,
                         estimate_principal_point, image_dimensions)
plot_optimization_step(observation, image) # 显示预测完成后的结果

Initial configuration:
Error focal length 187.253365
Err principal point 49.621742

Predicted configuration:
Error focal length 0.000000
Err principal point 0.000000

References

焦点与焦距

相机内外参数以及畸变参数

计算机视觉学习——相机内参数和外参数

Share