Mixed reality visualization system for 3D sound intensity using PAGE method and spatial interpolation

Ayame Uchida 1

Tokyo Denki University 5 Senju-Asahi-Cho, Adachi-ku, Tokyo, JAPAN

Yukiko Okawa 2

Tokyo Denki University 5 Senju-Asahi-Cho, Adachi-ku, Tokyo, JAPAN

Yusuke Ikeda 3

Tokyo Denki University 5 Senju-Asahi-Cho, Adachi-ku, Tokyo, JAPAN

Yasuhiro Oikawa 4

Waseda University 3-4-1 Ookubo, Shinjuku-ku, Tokyo, JAPAN

ABSTRACT In our previous study, we proposed the Mixed Reality visualization system for maps of three- dimensional sound intensities measured by a handy 4-ch microphone array. This system had a few shortcomings. The sound intensity was estimated by the cross-spectrum method. Because of this, the estimation errors at higher frequencies become larger owing to the spatial Nyquist frequency determined by the microphone intervals. Another problem is that the clarity of the sound intensity map is degraded because the intervals between the measurement points on the sound intensity map are irregular. In addition, the microphone array was moved manually. In this study, we improved the previous mixed reality system for sound intensity by applying the PAGE method (phase and amplitude gradient estimation), which can accurately estimate sound intensity up to higher frequencies. Furthermore, by using spatial interpolation of an irregularly sampled intensity map, we improved the clarity of the sound intensity map using mixed reality. The proposed system interpolates the sound intensities at equal intervals from those measured at irregular intervals. For evaluation, we conducted visualization experiments on the sound intensities surrounding a single loudspeaker.

1 22fmi08@ms.dendai.ac.jp

2 21fmi03@ms.dendai.ac.jp

3 yusuke.ikeda@mail.dendai.ac.jp

4 yoikawa@waseda.jp

a slaty. inter.noise 21-24 AUGUST SCOTTISH EVENT CAMPUS O ¥, ? GLASGOW

1. INTRODUCTION

Various sound field visualization methods have been proposed, such as beamforming [1] and acoustic holography [2]. Recently, several methods of data mapping in real space have been proposed for data visualization [3]. Inoue and Kataoka et al. proposed a real-time visualization system of sound intensities using mixed reality (MR) and a handheld microphone array [4, 5]. These systems make it easier to observe sound propagation by scanning a steady sound field with a handheld microphone array and then displaying the measured sound intensity in real space using mixed reality. Furthermore, Watanabe et al. proposed an MR visualization system for room impulse responses (RIR) that estimates sound intensities along the moving path of a microphone array [6]. In previous systems [5], because the sound intensity is estimated using the cross-spectral method [7], the estimation accuracy at higher frequencies is degraded owing to the spatial Nyquist frequency, which corresponds to the interval of microphones. Furthermore, because the sound intensity is measured by a freely moving microphone, the interval between measurement points becomes irregular. In addition, even if we want to visualize the sound propagation on a 2D plane to make the data more visible, it is di ffi cult to measure sound intensities on the same plane. Thus, the sound intensity data is spread irregularly, which reduces the distinguishability of the data. By extending our previous MR visualization system [5], we propose a visualization system to improve the clarity of the sound intensity map by spatially interpolating the sound intensities measured at irregular intervals into equally spaced intervals. Furthermore, to estimate the sound intensity at higher frequencies more accurately, the PAGE method is introduced into the MR visualization system. In the experiments, the sound intensities near the loudspeaker are measured by freely moving the handy microphone array and interpolated at equal intervals on a 2D grid for ease of visualization.

2. SYSTEM OVERVIEW

Figure 1 presents an overview of the proposed system. In this system, the user wears a Microsoft Hololens 2, a mixed reality device, and scans the space for the visualization target using a handy 4ch microphone to measure the sound pressure signal. Sound intensity is estimated from the sound pressure signal obtained from the measurement using the method described below and then interpolated to visualize it at equal intervals. The microphone’s measurement position was estimated by combining two AR markers placed at the reference position in the measurement room and attached to the microphone array [5]. The position of the MR headset was continuously estimated by simultaneous localization and mapping (SLAM) using Microsoft HoloLens2 [8].

3. SPATIAL INTERPOLATION OF SOUND INTENSITIES

3.1. Estimation of sound intensity using PAGE method Tomas et al. proposed the PAGE method [9], which achieves estimation of sound intensity at frequencies higher than the Nyquist frequency determined by the microphone interval. First, based on the least squares method, the phase gradient c ∇ ϕ is estimated as follows:

c ∇ ϕ ( X T X ) − 1 X T δ ( ϕ ) , (1) X [ r 2 − r 1 | r 3 − r 1 | ... | r N − r N − 1 ] T , (2)

δ ( ϕ ) " arg ( p 2

) , arg ( p 3

) , . . . , arg ( p N

)# T , (3)

p 1

p 1

p N − 1

where X denotes the vector of pairwise di ff erences, r i ( i = 1 , . . . , N ) denotes the position of the microphone, δ ( ϕ ) denotes the vector of pairwise phase di ff erences, and p i ( i = 1 , . . . , N ) denotes

Microsoft HoloLens2

Creation of conical 3DCG

Sound intensity

Mic. shape : Regular tetrahereon

Sencer SLAM

Speaker

Origin marker

HoloLens2s’ global coordinates

Relative coodinates of mic.

Mic.marker

4ch handy microphone

Mic. position

PC

Estimation of acoustic intensity with PAGE m ethod

Spatial interpolation at grid points

Measurement data

Camera detection Data transmisson

Figure 1: Overview of the proposed system. By using SLAM and AR makers, the measurement of the sound intensity with a 4-ch handy microphone array is stored on a PC with its positional information. For visualization, the stored sound intensities at irregular intervals are spatially interpolated on a grid with regular intervals.

the complex sound pressure. Considering the spatial Nyquist frequency due to the microphone interval, the phase gradients are obtained by applying phase unwrapping to each complex pressure. Then, the active intensity I a , which is the real part of the sound intensity, is estimated from Eq. 1 as follows:

I a = 1 ωρ 0 P 2 0 c ∇ ϕ, (4)

where P 0 is the amplitude at the center of the microphone array, ω is the angular frequency, and ρ 0 is the atmospheric density.

®, =

3.2. Spatial interpolation of sound intensity From the measurement data at irregular intervals, data at regular intervals were obtained by spatial interpolation of sound intensities. First, Delaunay triangulation [10] was performed at all measurement points to divide the measurement region. Subsequently, the active intensity P 2 0 c ∇ ϕ at an arbitrary point is obtained by interpolation in the corresponding segmented region. That is, it is interpolated using the three measurement points that form the vertices of the segmented region triangle. Let us consider two well-known methods of spatial interpolation: the nearest-neighbor algorithm and linear interpolation. In the simulation experiments, we compared the interpolated values of the sound intensities at equally spaced grid points with the measured sound intensities at random positions to determine the interpolation accuracy. The cross-spectral and PAGE methods were used to estimate the sound intensity at each point. Tables 1 and 2 list the input conditions of the simulation experiments. The target frequency was a 1 kHz centered octave band. The following equation was

Sound source

Y

0.5 m

Measurement point

X

Interpolation grid

0 m

ery ryy

0.1 m

2.0 m

0 m 2.0 m

ry ry 08 os

Figure 2: Arrangement of simulation experiments to compare the spatial interpolation methods. Black circles indicate the measurement positions. The blue squares are the grid areas for interpolation. The point source is placed at a distance of 0.5 m from the upper edge of grid.

Linear interpolation Nearest neighbor algorism

( i ) ( ii )

0

CrossSpectral method PAGE method

-5

y[m] y[m]

-10

-15

Error [dB]

( iii ) ( iv )

-20

( iii ) ( iv )

-25

-30

-35

-40

x[m] x[m]

Figure 3: Comparison of spatial interpolation methods on the grid points from random positions using simulation results. (i)Linear interpolation with the cross-spectrum method. (ii) Nearest neighbor algorithm with the cross-spectrum method. (iii) Linear interpolation with the PAGE method. (iv) Nearest neighbor algorithm with the PAGE method.

Table 1: Input conditions for the simulation experimen ts

Atmospheric density[kg / m 3 ] 1.293

Distance between microphones[m] 0.05

Sampling frequency[kHz] 48

Measurement point(before interpolation) 221

Grid point of interpolation 21 × 21

Distance between grid point[m] 0.1

Analysis length [Sample] 4096

Frequency range of phase unwrap [kHz] 0.5 - 16

used to evaluate interpolation accuracies:

Error = 10 log 10 ∥ I interp − I true ∥ 2 2 ∥ I true ∥ 2 2 , (5)

where I interp is the estimated sound intensity and I true is the desired sound intensity. Figure 3 shows the simulation results of linear interpolation and the nearest-neighbor algorithm. As shown in Figs.3 (i) and (iii), in the case of linear interpolation on the grid, the errors were approximately -25 dB or less, except for the data on the edges of the grids. From Figs. 3(ii) and (iv), in the case of the nearest neighbor algorithm, the errors were approximately -15 dB except when the position of random measurement points and the grid position were the same. Furthermore, we compared the interpolation accuracies of di ff erent sound intensity estimation methods. Comparing Figs.3(i) and (iii), (ii) and (iv) respectively, we see that the interpolation accuracies in both the cross-spectral and PAGE methods had no significant di ff erences. Therefore, we used linear interpolation of the grid points in the proposed system to visualize the sound intensities.

4. VISUALIZATION EXPERIMENT

4.1. Condition To evaluate the improvement in data discernment by spatial interpolation on the grid, we conducted visualization experiments near the sound source using the proposed system in a soundproof room at Tokyo Denki University (background noise of less than 20 dB). Table 2 lists the experimental conditions used. The cross-spectral and PAGE methods were used for sound intensity estimation at each point, and linear interpolation was used for spatial interpolation. The target frequencies were 1 kHz and 8 kHz centered octave bands. The measurement signal emitted from the loudspeaker was white noise.

4.2. Results Figures 4(a) and (b) show the visualization results at the 1 kHz and 8 kHz centered octave bands, respectively. Figures. 4(a)(i) and (iii) show the maps of sound intensities at original measurement points for the 1 kHz-centered octave band. As shown in the figures, the intervals of the original measurement points were irregular because of the movement of the handheld microphone array used to measure the sound intensities. As shown in Figs. 4(a)(i) and (iii), the original sound intensity maps are confusing and it is di ffi cult to understand sound propagation from them. Figures. 4(a)(ii) and (iv) shows the map of sound intensities at grid points by spatial interpolation. A comparison of the sound intensity maps in Figs.4(a)(i) and (ii) shows that spatial interpolation improves the clarity of the maps. Figures. 4(a)(ii) and (iv) on the right show the small di ff erences of sound intensity directions between the cross spectrum and PAGE methods. The same di ff erences can be observed in Figs.

(a) 1 kHz-centered octave band

No spatial interpolarion With spatial interpolation

( i ) ( ii )

) VK ft ail iinunan YAUVLAHS 6 1

80

CrossSpectral method PAGE method

Sound Intensity Level [dB]

75

AA Oa aos

70

( iii ) ( iv )

65

60

(b) 8 kHz-centered octave band

No spatial interpolarion With spatial interpolation

( i ) ( ii )

80

CrossSpectral method PAGE method

Sound Intensity Level [dB]

75

70

( iii ) ( iv )

poe IE YD lg ed Lhbeoe

65

oe. 4 fe EX PAATA NG Se Py ‘ —~ & oe east 1 5

60

Figure 4: Comparison of the sound intensity map using the actual experiment data. (a) Sound intensity map at the 1-kHz centered octave band (b) Sound intensity map at the 8-kHz centered octave band (i) No spatial interpolation (original data) using the cross-spectral method . (ii) Spatial interpolation of sound intensities using the cross-spectrum method. (iii) No spatial interpolation (original data) using the PAGE method. (iv) Spatial interpolation of sound intensities using the PAGE method. Each cone depicts the sound intensity at each point. The color of the cone indicates the sound intensity level. The vertex of cone indicates the direction of sound intensity.

Table 2: Input conditions for the visualization experiments Atmospheric density[kg / m 3 ] 1.293

Interval of microphones[m] 0.05

Sampling frequency[kHz] 48

Number of measurement points 221

Number of grid points for interpolation 17 × 11

Interval of grid points[m] 0.05

Analysis length [Sample] 4096

Frequency range of phase unwrap [kHz] 0.5 - 16

Sound pressure level (1 m from loudspeaker) [dB] 84

4(a)(i) and (iii). This is because the estimation using the PAGE method has small errors due to sound reflections from the measurement instrument [9]. Figure. 4(b)(ii) shows the spatial interpolation of sound intensities using the cross-spectral method. According to the figures, most of the sound intensities around the loudspeaker were directed in di ff erent directions than those in the 1-kHz band (Fig.4(a)(ii)). This is because the cross-spectral method is known to cause sound intensity estimation errors at high frequencies, owing to the spatial Nyquist frequency. From Fig. 4(b)(iv), in the PAGE method, most directions of sound intensities were aligned compared to cross-spectral method. At 8 kHz, the sound intensity map was more disturbed than that at 1 kHz because it was more sensitive to sound reflections from the measurement instrument itself.

5. CONCLUSIONS

In this study, to improve the clarity of the sound field intensity map, we propose a mixed reality visualization system using the spatial interpolation of sound intensity distributions with irregular intervals. Based on the visualization experiments, the discernment of the sound intensity data was improved. However, a small error was observed in the actual measurements when PAGE was used. In future work, our goal will be to investigate the details of the errors and reduce the errors of the PAGE method in actual measurements.

ACKNOWLEDGEMENTS

This work was supported by JSPS KAKENHI Grant Numbers 19K1249 and 22K12099.

REFERENCES

[1] J. C. Chen et al. ,“Source localization and beamforming,” Vol.19(2), pp. 30–39, IEEE Signal Processing Magazine 2002. [2] J. Maynard, E. Williams, and Y. Lee, “Nearfield acoustic holography: I.theory of generalized holography and the development of NAH,” Journal of the Acoustical Society of America, vol. 78, 11, 1985. [3] Y. Fukushima, H. Suzuki, and A. Omoto, “Visualization of reflectedsound in enclosed space by sound intensity measurement,” Acoustical Science and Technology, vol. 27, no. 3, pp. 187–189, 2006.

[4] A. Inoue et al. ,“Visualization system for sound field using see-through head mounted display,” Vol.40, pp. 1–11, Acoust. Sci. Tech., 2019. [5] Y. Kataoka et al. , “Real-time measurement and display system of 3D sound intensity map using optical see-through head mounted display,” SIGGRAPH Asia 2018 Posters, 2018. [6] Y. Watanabe, Y. Ikeda, Y. Kataoka, Y. Oikawa, and N. Osaka, “Visualization of spatial impulse responses using mixed reality and movingmicrophone,” SIGGRAPH 2020 Posters, 2020. [7] J. Y. Chung,“Cross-spectral method of measuring acoustic intensity without error caused by instrument phasemismatch,” The Journal of the Acoustical Society of America,64(6), 1613–1616, 1978. [8] Microsoft, “Microsoft HoloLens 2,” https: // www.microsoft.com / hololens. [9] Derek. C. Tomas et al. ,“Phase and amplitude gradient method for the estimation of acoustic vector quantities,” Acoustical Society of America 137, pp.3366–3376, 2015. [10] Amidror Isaac, “Scattered data interpolation methods for electronic imaging systems: a survey,” Journal of Electronic Imaging. Vol.11, No.2, pp.157–176, 2002.