Proceedings of the Institute of Acoustics

Adding SAS image processing capability to SAR image processing software

A. Crawford, Defence Research and Development Canada, Atlantic Research Centre
J. Secker, Defence Research and Development Canada, Ottawa Research Centre
S. Gong, MDA Ltd., under contract to DRDC Ottawa Research Centre
E. Shouldice, Defence Research and Development Canada, Atlantic Research Centre

1 INTRODUCTION

One aspect of an ongoing collaborative research project at Defence Research and Development Canada (DRDC) Atlantic Research Centre (ARC) is investigation into change detection techniques for seabed monitoring applications. This research focusses on imagery collected using sidelooking sonar, both conventional sidescan (SSS) and synthetic aperture sonar (SAS). This paper discusses recent work on implementation of tools for working with sonar seabed images in existing software developed at DRDC Ottawa Research Centre (ORC) for use with synthetic aperture radar (SAR) images, Image Analyst Pro (IA Pro).

Change detection (CD) is a method of finding objects or areas of interest by virtue of differences in imagery between present and historical surveys^1,2. It is a powerful technique for reducing false alarm rate and is particularly useful in cluttered environments or where an area can only be surveyed periodically. In general, the CD data processing chain consists of: a) selecting present and historical images, b) co-registering the images, c) generating a difference map and d) detecting changes of relevance. As applied to sonar seabed imagery (SSS or SAS), a precursor to this chain is preparation of the raw sonar data into geo-referenced images of the seabed. Some CD techniques utilize the phase part of complex-valued image data, primarily from SAS, referred to as Coherent CD (CCD). Techniques that work on image amplitude are referred to as Incoherent or Non-coherent CD (ICD or NCD). CCD techniques offer the possibility of detecting changes to a scene that are subtle enough to be invisible in amplitude, at the cost of requiring highly accurate image co-registration (within a fraction of an acoustic wavelength).

Change detection has been in operational use by the Canadian Navy for sonar image post-mission analysis (PMA) for well over 20 years. To date however, tools specifically designed to assist operators in doing change detection PMA have been lacking. Commercial software provides very good functionality for preparation of raw sonar data into geo-referenced images, and organizational methods for curating survey data have been developed by practice. Operators have relied on a process of identifying contacts and comparison of seabed images using “blink” or “swipe” tools, and on operators building knowledge of areas that are routinely surveyed. In other words, change detection is being done manually, which is demanding in both time and mental effort. This motivates development of automated change detection (ACD) tools for PMA. Automation will be increasingly valuable, even necessary, as the volume of survey data requiring analysis can only increase with wider operational adoption of SAS.

The development described here, incorporation of SAS image processing capability into existing SAR image processing software, was pursued for two reasons: 1) research in progress can benefit from a platform for experimentation with tools and techniques, and 2) work that is at or near completion can be demonstrated to potential users. In the first case, this could include tuning of parameters and testing on image pair datasets by researchers. In the second case, what was desired is a supported, well-developed and polished interface so that demonstrations to operators can run smoothly.

2 IMAGE ANALYST PRO

Image Analyst Pro (IA Pro) is an R&D prototype system developed by DRDC ORC, for development, testing, validation and demonstration of new algorithms for processing and exploitation of space-based SAR and space-based Intelligence, Surveillance and Reconnaissance (ISR) data from multiple-intelligence (multi-INT) systems and sensors, including Electro-Optical/Infrared (EO/IR), long-wave or Thermal Infrared (TIR), Automatic Identification System (AIS) ship self-reporting data and Radio Frequency (RF) detection and geolocation data³. The focus of IA Pro development is on algorithms and tools for Geospatial Intelligence (GEOINT) applications (e.g., ship detection, site monitoring and change detection) and Intelligence Preparation of the Operational Environment (IPOE) applications (e.g., terrain analysis, land-cover classification and shoreline extraction) to address Department of National Defence/Canadian Armed Forces (DND/CAF) requirements. IA Pro is proprietary software, and the intellectual property for IA Pro rests with the Crown.

IA Pro includes algorithms and tools that, for example: (a) permit the display, manipulation, processing and exploitation of civilian and commercial space-based SAR data, often in advance of Commercial Off-The-Shelf (COTS) systems; (b) provide capability that is missing from COTS systems or needs to be packaged differently for DND/CAF use; (c) address the DND/CAF requirements for exploitation of RADARSAT-1 (R1), RADARSAT-2 (R2) and RADARSAT Constellation Mission (RCM) data; and (d) introduce automation not present in COTS software. Note that DRDC developed (i) Geospatial Data Abstraction Library (GDAL) C++ driver code to open and ingest the C-band SAR images from RCM, and (ii) Python code to extract meta-data from the C-band SAR images, and they released these to the open-source community via GitHub⁴.

IA Pro development has been focused on civilian and commercial SAR satellites that are relevant to DND/CAF, including Canada’s R1, R2 and RCM, and the European Space Agency’s (ESA’s) Sentinel 1, and Germany’s TerraSAR-X and TanDEM-X. IA Pro provides full support for single, dual, quad and compact polarization SAR data, for both complex-valued and real-valued (i.e., detected) SAR data, and for multi-temporal and multi-sensor data processing and exploitation. IA Pro permits the user to incorporate geospatial information for contextual awareness (e.g., thematic vector layers, digital maps) into the multi-INT data exploitation process.

Figure 1: IA Pro primary window displaying a RADARSAT-2 Spotlight mode image acquired over west Ottawa.

IA Pro is developed in an R&D environment: however, factors considered during development included: speed and capability for image manipulation; support for required file formats; analysis tools; algorithm processing speed; graphical user interface (GUI) and graphical toolbar; and image annotation tools. The result is that IA Pro performs image manipulation with speed comparable to that of commercial Electronic Light Table (ELT) software, and it supports multi-layer display of raster and vector data. IA Pro version 3.1 is a significant upgrade from version 2.x. It was re-engineered to a 64-bit Windows 10 application, the software components (including Python, GDAL, and the GNU Toolkit, GTK) were updated to newer versions, and the software development was harmonized to Microsoft Visual Studio 2019, using Git for version control. Note that IA Pro is also built on the OpenEV software library and application for viewing and analyzing raster and vector geospatial data⁵.

Figure 1 shows the main window of IA Pro version 3.1.3. The IA Pro menu items (File, Project, Tools, etc.) are located along the top of the IA Pro window. Below this are two graphical toolbars: the upper row of buttons is primarily for image contrast adjustment and zoom level, while the second row of buttons drive specific algorithms. Information on the R2 image is given in the lower panel. Information on the pixel (cursor) location is given along the status bar at the bottom of the window frame. In this image, a R2 Spotlight mode image acquired on 24-Apr-2015 at 11:05:48 UTC on a descending orbit over west Ottawa is displayed. The azimuth and range directions are indicated in the upper left legend, and the North direction (upwards) is given in the upper right of the image. The background EO/IR satellite imagery was loaded using IA Pro’s Web Map Service (WMS) tool. Between the R2 image and the background EO/IR imagery is an airborne EO/IR image, most visible because of the dark (zero-valued) region near the center of the IA Pro image display window. Finally, a user-defined vector polygon Area of Interest (AOI) is visible as the yellow rectangle.

Figure 2 shows a dual-polarization RCM ScanSAR mode image collected by RCM-3 on 21-Dec-2020 at 10:54:51 UTC on a descending orbit using the Medium Resolution 50-metre beam mode. When delivered, the image data consisted of 492 separate image chips (GeoTIFF) for each of the two polarizations (Horizontal transmit-Horizontal receive, HH, and Horizontal transmit-Vertical receive, HV). To show the RCM image in Figure 2, the IA Pro Stitch RCM ScanSAR SLC tool was used to construct this image from the 492 separate image chips for each of the two polarizations. This data product formerly could not be displayed in IA Pro or other systems, and the Stitch RCM ScanSAR SLC tool was developed with the following processing steps: (i) mosaic individual image chips to yield the full ScanSAR image; (ii) modify the product.xml and noiseLevels_##.xml; (iii) for GeoTIFF products, combine ground control points (GCPs) from individual image chips; and (iv) for NITF products, use individual image chip corner longitude and latitude as GCPs.

Figure 2: The output of the IA Pro Stitch RCM ScanSAR SLC tool, constructed from 492 separate GeoTIFF image chips for each of HH and HV polarizations.

3 IMPLEMENTING SAS IMAGE PROCESSING CAPABILITY

This work has started with importing sonar imagery, by implementing a “Sonar” image layer in IA Pro to ingest meta-data such as acquisition date/time, frequency, sensor roll/pitch/yaw/sway, platform speed and altitude, sound speed, and geo-referencing information. This meta-data can then be displayed in the panel beneath the image. The cursor position row and column numbers and corresponding geographic coordinates (longitude and latitude) are displayed on the bottom status bar. This meta-data is included in a text file that accompanies each sonar image. An example of imported SAS images is shown in Figure 3.

Once the sonar images are imported, IA Pro has existing functionality for the full suite of standard image manipulation tasks (e.g. colour map adjustment, layer management, accessing web map services, etc.), as discussed in the previous section.

In most senses, from the software point of view there is little difference between sonar images and SAR images. For better display of sonar images, the NODATA field is set to be zero so that the zero padded edge is transparent. Sonar amplitude images can have one band or three depending on the colour mapping of the image, not on frequency bands – in future, sonar systems with multiple frequencies could be handled in IA Pro using image bands. The inherent differences between sonar and SAR data, such as ground resolution and signal-to-noise characteristics, will impact the processing techniques subsequently applied to the imported data.

Figure 3: Example of Kraken Robotics SAS images loaded in IA Pro, with geographic and other information taken from meta-data files. The zoom inset was added to the Figure later.

The first sonar image example in Figure 3 shows imported SAS images overlaid on a raster chart of Halifax Harbour in IA Pro v.3.1.5, a version which includes new functionality for loading sonar images. The information included in the bottom banner is taken from the accompanying meta-data file for the topmost sonar image. The current cursor position in geographic coordinates is shown across the bottom. The images were ingested as the new “Sonar” type, not as generic geo-referenced images, demonstrating basic functionality to import sonar image data with accompanying position and other information, as would be required for follow-on analysis. The zoomed in inset was added to the Figure later, to better show the interesting features in the SAS images.

The second sonar image example (Figure 4) shows a pair of co-registered Centre for Maritime Research and Experimentation (CMRE) MUSCLE SAS images, using the Two Colour Multi-View (2CMV) display tool. The image pair is part of the Coalition Underwater Mine and IED Defeat (CUMID) common data set, curated as a research tool for change detection algorithm development and testing. In the two-colour scheme, the underlying image is plotted in reds (the lower edge extends below the overlying image at the bottom) and the overlying image is plotted in cyan. Differences between the two images show as red and cyan, i.e. the new target in the upper left corner and small changes in the ripple field, while persistent features, such as the targets in the upper right quadrant, are in grey scale.

Figure 4: Example showing co-registered CMRE MUSCLE SAS image pair from the CUMID common data set, displayed using 2CMV tool.

The final example, Figure 5, shows output from the Raster Band Calculator/Incoherent Change tool, using the same pair of co-registered CMRE MUSCLE SAS images as shown in Figure 4. The new object in the upper left quadrant is very obvious in this change map. The images were pre-processed to co-register to sub-pixel accuracy prior to importation into IA Pro. Eventually, co-registration tools could be implemented within IA Pro. Figures 4 and 5 illustrate examples of existing display and processing tools developed for SAR images applied to SAS images.

4 FUTURE PLANS

Change detection is a technique based on detecting anomalies between two images that are temporally separated, whether those images are collected using radar or sonar. Automation of change detection processing will make adoption of these techniques more attractive to civilian or military operators. DRDC ORC has developed IA Pro, proprietary software for visualizing SAR imagery and performing PMA using various techniques including change detection. Recently, IA Pro has been expanded to include functionally for intake of sonar images.

Figure 5: Output from IA Pro Incoherent Change tool, using the same CMRE MUSCLE SAS image pair as shown in Figure 4.

The work described here lays the foundation for development of ACD tools for SSS and SAS seabed imagery and the ability to demonstrate these tools to operators. The latest version of IA Pro can ingest sonar imagery, with accompanying meta-data files. Near-future plans include adding ability to import raw sonar data in native format. Subsequent development will then include incorporation of image co-registration tools, automated ICD and CCD algorithms and area processing of image statistics specific for sonar image data. As future development of tools to work with sonar images progresses, the meta-data requirements may change and it should be possible to automate generation of the meta-data files when working from raw sonar data.

There is already considerable functionality built into the existing software for more sophisticated image analysis than the examples that have been shown here, developed over years for SAR image processing. Not all of these will be suitable for use with sonar images, but the possibilities are exciting for cross-over of tools and techniques between the SAR and SAS communities of practice.

5 ACKNOWLEDGEMENTS

Figures 1 and 2: RADARSAT-2 Data and Products © MacDonald Dettwiler and Associates Ltd. (2015 Fig. 1 and 2020 Fig. 2) – all rights reserved. RADARSAT is an official trademark of the Canadian Space Agency. Applanix airborne EO/IR image © Canadian Armed Forces Mapping and Charting Establishment (2005) - all rights reserved (Fig. 1).

Figure 3: Background nautical chart © the Crown.

6 REFERENCES

1. V.L. Myers, D.D Sternlicht, A.P. Lyons and R.E Hansen., “Automated seabed change detection using sythetic aperture sonar: current and future directions,” Proc. Institute of Acoustics, Vol. 36, No. 1, 77-86 (September 2014).
2. Ø. Midtgaard, R. E. Hansen, T. O. Sæbø , V. Myers, J. R. Dubberley and I. Quidu., “Change detection using synthetic aperture sonar: preliminary resuls from the Larvik trial,” Proc. MTS/IEEE OCEANS’11, 1-8, Kona, HI (2011). doi: 10.23919/OCEANS.2011.6107272.
3. J. Secker and P. W. Vachon, "Exploitation of multi-temporal SAR and EO satellite imagery for geospatial intelligence," 2007 10th International Conference on Information Fusion, Quebec, QC, Canada, 2007, pp. 1-8, doi: 10.1109/ICIF.2007.4408199.
4. GitHub/DND-DRDC-RDDC,“OS_DriverMetadataExtractionCodes_RCM_C-band_SARdata”. Accessed 11-Jul-2023 online at: https://github.com/DND-DRDC-RDDC/OS_DriverMetadataExtractionCodes_RCM_C band_SARdata
5. Sourceforge.net, “What is OpenEV? Accessed 11-Jul-2023 online at: https://openev.sourceforge.net/