Help Shape the Future of Space Exploration

Join The Planetary Society Now  arrow.png

Join our eNewsletter for updates & action alerts

    Please leave this field empty

Headshot of Alfred McEwen

Future High-Resolution Imaging of Mars: Super-Res to the Rescue?

Posted by Alfred McEwen

29-04-2016 14:27 CDT

Topics: explaining technology, Mars, explaining image processing, Mars Reconnaissance Orbiter

MEPAG recently posted a report from the Next Orbiter Science Analysis Group that describes the need for a new orbiter to launch in 2022 or 2024 to support future Mars robotic and human exploration. Some of the top priorities are to understand active processes and to locate and characterize candidate landing sites and surface exploration regions with HiRISE-class (30-centimeter-per-pixel) or better (10- to 15-centimeter-per-pixel) imaging. They recommend using solar-electric propulsion as well as advanced telecommunications (Ka-band and high-power RF amplifiers, and a laser communication experiment), which together enable a much higher data rate to Earth than past orbiters. Solar-electric propulsion also increases the potential mass of the science instrument payload that can be launched to Mars by a particular rocket. The desire to achieve a spatial resolution better than HiRISE is driven by science objectives such as understanding recurring slope lineae (RSL) and also to support landing site characterization, surface activity planning, and even diagnosis of engineering anomalies (such as how the Beagle-2 lander failed).

A factor of two or three improvement over HiRISE image resolution doesn’t sound very difficult, but it is. Increasing a small camera’s resolution by a factor of two or three is not difficult, but HiRISE -- the largest telescope ever sent to another planet -- is already pushing against several limits. Large telescopes are difficult to accommodate, including the mass, volume, and pointing stability. Resolution is inversely proportional to the diameter of the telescope aperture. HiRISE has a half-meter aperture, so doubling the optical resolution requires a primary mirror about a meter in diameter, which, in turn, enlarges the other mirrors and the structure. So the mass and volume must increase substantially compared to HiRISE. Alternatively, new technologies can reduce the mass of the telescope (like James Webb Space Telescope), but that increases cost and introduces risk, and NASA is very risk-averse.

The challenges don’t stop with simply flying a larger telescope. If the pixel scale is smaller or if more time-delay integration lines are required, then the pointing stability requirement for the entire spacecraft becomes more challenging. This stability challenge may be worsened by flying the large solar arrays needed for solar-electric propulsion.

High-resolution imaging of Mars from orbit is challenging. The atmosphere of Mars causes two problems. First, the orbit cannot be lower than about 250 kilometers (or even 300 kilometers when the air is especially dusty and warm) because of drag on the spacecraft. We have substantial coverage of the Moon at half a meter per pixel from the Lunar Reconnaissance Orbiter Camera (LROC), which is much smaller than the HiRISE camera. The intrinsic resolving power of LROC is about 10 times coarser than HiRISE, but in its prime mission Lunar Reconnaissance Orbiter traveled at an altitude of just 50 kilometers, compared to about 300 for HiRISE. The thin atmosphere of Mars is also surprisingly bright due to suspended dust. As a result, more than half of the brightness of Mars orbital images is from atmospheric scattering; less than half of the light reaching the camera is from direct reflection from the surface that reveals small-scale features. This atmospheric haze can be subtracted from the images to increase contrast, but it still contributes photon noise proportional to the square root of the signal. The solution is to acquire images with a very high signal-to-noise ratio (SNR). The top-of-atmosphere SNR needs to be about 150:1 (a typical value for HiRISE) to insure that the SNR of surface features is at least 50:1 in the majority of images. (Sometimes the dust is so thick that the surface SNR is much worse, approaching zero.)

Another challenge is that the mass of Mars dictates that low circular orbiters must travel at a high velocity of about 3.2 kilometers per second. To image at a scale of 0.3 meters per pixel, this means the line time is only (0.3m)/(3200m/s) = 0.000094 second or 94 microseconds. With HiRISE optics, using that line time as the integration time produces an image with SNR of only 10, rather than 150. The HiRISE solution is use of time-delay integration, imaging each surface patch 128 times and summing the signal on the CCD detector. What if the imaging scale is reduced to 15 centimeters per pixel? Then the integration time is just 47 microseconds and ~256 time-delay integration lines are needed (assuming a 1-meter primary mirror) for adequate SNR, and the columns are half as wide, so the stability must be four times better. The spacecraft pointing stability needs to be very precise to make sure a surface feature is re-imaged in the same TDI column, not jittering or wandering over multiple columns, smearing the image. Mars Reconnaissance Orbiter went to considerable effort (and cost) to achieve high stability for HiRISE. The cost of a factor of two improvement in image resolution is not confined to the cost of the camera; it drives the cost of the entire spacecraft.

Super-Resolution Restoration to the Rescue?

A recent press release from the University College of London presented their Super-Resolution Restoration (SRR) technique, including comparison of regions near Spirit rover (Figure A).

Figure A. Super-Resolution Restoration (SRR)


Figure A. Super-Resolution Restoration (SRR)
Comparisons of portions of map-projected HiRISE image PSP_001513_1655 (top) to SRR processing of 8 HiRISE images (bottom), covering regions near the Spirit rover at “home plate” in Gusev Crater. Boulders are more clearly defined (left) and the rover tracks over home plate are prominent (right) in the SRR images. However, as described below, the “before” images do not show the full resolution of single HiRISE images—see Figure D. Also, note that Spirit rover had not yet arrived at home plate when PSP_001513_1655 was acquired, so it wasn’t SRR that made the tracks appear, it was the rover actually driving over the surface. Other features are sharper in the SRR image, but by how much?

This UCL press release made some startling proclamations:

"Exciting pictures of the Beagle-2 lander, the ancient lakebeds discovered by NASA’s Curiosity rover, NASA’s MER-A rover tracks and Home Plate’s rocks have been released by the UCL researchers who stacked and matched images taken from orbit, to reveal objects at a resolution up to five times greater than previously achieved."

"Co-author Professor Jan-Peter Muller from the UCL Mullard Space Science Laboratory, said: "We now have the equivalent of drone-eye vision anywhere on the surface of Mars where there are enough clear repeat pictures. It allows us to see objects in much sharper focus from orbit than ever before and the picture quality is comparable to that obtained from landers."

"By stacking and matching pictures of the same area taken from different angles, Super-Resolution Restoration (SRR) allows objects as small as 5cm (about 2 inches) to be seen from … stacks of between four and eight 25cm images of the Martian surface taken using the NASA HiRISE camera…"

"In the future, we will be able to recreate rover-scale images anywhere on the surface of Mars and other planets from repeat image stacks" said Mr Yu Tao, Research Associate at UCL and lead author of the paper."

These statements sound too good to be true. A factor of 5 resolution enhancement implies 25 times the information content, yet they only processed 4 to 8 images. In addition, the HiRISE dataset is far from ideal for super-resolution restoration. Here’s what is needed for ideal super-resolution restoration:

  1. High SNR images, preferably >100:1.
  2. Aliased images with a well-measured point-spread function.
  3. A set of images with identical resolution and illumination and viewing angles.
  4. Images are evenly offset by subpixel amounts. For example, acquire 16 images each offset by ¼ pixel increments in both row and column.
  5. Image scene does not change between images.

HiRISE sometimes satisfies #1 above, but the typical SNR of the surface is ~50:1. As for #2, most planetary mission images are aliased (few pixels across the point-spread function) because data rate is limiting. In other words, it is too expensive in data volume to "oversample" the point-spread function as is done by Earth-based telescopic imaging (although they call it "adequately sampling" the point-spread function). The HiRISE image point-spread function is about 2 pixels wide at half max, which is useful but not ideal for super-resolution restoration. For #3, Mars Reconnaissance Orbiter's orbit enables crudely identical resolution and illumination conditions via its sun-synchronous orbit, although sun angles vary considerably with season. The viewing angles vary from about -30° to +30° cross-track, which is the only way to image a surface location multiple times per Mars year. The resolution varies by up to 15% with viewing angle. To co-register the images, the geometry difference with viewing angle can be overcome with sufficient processing, as done at UCL. Given the changing viewing angles plus pointing jitter, the sub-pixel offsets from image to image (#4) are random, which is far from ideal. This means that the resolution improvement varies from place to place in the super-resolution image. For #5, the surface of Mars does change over time, especially in places we favor for multiple images, to monitor the changes.

Clearly, HiRISE images are not ideal for super-resolution restoration. However, the Pancam on Opportunity rover is able to acquire near-ideal super-resolution datasets because they can sit still on the surface and move the camera pointing by fine increments, so conditions #3, 4, and 5 are well satisfied, and the SNR is high. This effort is described in a paper by Planetary Society president Jim Bell et al.; their Figure 25 is copied below. They did not provide a quantitative estimate of resolution improvement, but based on artificially degrading images by various factors (see Figure C), it appears that the improvement in resolution is about 2x. The pixel scale is 8x smaller, but that is not the same as resolution (capability to measure the size of image objects).

Figure B. Super resolution imaging with Opportunity

From Bell et al. (2006).

Figure B. Super resolution imaging with Opportunity
Figure C. Reduced-resolution HiRISE

NASA / JPL-Caltech / UA

Figure C. Reduced-resolution HiRISE
Comparing resolution of original (middle) image degraded 5x (left) and 2x (right); portion of HiRISE image ESP_016875_2265. Most individual boulders remain detectable at 2x reduction but disappear at 5x (or clusters of boulders appear to be single larger objects). Resolution reduction was produced by first binning images 2x or 5x (summing 2x2 or 5x5 pixel areas), then resampling back to the original pixel scale. The original image was acquired with 2x2 binning so most of the PSF is captured in single pixels. To the degree that the original PSF exceeds 1 pixel width, this method underestimates true 2x and 5x reduction in resolution.

Figure A, from the UCL paper, shows obvious sharpening in comparison of the super-resolution image to a single HiRISE image (labeled "original"). However, the "before" image does not fully represent the resolution potential of single HiRISE images. The "original" image (PSP_001513_1655) was acquired during moderately high dust opacity, so the SNR is poorer than other images, and the image has been map-projected at the original scale, and this resampling of the pixels reduces the resolution a bit. A better choice of "before" images is shown below, in Figure D. There is still clearly improvement in the super-resolution product, both in resolution and SNR, but comparison to Figure C suggests that the resolution improvement is closer to a factor of 2 than 5. Tau and Muller (2016) estimate super-resolution enhancement of "2–5x" based on rock counts on the "original" and super-resolution images, but the smaller rocks "not detected" in the single image are in fact detectable as dark spots, so the issue with that approach is how the automated rock-counting algorithm decides whether or not to measure a feature. A better test would be to apply super-resolution processing to a set of images over features of known size, then measure the sizes of those features in the original images and super-resolution product. The size measurement error is the resolution in each case.

Figure D. HiRISE Super Resolution Restoration

NASA / JPL-Caltech / UA / UCL

Figure D. HiRISE Super Resolution Restoration
Comparison of the best HiRISE single images (not reprojected) to the SRR products, for comparison to Figure A. The boulders are still sharpened, but not by as much as it appears from Figure A. The rover tracks appear quite similar before and after SRR, because this figure compares the best single HiRISE image of these tracks rather than an image acquired before the rover arrived here. Portions of HiRISE images ESP_025815_1655 (upper left) and PSP_006735_1650 (upper right).

How useful is super-resolution restoration for future Mars imaging?

HiRISE has captured enough (more than 4) repeat images useful for super-resolution restoration of only about a dozen areas on Mars. Areas with larger numbers of images are targeted repeatedly because of surface changes over time, so a super-resolution product would defeat the purpose of understanding these changes. For large landing sites such as for the ExoMars rover or InSight lander, it takes years to acquire even single HiRISE coverage over the whole "3-sigma" landing error ellipses, and twice as long to acquire full stereo coverage. It would not be wise to sacrifice full coverage of landing ellipses for the sake of super-resolution processing on a just a small part of the ellipse. However, the next Mars orbiter may have a much larger data rate and the landing site ellipses are shrinking due to advances in entry, descent, and landing techniques, which makes super-resolution restoration more attractive. In conclusion, for the highest-priority sites, super-resolution restoration may provide an economical alternative to flying a more capable camera and spacecraft, in order to achieve a modest improvement in resolution (probably no better than a factor of two).

Drones (tiny helicopters) are not made obsolete by super-resolution restoration—they can acquire images with much higher native resolution, and can themselves acquire better sets of images for super-resolution restoration than can a fast-moving orbiter. In fact, a drone might accompany the 2020 rover. However, this drone is very limited in range, and can only travel about half a kilometer per day before stopping to recharge, so we need orbital imaging to first find the best places to land.

See other posts from April 2016


Or read more blog entries about: explaining technology, Mars, explaining image processing, Mars Reconnaissance Orbiter


matt : 04/30/2016 01:47 CDT

It seems like the ideal communication relay would tend to be farther from the planet than the imaging portion of the mission. I almost hesitate to post such a wacky idea, but suppose for a second you took the second Hubble-sized NRO gift mirror (WFIRST gets the first, right?). At what altitude could you reasonably expect to get HiRISE resolution data? Could you get double resolution data? My understanding was that the mirrors were mounted in a rig that supported some pretty fancy 3-axis slewing and might contribute toward pointing control, though perhaps I have garbled that. Of course such a beast would need a very large imaging array, and of course all of this capability grossly exceeds the communication bandwidth from Mars. But how much does the Red Dragon capsule weigh? Falcon Heavy boosts the size of what is practical to send to Mars. And the size of flash memory, and the potential computing power available, ought to be increasing. Storing and operating on large quantities of image information could reasonably be available on the spacecraft, which can use SRR if desired and only transmit a select fraction of its imaging. If the optical communication experiment package were successful, perhaps the opportunity would exist to transmit substantially more of its imaging. Is it possible such a monster device could be used to make the rest of the instrument cheaper and less state of the art, by resisting the temptation to cram as much capability on it as possible? Is it possible for the science community to avoid that temptation? :)

matt: 04/30/2016 02:02 CDT

To close the thought loop, the idea was to raise the orbit, making the vehicle more useful as a relay. I suppose an areostationary orbit would be ideal for relay, but perhaps not useable for imaging. This monster might serve as a thought experiment for an upper bound on how far out you could orbit an imaging sensor.

Pete: 05/02/2016 03:44 CDT

I work with remotely sensed data in me day job. This article is very well-written. Kudos. Currently for COTS satellite imagery of earth, 30-40cm/px is what the newest generation of imaging satellites provide (WorldView-3). The file sizes are yuuuge, and download links would need to be extremely the laser idea. Re: Super resolution restoration. Fascinating idea, but how does one register and orthometrically-correct a set of images so accurately that this is possible? Smarter folks than me on the case apparently. Sounds very resource intensive, and impractical except on a small scale. Aerial (vs. satellite imagery is always better both in resolution and clarity/SN.. Mars drone would be amazing.

Leave a Comment:

You must be logged in to submit a comment. Log in now.
Facebook Twitter Email RSS AddThis

Blog Search

Essential Advocacy

Our Advocacy Program provides each Society member a voice in the process.

Funding is critical. The more we have, the more effective we can be, translating into more missions, more science, and more exploration.


Featured Images

Comparison of Schiaparelli and Opportunity landing locations
Mars Reconnaissance Orbiter Context Camera image of Curiosity landing site
Schiaparelli landing site, after landing attempt
Ewen Whitaker
More Images

Featured Video

The Planetary Post - Star Trek 50th Anniversary

Watch Now

Space in Images

Pretty pictures and
awe-inspiring science.

See More

Join The Planetary Society

Let’s explore the cosmos together!

Become a Member

Connect With Us

Facebook, Twitter, YouTube and more…
Continue the conversation with our online community!