Image Stacking — Sky & Telescope, April 2022 — Errata

The following errata correct the April 2022 Sky & Telescope Article, Image Stacking Demystified, by Richard S. Wright, Jr.

  1. The cumulative image noise is proportional to the square root of the number of frames being stacked.
  2. Mathematically, the shot noise from the imager and associated electronics can be considered to be added to the signal.
  3. The horizontal axis in both graphs on p.56 should be labeled “number of stacked frames,” and the vertical axis should be labeled “quantity.”

EXPLANATION

1.  The S&T author writes, “Shot noise is also quantifiable — it’s simply the square root of the signal value.” This is incorrect, as are the numerical examples that follow the statement.1 The noise and signal components are separate entities, and one cannot say that one of them is a function (square root) of the other.  (Actually, some systems do have an interaction between the two, but those are usually 2nd-order, minor effects, and are not relevant in the image stacking situation.)  The noise value is completely determined by the physics of the imaging device and the transistors in the related electronics, and is independent of the signal – it’s even there when there is no signal (e.g., a dark frame).  The incorrect statement would imply that a dark frame has zero noise, which is not true: in addition to fixed pattern noise (which we can reduce by subtracting a dark frame when doing advanced image processing), the dark frame will have its own random noise, too.

What is really happening is the following.  When we stack multiple images, we are literally adding the images together, pixel-by-pixel.  That means that the signal components get added together, and so do the noise components.  When the images are properly aligned (registered), the signal components at each pixel from one frame to the next add together as correlated data, since they are part of the same image. This combination is literally a simple addition, so image stacking increases the signal component in proportion to the number of frames being stacked.

However, the noise component at each pixel from frame to frame is uncorrelated, because it is a random process.  The noise components add together as orthogonal vectors, which means that the noise value increases by the square root of the number of frames being added together.  (The stacked images are then re-scaled, so that the resulting image doesn’t get progressively brighter everywhere – but this, of course, scales the noise by the same amount.)  The signal-to-noise ratio improvement is therefore proportional to the square-root of the number of frames that are stacked.

2.  The S&T author writes, “It’s important to bear in mind that this noise is not something that gets added so much as something that’s missing.” This is incorrect.  The mathematical modelling and analysis of signals with noise accounts for each of these elements as an added component; there is nothing “missing” from the original signal, which still exists in the image capture.  In practice, this can be readily seen by using a spectrum analyzer, which will show that the signal and noise are separate components.  The shot noise from the imager and associated electronics should be considered to be added to the signal.

3.  The graphs on p.56 are labeled incorrectly. The horizontal axis in both graphs should be labeled “number of stacked frames,” and the vertical axis should be labeled “quantity,” as it represents either signal or noise in the left-hand graph, and signal-to-noise ratio in the right-hand graph.

—agc


FOOTNOTE (1):

There is a different quantity, known as photon noise, which is characterized as the square root of the photon signal, but this is not the dominant factor in our calculation of signal-to-noise ratio, because we are considering the net effect over a set of stacked frames.


REFERENCES:

Wikipedia: Shot Noise – note the discussion regarding “square root of the expected number of events.”

Wikipedia: Gaussian Noise – “values at any pair of times are identically distributed and statistically independent (and hence uncorrelated).”

Philippe Cattin, Image Restoration: Introduction to Signal and Image Processing.

Robert Fisher, et al, Image Synthesis — Noise Generation.

 

 

Cugnini to Present Talk on New Audio/Video Technologies

AGC Systems’ President Aldo Cugnini will deliver an online talk, entitled “New Audio/Video/Wireless Technologies For Home Entertainment.”  Scheduled for Thursday, November 18, 2021, 6:30PM EST, and hosted by the IEEE Consultants’ Network of Northern New Jersey, the talk will explain how new technologies like UHDTV, HDR, and HEVC enable audio and video devices to efficiently deliver the latest entertainment to consumers.

The talk is free, and can be accessed by registering here.


This presentation is partly sponsored by Elecard.  Click here for more information on their video and stream analysis tools.

Video Pioneers Remember Historic HDTV Debut

Grand Alliance Prototype

Twenty-five years ago this week, the world’s first HDTV broadcast system was unveiled in Las Vegas at the 1995 NAB Show.  AGC Systems’ Aldo Cugnini was there, as one of the many engineers who developed the “Grand Alliance” digital HDTV system.  Then at Philips, Aldo had a leadership role in the system’s development, which went on to become the ATSC digital television system.

Click here to see historic videos of the debut of HDTV.

Transit of Mercury – Live!

There will be a rare astronomical event this Monday, November 11, 2019, when Mercury passes in front of the sun. We’ll be streaming it live from NJ using a telescope. The event occurs only about 13 times a century.

Click on the player to see the live stream during the event!  We’ll also stream the end, at 1:00pm EST, and on the quarter-hour in between, as well.  Weather permitting!

transit of Mercury
Transit at 11:47am EST

Technical information

  • Questar 3.5 Cassegrain-Maksutov telescope with chromium solar filter
  • Lumix GH2 camera, 1080p24 source video
  • KanexPro HDMI/3G-SDI converter
  • Haivision Makito X video encoder /streamer, down-converted to 720p
  • HLS playback on HTML5 with Flash fallback for older browsers

Special thanks to …

  • John Turner, Turner Engineering − encoder, technical support
  • Wowza Media Systems − Wowza Streaming Cloud
  • Cathy McLaughlin − location support
  • Charlotte Cugnini – production assistance
  • Lizzie Cugnini – production assistance

Selected Papers

  • The Promise of Mobile DTV, NAB Broadcast Engineering Conference Proceedings, 2011.
  • A Revenue Model for Mobile DTV Service, NAB Broadcast Engineering Conference Proceedings, 2010.
  • Considerations for Digital Program Insertion of Multiple-Video Programs, NCTA Fall Technical Forum, 2002.
  • Digital Video and the National Information Infrastructure, Philips Journal of Research, Volume 50, Issues 1–2, 1996.
  • MPEG-2 video decoder for the digital HDTV Grand Alliance system, IEEE Transactions on Consumer Electronics, Vol. 41, Aug 1995.
  • Grand Alliance MPEG-2-based video decoder with parallel processing architecture, Intl. J. of Imaging Systems and Technology Vol. 5, No. 4, 1994.
  • The ISO/MPEG Audio Coding Standard, Widescreen Review, June/July, 1994.

NAB 2019 Conference Widely Featuring ATSC 3.0

More than 100 NAB Show sessions and more than 50 exhibitors will feature Next Gen TV technology that is now voluntarily spreading to cities throughout the country. Powered by the ATSC 3.0 next-generation broadcast standard, Next Gen TV promises to deliver sharper, more detailed pictures and lifelike multichannel audio with upgraded broadcasts that will be transmitted and received in the same Internet Protocol language as Internet-delivered content.

Jointly sponsored by the Advanced Television Systems Committee, the Consumer Technology Association and NAB, the “Ride the Road to ATSC 3.0” exhibit will be featuring a series of free presentations about all facets of the ATSC 3.0 standard. And attendees can pick up a free Guide to 3.0 at the Show in the Central Lobby of the Las Vegas Convention Center during the show.

Single Frequency Network Demonstrations

The NAB, with support from a number of technology companies, will demonstrate the Single Frequency Network (SFN) capabilities of the Next-Gen TV standard, showing how reception can be improved in difficult locations and in moving vehicles by deploying multiple broadcast towers transmitting the broadcast signal on the same channel.

Using several local transmissions, special SFN viewing kiosks will showcase the flexibility of the ATSC 3.0 standard. Dozens of sessions planned in the exhibit will include updates on the Dallas, Phoenix, Santa Barbara, East Lansing, Cleveland, and Korea ATSC 3.0 deployments.

Scores of papers and sessions will be presented about Next-Gen TV during the 2019 NAB Show, with session topics that will cover consumer research, consumer device plans, conformance testing, audio enhancements, station build-out advice, watermarking, advanced emergency information, channel security, advanced advertising and interactivity. In addition to ATSC, CTA and NAB, exhibit sponsors include Pearl TV, Sinclair Broadcast Group, LG Electronics, Dolby, Sony, Samsung and the AWARN Alliance. The centerpiece of the Ride the Road stage is a giant new LED videowall optimized for broadcast applications, provided by LG Business Solutions.

AGC Systems president Aldo Cugnini will be at the show, and available for discussions regarding support for ATSC and other related ventures. If you’d like to meet up, please contact us.

FCC Opens Up Spectrum Above 95 GHz

This month, the Federal Communications Commission allowed a plan to make the spectrum above 95 GHz more readily accessible for new innovative services and technologies. Calling the initiative “Spectrum Horizons Experimental Radio Licenses,” the plan is outlined in a First Report and Order, which allows a number of changes to existing rules, including:

  • a new category of experimental licenses, to increase opportunities for entities to develop new services and technologies from 95 GHz to 3 THz, with no limits on geography or technology; and
  • making 15.2 gigahertz of spectrum available for unlicensed use.

The Order specifically allows two types of operations:

  • A Spectrum Horizons experimental radio license can be issued for the purpose of testing and marketing devices on frequencies above 95 GHz, where there are no existing service rules.  Licenses are issued for a term of 10 years and may not be renewed.
  • Unlicensed operations are allowed in the bands 116-123 GHz, 174.8-182 GHz, 185-190 GHz, and 244-246 GHz, that are consistent with the rules proposed in the Spectrum Horizons, Notice of Proposed Rulemaking and Order.

Part 15 of the FCC Rules was also amended to extend operational limitations and interference measurements covering frequencies above 95 GHz.

The new rules provide that the Commission may, at any time without notice or hearing, modify or cancel a Spectrum Horizons License, if, in its discretion, the need for such action arises.  Some commenters raised the issue that this could result in an abuse of the complaint process, but the Commission pushed back, saying they “routinely work with parties to resolve potential or actual issues…”

The Commission withheld action on their proposal for licensed fixed point-to-point operations in a total of 102.2 gigahertz of spectrum, and opposed the concerns of the ham-radio organization ARRL regarding protection from interference.  In defending the latter position, the Commission states, “both the amateur radio service and the experimental licensing program are designed to contribute to the advancement of radio knowledge,” and goes on to say that “we will instead require all Spectrum Horizons License applicants to submit an interference analysis that would address the potential effects of the experimental operation on existing services.”  

In addition to Chairman Ajit Pai, the proposal has general support — albeit with certain cautions — from all four of the other commissioners, who evenly represent both sides of the political aisle. 

— agc

HEVC and VP9 Comparison for IP Video: Which Will Win the Day?

Internet video has become a practical medium for the delivery of video content to consumers. What has made this possible is the development of video compression, which lowers the enormous amount of bandwidth required to transport video to levels practical with most Internet connections. In this article, we’ll examine some of the technical and business issues associated with two video codec frontrunners: HEVC and VP9.

HEVC (High Efficiency Video Coding, also called MPEG-H Part 2 and ITU-T H.265) is a state-of-the-art video compression standard that provides about a 50 percent bit rate savings over H.264/MPEG-4 AVC, which in turn provided a similar efficiency over its MPEG-2 predecessor.

AVC solutions have already become widespread in many professional and consumer devices. HEVC, having been ratified by ISO and ITU in 2013, is similarly growing in the same applications, and would appear to be on the road to replacing the earlier codecs. But while MPEG and HEVC have been developed by standards committees representing a legion of strong industrial players, other forces have sought to displace their primacy, most notably, Google, with its VP9. First, let’s look at the toolkit of each codec.

HEVC incorporates numerous improvements over AVC, including a new prediction block structure and updates that include intra-prediction, inverse transforms, motion compensation, loop filtering and entropy coding. HEVC uses a new concept called coding units (CUs), which sub-partition a picture into arbitrary rectangular regions. The CU replaces the macroblock structure of previous video coding standards, which had been used to break pictures down into areas that could be coded using transform functions. CUs can contain one or more transform units (TUs, the basic unit for transform and quantization), but can also add prediction units (PUs, the elementary unit for intra- and inter-prediction).

HEVC divides video frames into a hierarchical quadtree coding structure that uses coding units, prediction units and transform units. CUs, TUs and PUs are grouped in a treelike structure, with the individual branches having different depths for different portions of a picture, all of which form a generic quadtree segmentation structure of large coding units.

While AVC improved on MPEG-2 by allowing multiple block sizes for transform coding and motion compensation, HEVC coding tree blocks can be either 64×64, 32×32, 16×16 or 8×8 pixel regions, and the coding units can now be hierarchically subdivided, all the way down to 4×4 sized units. The use of tree blocks allows parallel processors to decode and predict using data from multiple partitions—called wavefront parallel processing (WPP), which supports multi-threaded decode.

Because this new coding structure avoids the repetitive blocks of AVC, HEVC is better at reducing blocking artifacts, while at the same time providing a more efficient coding of picture details. HEVC also specifies several planar and DC modes, which reconstruct smooth regions or directional structures in a way that hides artifacts better. An internal bit depth increase allows encoding of video pictures by processing them with a color depth higher than 8 bits.

Motion compensation is provided with two new methods, and luma and chroma motion vectors are calculated to quarter- and eighth-pixel accuracy, respectively. A new deblocking filter is also provided, which operates only on edges that are on the block grid. After the deblocking filter, HEVC provides two new optional filters, designed to minimize coding artifacts.

VP9 Web Video Format Improves on VP8

With YouTube carrying so much video content, it stands to reason that the service’s parent, Google, has a vested interest in not just the technology behind video compression, but also in some of the market considerations attached therein. To that end, VP9 has been developed to provide a royalty-free alternative to HEVC.

Many of the tools used in VP9 (and its predecessor, VP8) are similar to those used in HEVC—but ostensibly avoid the intellectual property used in the latter. VP9 supports the image format used for many web videos: 4:2:0 color sampling, 8 bits-per-channel color depth, progressive scan, and image dimensions up to 16,383×16,383 pixels; it can go well past these specs, however, supporting 4:4:4 chroma and up to 12 bits per sample.

VP9 supports superblocks that can be recursively partitioned into rectangular blocks. The Chromium, Chrome, Firefox and Opera browsers now all support playing VP9 video in the HTML5 video tag. Both VP8 and VP9 video are usually encapsulated in a format called WebM, a Matroska-based container also supported by Google, which can carry Vorbis or Opus audio.

VP8 uses a 4×4 block-based discrete cosine transform (DCT) for all luma and chroma residual pixels. The DC coefficients from 16×16 macroblocks can then undergo a 4×4 Walsh-Hadamard transform. Three reference frames are used for inter-prediction, limiting the buffer size requirement to three frame buffers, while storing a “golden reference frame” from an arbitrary point in the past.

VP9 augments these tools by adding 32×32 and 64×64 superblocks, which can be recursively partitioned into rectangular blocks, with enhanced intra and inter modes, allowing for more efficient coding of arbitrary block patterns within a macroblock. VP9 introduces the larger 8×8 and 16×16 DCTs, as well as the asymmetric DST (discrete sine transform), both of which provide more coding options.

Like HEVC, VP9 supports sub-pixel interpolation and adaptive in-loop deblocking filtering, where the type of filtering can be adjusted depending on other coding parameters, as well as data partitioning to allow parallel processing.

Comparing HEVC and VP9

As you would expect, performance depends on who you ask. Google says VP9 delivers a 50 percent gain in compression levels over VP8 and H.264 standards while maintaining the same video quality. HEVC supporters make the same claim, which would put VP9 close to HEVC in quality. But some academic studies show that HEVC can provide a bit rate savings of over 43 percent compared to VP9. Why the disparity? One likely reason is that using different tools within each codec can yield widely varying results, depending on the video material. The other is that, despite some labs having developed objective tools to rate image quality, the best metric is still the human visual system, which means that double-blind subjective testing must be done, and that will always have statistical anomalies.

But another important factor must be considered as well, and that’s complexity. While both HEVC and VP9 demand more computational power at the decoder, the required encoding horsepower has been shown to be higher (sometimes more than 10 times) for HEVC in the experiments where it outperformed VP9 on bit rate.

There’s a strong motivation for advancing an alternative to HEVC: VP9 is a free codec, unencumbered by license fees. Licenses for HEVC and AVC are administered by MPEG LA, a private firm that oversees “essential patents” owned by numerous companies participating in a patent pool.

Earlier this year, MPEG LA announced that a group of 25 companies agreed on HEVC license terms; an AVC Patent Portfolio License already provides coverage for devices that decode and encode AVC video, AVC video sold to end users for a fee on a title or subscription basis, and free television video services. Earlier, MPEG LA announced that its AVC Patent Portfolio License will not charge royalties for Internet video that is free to end users (known as “Internet Broadcast AVC Video”) during the entire life of the license; presumably, this means the life of the patents.

Last year, Google and MPEG LA announced that they had entered into agreements granting Google a license to techniques that may be essential to VP8 and earlier-generation VPx video compression technologies under patents owned by 11 patent holders. The agreements also grant Google the right to sublicense those techniques to any user of VP8, whether the VP8 implementation is by Google or another entity. It further provides for sublicensing those VP8 techniques in one next-generation VPx video codec.

So, while there is no license fee required to use VP8, there are other terms imposed—a so-called FRAND-zero license—and users may need a license to fully benefit from the Google-MPEG-LA agreement. One result of the agreements is that MPEG LA decided to discontinue its effort to form a VP8 patent pool.

Apparently, VP9 is a further attempt to provide a shield against the MPEG patent owners, by using elements thought to evade existing granted patents. But HEVC has already made inroads into commercial hardware and software, following on the heels of the already widespread MPEG-4/AVC rollout, and this could make an uptake of VP9 difficult. And even the best intents of the VP8/VP9 developers can be subverted: it’s always possible that a “submarine patent” could emerge, with its owner claiming infringement.

This has already happened, with smartphone maker Nokia suing HTC over its use of the VP8 codec. In this particular case, a court in Mannheim, Germany, ruled that VP8 does not infringe Nokia’s patent—but the possibility always exists of another challenge. While the specter of another contest could be enough to give some manufacturers pause, tilting support toward the “safer” HEVC, it could just as well be subject to some other submarine patent.

A final note: Google has announced that development on VP10 has begun, and that after the release of VP10, they plan to have an 18-month gap between releases of video standards.


This article was originally published in Video Edge Magazine.