|
Stephen Burch's Birding & Dragonfly Website |
|
Home | Trip Reports | Gallery | UK index | Oxon pics | UK pics | Dragonflies | Other Nature | Links |
||
|
Stephen Burch's Birding & Dragonfly Website |
|
Home | Trip Reports | Gallery | UK index | Oxon pics | UK pics | Dragonflies | Other Nature | Links |
||
Now including the R5 Mk II! Contents of this page
Introduction Given these results, I thought it would be interesting to see if I could access the raw camera sensor data, before processing (de-mosaicing) by a raw converter. This should allow me to make a direct comparisons between the different cameras. It should also provide values that could be compared directly with the measurements of signal-to-noise ratio given on the DXOMARK website, which are said to made on the raw sensor data. To make measurements on the raw sensor data, prior to raw conversion, I could not find any software that would allow me to do exactly what I wanted. So I found it was necessary to devise a somewhat involved multi-staged processing method. This is described below, as it might be of interest to others. The images from multiple cameras, including the new Canon R5 Mk II, that I processed using the method given below were generally of the "step wedge" test chart I have used for many years for measuring noise levels in digital camera images. See this page for more information. I am very grateful to Mark Chivers for allowing me to use his R3 to take some images of the test chart, so I could include this camera in the analysis given below, in addition to the cameras I have (or had) more direct access to. Sensor data before raw conversion It is important to appreciate that most camera sensors do not record full resolution colour images (i.e. 3 separate red, green and blue channels for every pixel on the sensor). Instead they use a set of RGB colour filters, arranged in front of the sensor elements, in 2x2 blocks of pixels, known as a Bayer mask or Bayer filter. Within each 2 x 2 block of sensor elements (or pixels) there are generally two green filters, one red and one blue. There are various possible arrangements for these Bayer filters as shown below. Surprisingly, in the six Canon cameras I have investigated, I have examples of all of these four types. It would seem easier for Canon to have standardised on one of them! So the sensor output is an image of single channel (monochrome) pixels, with the colour information encoded within it using the colour filters in front of every pixel.
The main function of any raw converter is to perform an interpolation using the 2 x 2 pixel colour information to generate separate R, G and B values for all 4 pixels in each block of 2 x 2, while maintaining the spatial resolution of the data as much as possible. The resultant full RGB image is then saved in standard image formats such as jpeg or tiff. This process is usually called de-mosaicing. Raw converters also perform various other manipulations of the sensor data, one of which is to apply a non-linear transformation (inverse gamma correction) to the output values, so that the grey levels are no longer directly proportional to the incident light intensity. To access the raw sensor data before conversion it is necessary to go back to the original raw file from the sensor and read out the sensor data directly before any manipulation of the data takes place. This requires a specialised set of processing stages, as described below. Processing principle and method 1. Aim of processing For each 2 x 2 block of Bayer filter values, I aimed to generate one pixel with the three channels of colour information. So the output RGB image from this process had a quarter of the number of total pixels in the original sensor data. Hence, for example, if the original sensor data was 6000 x 4000 single channel pixels, the output from this processing would be 3000 x 2000 RGB pixels. Note that this approach effectively preserves the original colour information in the Bayer Mask without any interpolation. In fact, there are 2 x 2 = 4 input values and only 3 output values, as there are two green pixels in the the Bayer Mask. The most obvious way of handling these was to average these two green pixels. Note that my analysis of the resulting "half-size" images (quarter the number of pixels) is likely to give values for signal-to-noise ratio that will be higher than those of the full size image after de-mosiacing the image, possibly by as much as 6 dB (averaging over a 2 x 2 set of pixels will reduce the noise by the square root of 4 = 2, or 6dB). However this overestimate will be exactly the same for every image and hence the method will still allow a valid comparison to be made between the values obtained from the different camera sensors. I have not been able to find any publicly available software that does the required re-arrangement of values in one step. Instead, I needed to devise a multi-stage method as described below, which also entailed me refreshing my rather rusty software development skills! 2. RawDigger I then found that with the commercially available RawDigger I could access the required raw sensor values and output them as a single channel (monochrome) image, in 16-bit tiff format. This software is a free download but only for the first 30 days, after which a reasonably modest licence fee is needed. The RawDigger export options needed to achieve the required output are given below:
By looking at the resulting tiff image in Photoshop (or similar) it was possible to inspect the individual values to determine what type of Bayer Mask was being used. To do this, you need to realise that for an image of a monochrome subject (e.g. my steps test chart), the blue filter produces the lowest grey level, while the two greens are diagonally opposite one another and of closely similar grey level. As example, the image below shows a small section of the top-left corner of an image from my Canon R5. From this, it can be seen that the Bayer mask arrangement is type 1, as shown above (i.e. red top left and blue bottom right).
By looking at these images from all the cameras, I was able to work out which type of Bayer mask was present on each one. I show my findings below:
The output from RawDigger was a single channel (monochrome) 16-bit tiff file containing the required raw sensor data that had 14-bit values for all the cameras apart from the 12-bit 350D. The maximum 14-bit number is 16383 whereas for 12-bits it is 4095. Note however that the zero value is offset substantially from zero for Canon cameras. RawDigger reads auto values for the zero level and then subtracts this from the stored sensor values. I presume any small negative values are then clipped to zero. This means the maximum possible values in the resulting files are less than the maximum possible values of 16383 (for 14-bit cameras) and 4095 (for 12-bit camera). 3. ImageJ 4. My program The output from my program is 3-channel 16-bit data written as unformatted binary numbers. 5. Import into Photoshop
By changing the defaults in this menu to those shown above (the width and height of the input data is needed here), Photoshop then reads the file in as a 16-bit 3-channel (RGB) colour image. This needed some processing of the colour information contained in this file, as described below. 6. Photoshop processing An example of the image initially open in Photoshop is shown below. Note the strong green cast.
It is isn't clear exactly what the best approach was to handling this colour cast, but I decided to handle it using the PS option Filter ---> Camera Raw Filter. Having opened the image in this option, I used the dropper tool to set the white balance anywhere in the white border of the test chart. This changed the colour balance of the image and gave an image of the monochrome test chart that looked more like it should:
To further ensure there was no significant spurious colour information in the image, I also applied the Photoshop option Image ---> Adjustments ---> Black & White using the default settings. Important option to avoid: Initially I also applied the option Image ---> Mode ---> Grayscale to reduce the 3 channel RGB file into a single monochrome channel, hence saving storage space. However this turned out to be a time consuming mistake. This seemingly simple routine was actually quite involved and took it upon itself to modify the linearity of the grey levels and even reduced the noise preferentially in any large areas of mid-grey. Very bizarre and not described in any Adobe documentation I have seen, and definitely not what I wanted! 7. Measurements using ImageJ or Isee! Linearity of sensor grey levels before raw conversion There is information on the web that suggests that the raw sensor data, before conversion, is linear with respect to incident light intensity, which is certainly not the case for the values after raw conversion. To investigate this, I took photos of my test chart (as shown above) over a wide range of exposures for each camera tested. To do this, I used the manual settings to keep the ISO and aperture constant, while varying the shutter time from the shortest possible (e.g. 1/8000) up to the value that started to give saturated values ("blown" highlights). To take these images, I had each camera mounted on a tripod so that there was no movement between exposures. I took these outside, and acquired the images with each camera as quickly as possible, one after the other, to minimise any changes the ambient light. For these images I kept the ISO constant at 1600 for all cameras. Note however that I took the exposures on the different cameras on different days and hence under different overall levels of ambient light. Analysis of the mean 14-bit grey level in an area of the white background surrounding the grey scale in the image gave values for all the different exposures taken. The area used was quite large, varying according to the pixel resolution of the sensor. For the highest resolution sensor, the area used was 400 x 400 pixels. The results are plotted below for the five different cameras I still have available for these tests. Note that the oldest camera, the 350D, was only 12-bits, so I scaled its values up to match the 14-bit values from the other cameras. The factor I used was not exactly the expected 4, due to the different zero levels for the raw sensor values, and the values at which the data saturated.
Mean 14-bit grey levels from raw sensor data as a function of exposure time The results shown above confirm that the various camera sensors gave values directly proportional to exposure for grey levels up to about 10,000. For higher grey levels they became highly non-linear and did not become fully saturated until much higher exposures are reached. Noise level measurements and results at ISO 1600 Important note: for any noise measurements, it is important to ensure all images are reasonably out of focus. This is to avoid any effects due to minor unevenness in the printing process used to produce the grey scale wedge. I also tried measuring the different steps in the grey-scale image from one well illuminated R5 exposure to check for consistency with the multi-exposure approach above. I also used this for the 50D, which I no longer have available for new images. The results are shown below. For the R5, the values found from the grey scale steps agree quite well with the multi-exposure values, although there are small differences, mainly at the lower grey levels, for unknown reasons.
I found the results shown above quite surprising. The overall differences between the cameras were less than expected. For the R5, although it had the lowest overall noise levels, it was very close indeed to the 7D Mk II, especially at higher grey levels. The other four cameras, including the R7, showed somewhat higher noise levels, but the differences were quite small. Signal-to-noise ratio (SNR) at 18% grey level results at ISO 1600 Derivation of 18% SNR values In order to try to reproduce the DXOMARK measurements as closely as possible, I tried using a measure of a grey level corresponding to 18% of the saturated value for each sensor. For the 14-bit sensors, this varied slightly between cameras but was around 14340, so 18% of this was 2581. I do not know if this is the method used by DXOMARK to define their 18% SNR values, as their website does not specify this detail. To find the noise levels at 18% grey level, I fitted power law curves to the above measurements, which were straight lines on the above log-log plot. The fitted lines had gradients of close to the expected 0.5, and I then found the noise level of the fitted lines for the 18% grey level values. This was easier than performing some form of interpolation between adjacent measurements and should have reduced the "noise" from the minor variations about the fitted lines shown on the above plot. To derive the SNR, I simply took the 18% grey level value and divided by the corresponding noise level (standard deviations of the values in small regions). Comparison between 18% grey SNR and max noise levels after ACR On another page, I show measurements of the maximum noise levels in images after raw conversion using Adobe Camera Raw (ACR) (and Digital Photo Professional). It is interesting to compare these values with those found as described above for the SNR at 18% grey. The scatter plot below shows the ACR maximum noise levels for eight Canon cameras against the values for SNR at 18% grey. I find it surprising that the correlation between these two measurements of noise, before and after raw conversion, shows so much scatter. For example, before raw conversion, the SNR for the R5 is only 0.5 dB higher than for the 7D Mk II. But after ACR raw conversion, the maximum noise level for the R5 is appreciably lower than the 7D Mk II by 2.4 dB. These measurements also show that the R5 Mk II raw sensor data has a very similar 18% grey signal-to-noise ratio to that of the original R5, but that after raw conversion using ACR, the R5 Mk II image is somewhat more noisy than the original R5. It seems that ACR is applying some camera specific noise reduction processing that affects some cameras more than others. How this works, I have no idea!
My measurements of 18% SNR from the sensor data before raw conversion, compared with my measurements of maximum noise level after raw conversion using ACR. All measurements at ISO 1600. Note that the R5 Mk II has a similar SNR to the R5 but using ACR the R5 Mk II image is noisier than the R5. DXOMARK is an organisation that performs independent assessments/tests on photographic equipment. Importantly, DXOMARK state that their measurements are made from the raw sensor data, i.e. before any de-mosaicing and other processing performed by a raw converter. Of the various parameters DXOMARK give for camera sensors, the one with most relevance here is their measurements of what they call 18% SNR. This I take to mean the signal-to-noise ratio (SNR), as measured at a grey level of 18% brightness, which typically corresponds to mid-grey. A description of their measurement procedure is given on their website, but some of details are not fully defined nor explained as far as I can tell. For example, it is not clear exactly how the grey level corresponding to 18% is determined for an image. In addition, it is not completely clear how their measurements of the individual noise levels in the red, green and blue channels are combined to provide a single value. I'd also be interested to know how their green channel is defined as typical Bayer masks on camera sensors have two green pixels. Are these averaged in some way to provide the green channel information? It does however appear that the DXOMARK measurements are made very rigorously using specialised transmission test charts and analysis methods, so they should provide objective and accurate measurements of the parameters measured. But I am not aware of any other sources of this type of information, so there is nothing to cross check these measurements against. In my most recent update of this page (8 October 2024) I find it highly notable that DXOMARK now show two different sets of 18% SNR values for each camera sensor, whereas previously I was only aware of one set. It seems that the previous values were what DXOMARK now refer to as their "Print" values and a whole new set of values referred to as "Screen" is now given. I reproduce below the small text from their website that explains the meanings of these two terms below. Note that the explanation for the "print" values includes normalisation to an 8 Mb image, irrespective of the camera sensor resolution and various arbitrary values, completed unrelated to the camera sensor such 300dpi and 8"x12" format. On the other hand, the DXOMARK definition of their new "screen" values looks like this is much more in line with what I am interested in measuring: the noise on the raw sensor values prior to de-mosaicing. I find these new DXOMARK definitions and measurements quite surprising but they provide an explanation for previous discrepancies I was finding between my measurements and their values. Prior to this "clarification", the clear inference was that their 18% SNR values referred to the raw sensor data, and not a value related to some arbitrary sized image (8Mb), printed at a particular resolution (300dpi) on an 8" x 12" sized print! These new "screen" values are in better agreement with my measurements, than the previous "print" ones, albeit with a need to increase the DXOMARK "screen" values by a constant (see below for more details).
Comparison with DXOMARK 18% SNR values The results for the 18% SNR values, derived from my measurements at ISO 1600 as above, are shown below for direct comparison with the corresponding DXOMARK "screen" values, where available, for ISO 1600. All the SNR values are shown as dB, which is as DXOMARK shows them. Note that all the DXOMARK values were substantially lower than the values I have derived, for reasons no doubt related to the details of exactly how the measurements were made. However by finding the average difference for all cameras (+4.7 dB), and adding this to the DXOMARK values, I then get very good relative agreement with the values I derived as described above.
Having adjusted all the DXOMARK values by +4.7dB, the chart above shows very good agreement between my measurements and the "screen" DXOMARK values, over all the cameras assessed. The maximum difference between the two sets of measurements is only 0.6 dB - for the 7D Mk II. As expected, the highest SNR was obtained with the Canon R3, which had the largest sensor element (pixel) size of all those shown above, with the exception of the very old Canon 350D. In the plot above, I also show SNR values for the new R5 Mk II when using the mechanical (mech) and electronic (el) shutter. These are both very similar to original R5 value (for the mechanical shutter only). Hence the electronic shutter does not appear to increase the image noise level. After the R5 and R5 Mk II, the old 7D Mk II had an SNR only about 1 dB less. Rather surprisingly the next highest SNR was from the 50D (about 1.5 dB less than for the R5), with the lowest SNR from the oldest camera - the 350D, which was about 3 dB (just 1 stop) down on the R5. It is also notable that the R7 gave a relatively low SNR value, which is no doubt due to its small pixel size (3.2 micron) - the smallest of the cameras shown above. High ISO Noise of the R5 Mk II compared with the R5 It seems that there has been much discussion online about the high ISO noise performance of the R5 Mk II, with claims that the R5 Mk II is significantly worse than the original R5 at ISO values around 12,800 and 25,600. However these suggestions appear mainly to be based on subjective assessments of various different images obtained at different times with the two cameras. The chart above, based on quantitative measurements of a standarised test chart. shows no evidence for poorer SNR in the R5 Mk II compared with the R5, but is based on measurements at the relatively low ISO of 1600. Hence I have made quantitative measurements of signal-to-noise ratio (SNR) for higher ISO values up to 25,600 to try to find this reported large reduction in the R5 Mk II's performance. My high ISO measurements are shown in the chart below which gives two sets of SNR values. The first is for the SNR of the images for mid-grey (18%). I also show values for 5% SNR which correspond to the dark grey/black areas in the images.
The chart above shows no evidence for a major reduction in noise performance for the R5 Mk II at high ISO. There is however a small effect for the electronic shutter images from the R5 Mk II which are slightly noisier than those for the R5. This effect is larger in the darker areas of the image but even then the 5% SNR value for the R5 Mk II is only about 0.8 db (or a quarter of one stop) lower than for the R5. Hence these quantitative measurements of the raw sensor data from the R5 Mk II and the R5 show very similar noise levels all the way up to ISO 25,600. So the reason for the reports of noisier high ISO images from the R5 Mk II are unclear. It is however possible that the raw converters (e.g. ACR, DPP etc) which are inevitably needed to view the images are in some way responsible for these reported differences between the R5 Mk II and the R5.
|
|||||||||||||||||||||||||
Home | Trip Reports | Gallery | UK index | Oxon pics | UK pics | Dragonflies | Other Nature | Links |
