rawopint v0.21 - Fast, accurate intervalometer with raw exposure metering
This script requires CHDK 1.4.1 or later
The basic idea is to shoot rapidly without invoking Canon firmware auto-focus or auto-exposure between shots. The exposure for each subsequent shot is calculated from the previous image data. Shooting is done either using Canon continuous mode with delays inserted to maintain the desired interval, or by simulating holding the shutter button at half press and repeatedly clicking full press. In continuous mode, the script can achieve a frame rate close to the Canon continuous shooting specification.
This script intended for situations where a relatively high frame rate, smooth exposure change and fixed focus are desired. Because exposure is calculated from the raw image data, it performs well in low light but may not perform well in situations where the light changes rapidly. It is well suited for daytime landscapes, clouds, sunsets, sunrises etc.
This is still under development, and is not considered entirely finished. In particular:
- Doesn't handle aperture adjustments yet. This means the initial aperture value set by the canon firmware or CHDK overrides will be used for every shot.
- ND filter control is only available on cameras without an adjustable aperture, and the ND value may be inaccurate if the canon firmware did not set the ND filter in for the first exposure.
- Over exposure control doesn't deal well with really strict limits.
- There are too many options, and their meanings are not clear.
- There are not enough options to adjust all the things a user might want to adjust.
Despite the above, it can still produce good results in many situations.
- Fixed a bug that caused exposure to stop increasing when Max ISO was hit, regardless of Max Tv. Thanks udo for the report.
- Added USB remote interval control option
- Added interval warning LED and beep options, suggested by c_joerg in http://chdk.setepontos.com/index.php?topic=11081.msg125794#msg125794
- Changed some defaults. Meter now defaults to 90%, meter step 15.
- Log mode now has options for "none", "replace" and "append", defaults to append.
- Fix delay on script exit reported by c_joerg in http://chdk.setepontos.com/index.php?topic=11081.msg125651#msg125651
Discussion / Bug reporting thread: http://chdk.setepontos.com/index.php?topic=12697.0
Download version 0.21: https://app.box.com/s/60na934xplrnwvhr83fed6mlfqdtq5gd
Earlier discussion can be found in the raw hook development thread: http://chdk.setepontos.com/index.php?topic=11081.msg119265#msg119265
Many of the concepts in the script and CHDK shooting hooks were inspired by Lapser's work in: http://chdk.setepontos.com/index.php?topic=8997.0
This script has a lot of options which affect the final behavior. While the defaults can produce reasonable results in many situations, some experimentation is recommended to get a feel for the impact of the various options. The log file provides a wealth of information to understand what happened in a given run. Some general descriptions and usage guidelines are provided below.
Terms like Ev and Bv below are used loosely to mean APEX-like increments of exposure and scene brightness, but may not correspond exactly to their APEX definitions.
Under optimal conditions, the shortest achievable interval should be slightly more than the Canon continuous mode specification.
If the interval is long enough to accommodate all processing, the interval between shots will be maintained with high accuracy, generally well under 1/10th of a second.
Interval should be long enough that the camera can keep up, including time for metering and the actual exposure. For nighttime, sunrises or sunsets, exposure time can be significant. Canon firmware dark frame subtraction and noise reduction may also substantially reduce the achievable interval. These are described in the Exposure Limits section below.
SD card firmware garbage collection, wear leveling etc. may cause shots to occasionally take much longer than normal. Unfortunately, this behavior is not well specified, and does not necessarily correspond to card cost or speed rating.
Metering and exposure controlEdit
Metering is done using an adjustable sized rectangle in the middle of the sensor, referred to below as the "meter". Additionally, a histogram of the entire sensor area is analyzed to limit the fraction of over and under exposed pixels to user specified ranges. The relative influence of these limits may be adjusted, and exposure changes are smoothed to avoid flickering. The target exposure defaults to a "neutral" value, roughly equivalent to what the Canon firmware auto-exposure would use for an evenly lit subject.
To meter the whole scene, use large meter height and with, e.g. 90%. Use smaller sizes for a "spot" meter. Note that the meter value is the average value of the metered area, so a scene that is half light and half shadow will be treated the same as a even value half way between the two.
With the default settings, the influence of the meter, over and under exposure balance in way that tends to keep the whole scene reasonably well exposed, at the expense of allowing some under or over exposure. For some scenes, you may want to make one dominant over the others. For example, to keep the moon from getting over-exposed at night, the under exposure limits should be turned off. On the other hand, if the sun will pass through the scene, you may want an over exposure fraction heigh enough to let it be blown out, without driving the rest of the scene completely dark.
Bv /Ev ShiftEdit
By default, the script tries to keep the meter area at a fixed exposure value. This means the subject will stay the same brightness as lighting changes, which looks unnatural in many situations. For example, if a cloud passes in front of the sun, the shadows get lighter but the overall scene brightness changes little. Similarly, a sunset will only change colors without getting any darker, until the exposure limits are hit, at which point the scene rapidly darkens. The Bv / Ev Shift setting can be used to avoid this. This setting changes the target exposure (equivalent to an Ev shift in the Canon firmware P mode etc) in response to the absolute brightness of the scene (Bv). A value of 30% means that for every stop of actual brightness, the exposure will change by ~1/3 of a stop. This shift is limited by the over and under exposure limits and the meter limits.
The Bv/Ev shift base Ev controls what absolute brightness is unshifted (i.e., what real light level will correspond to the "neutral".) This can either "first" to use the initial scene, or a fixed Bv value. For fixed values, a scene in mid-day sun is around 10, while indoor artificial lighting might be around 1. For a sunset, you could use "first", causing the scene to go from normal exposure to darkness. For a sunrise, using a daylight value like 10 would give better results.
Very high shutters speeds have been observed to cause unstable exposure. Setting Min Tv Sec/100K to less than or equal to the cameras factory shutter limit is suggested. High shutter speeds may also impact shutter life.
Long exposures may trigger Canon dark frame, which doubles shooting time. This can be overridden in CHDK on most cameras, but quality may suffer.
The Target ISO should normally be set to the lowest value supported by the Canon firmware. It should not be set lower.
High ISOs usually trigger noise reduction (separate from dark frame) which can substantially increase processing time and affect the ability of the script to maintain the desired interval. The value where this occurs is camera dependent, but is often at 400, 800 or 1600. Set the Max ISO to a value slightly below the value that triggers the "busy" screen to maintain shorter intervals.
On cameras which have an ND filter and no adjustable aperture, the ND will be put in or out based on the ND TV Sec/10000 setting. This is recommended to keep shutter speeds from getting too short, but if the Canon firmware didn't put the ND in for the initial exposure, the script does not know the exact value, and there may be a small jump in exposure when the ND state changes.
The focus distance of the initial shot is used for all shots in a run. The initial focus is determined by normal Canon and CHDK settings, and is not currently changed by the script. Using Canon firmware MF, AF Lock, "focus at infinity" mode, or CHDK SD overrides is recommended.
Camera specific issuesEdit
- D10 has been observed shoot with the shutter closed with some combinations of settings, particularly if the debug OSD is off, and either raw hook sleep is zero, or disable script yield is not set. It's unclear whether this represents a hardware failure, or is a side effect of how the script interacts with the canon firmware. This has only been observed on one camera, and has not been seen other models. See http://chdk.setepontos.com/index.php?topic=12258.0
- G1x has problems with ISO over 320. See http://chdk.setepontos.com/index.php?topic=12165.30
- SX230 has problems with raw in continuous mode.
The script has a lot of options which interact in non-obvious ways. The log file can provide insight into why the script behaved in a particular way. The _weight columns show the relative contribution of meter, under exposure protection and over exposure protection to the final exposure value.
- Moonrises, with settings posted
- Moonrise, sunset https://www.youtube.com/watch?v=XguvoBYP2vE
- Sunset, clouds, moonset https://www.youtube.com/watch?v=OIPLE-G8QKw
Note the settings described in posts linked above may not be for the most recent version of the script. Saved script settings files posted in the forum are generally only valid for the specific version they were saved with.
- Markb https://chdk.setepontos.com/index.php?topic=12766.0 (includes extensive discussion of settings)
- c_joerg https://chdk.setepontos.com/index.php?topic=12761.0
Number of shots. If set to 0, script will end on menu key press or low battery/low remaining space only.
Time between shots, in 10ths of a second.
USB remote interval controlEdit
Uses the CHDK USB Remote to trigger shooting. In this mode, the script will wait up to the interval value for a pulse (low-hi-low) on the USB power input. If a pulse isn't received in time, it will shoot anyway. If the pulse is received during the shooting process, the next shot will trigger as soon as the possible. The CHDK USB remote enable option will be set by the script if this option is enabled, and restored to its previous state on script exit. If the script is ended with the menu key, one additional shot may be taken.
Meter width %Edit
Width of meter area, in percent of active area.
Meter height %Edit
Height of meter area, in percent of active area.
Every step-th pixel is measured. To sample all colors, it should be odd. To prevent overflow, the total number of pixels (size/step)^2 must be less than unsigned_max / white_level. (~1M for 12 bpp, 250K for 14 bpp). Larger numbers of pixels will be slower.
Max Ev changeEdit
Maximum exposure change in a single step, in fractions of a stop. The actual exposure change is smoothed, so the max value will be only used if there is a sufficiently large, consistent trend in scene brightness.
Use initial Ev as targetEdit
Use the exposure of the initial shot as the target Ev instead of the pre-defined neutral value.
Shift the target exposure by specified number of stops. If "Use initial Ev as target" is enabled, the shift is applied to the initial value. Otherwise, it is relative to the neutral value.
Bv Ev shift %Edit
Vary target Ev based on absolute scene brightness, so every stop of scene brightness from the base Bv corresponds to x % change in target Ev. The range is limited by Meter thresh and limit options described below. Use the option to make the exposure follow scene brightness, so for example, a sunset will get darker instead of maintaining constant exposure.
Bv Ev shift base EvEdit
If Bv Ev shift is in effect, this sets the Bv at which the target Ev will equal the initial target Ev. If set to "First", the Bv of the first shot is used. A light surface in mid day sun is around 10.
Max Tv Sec/1000Edit
Longest shutter speed to use, in 1/1000ths of a sec. If required exposure is longer, ISO will be used up to ISO limits.
Min Tv Sec/100KEdit
Shortest shutter speed to use, in 1/100000ths of a sec. If required exposure is shorter, it will be limited to this value.
ISO value to use if shutter limits (described below) not reached. Should normally be lowest ISO. In Canon UI units.
ISO adj TV Sec/1000Edit
Shutter speed (in 1/1000ths of a sec) at which ISO will start to be increased. If the exposure time exceeds this value, exposure will changes are split 50/50 between shutter and ISO, subject to limits below
Absolute max ISO to be used. In Canon UI units.
ND Tv Sec/10000Edit
Shutter speed (in 1/10000ths of a sec) below which ND filter is put in. Ignored on cameras with an adjustable aperture, even if they also have an ND filter. 0 disables.
Meter high thresh EvEdit
Meter value in to start limiting under exposure protection, in stops above neutral. As the meter value exceeds this value, the influence of under exposure protection and positive Bv / Ev shift is reduced.
Meter high limit EvEdit
Maximum meter value allowed from under exposure protection, in stops above neutral.
Meter high max weightEdit
Controls the maximum influence of the meter high limit relative to histogram under exposure. The influence of the meter increases proportional to the square of the fraction by which it exceeds the threshold, such that the max weight value is be reached at the limit Ev. If set to 100 (the standard weight before threshold), this behavior is disabled.
Meter low thresh EvEdit
Meter value to start limiting over-exposure protection, in stops below neutral. As the meter value falls the number of stops specified below neutral, the amount that over exposure protection or Bv / Ev shift can reduce exposure is reduced.
Meter low limit EvEdit
Minimum meter value allowed from over exposure protection, in stops below neutral.
Meter high max weightEdit
Controls the maximum influence of the meter high limit relative to histogram over exposure, similar to max weight.
Overexp thresh x/100kEdit
The fraction in parts per 100,000 of pixels that must be near to white level to trigger over exposure protection. 0 Disables over exposure protection. Note: for low values to work, the histogram step (described below) must be set to allow a sufficient number of pixels to be sampled. If the threshold represents less than 10 pixels, are warning is recorded in the log.
Overexp Ev rangeEdit
What fraction of a stop below white level is considered over exposed.
Overexp max weightEdit
Maximum weight for over exposure protection. This controls how far beyond the overexposure threshold the weight continues to increase.
Priority modifier for over exposure protection. If greater than 0, the weight of under exposure and meter driven exposure increases is reduced proportional to the amount of over exposure, such that the "prio" value is subtracted from the weight when overexp thresh is reached.
Underexp thresh x/100kEdit
Similar to overexp thresh above, but for pixels near black level. 0 Disabled under exposure protection.
Number of stops below neutral a value must be to count as underexposed. Pixels with 0-4 histogram value are not counted as underexposure, since this usually indicates defective pixels.
Underexp max weightEdit
Maximum weight for under exposure protection, similar to Overexp max weight above.
Priority modifier for under exposure protection, similar to Overexp prio above.
The step between pixels sampled for under / over protection. Smaller values sample more pixels, allowing smaller areas to be detected and lower under/over thresholds to be used, at the expense of increased processing time. On D10 (12 megapixel) a step of 11 samples ~100k pixels, and takes 60-70ms to process, equating to ~1500 pixels/msec.
Override Canon jpeg resolution. "Default" uses current Canon UI setting, remaining values correspond to Canon UI. Not all values are valid on all cameras, and sizes for some modes vary between models.
Override current CHDK raw setting. "Default" uses current CHDK UI setting, otherwise, force on or off.
Use cont. mode if setEdit
Shoot by holding the shutter button down, if continuous shooting is enabled in the Canon menu. If not checked, or if continuous mode is not enabled in the Canon menu, shooting is done by holding down half shoot and clicking full.
Display power saving mode.
- On = always on,
- Off use set_lcd_display off.
- Blt_Off use set_backlight every shot.
Pressing set while the script is running turns the display on for 30 sec.
Shutdown on finishEdit
Shutdown after "shots" images are taken.
Shutdown on low batteryEdit
Shutdown if the battery voltage falls below the CHDK OSD low battery value.
Shutdown on low spaceEdit
Shutdown if the SD card space falls below 50 MB.
Interval warn LEDEdit
Blink LED if shooting falls below the rate specified by Interval above. The mapping of LED numbers to LEDs is cameras specific, and not all cameras support script LED control.
Interval warn beepEdit
Play a beep shooting falls below the rate specified by Interval above. The beep will only be played the Canon UI has not been set to Mute.
Draw Debug infoEdit
Draw metering information on the raw buffer. From top to bottom
- A bar near the top of the screen, representing the metered value as a % of white level (log scale), with tick marks at 1 EV intervals. The larger ticks are black level, neutral, white level, and Ev shift if enabled. Yellow and red ticks above the bar show the meter thresh and limit values, respectively.
- A bar showing the change in exposure as a fraction of the max EV change value described below. The origin is at the center, with longer exposure to the right, and shorter exposure to the left.
- A bar extending from left to right, representing % underexposed pixels. Red if above threshold, max at middle of screen.
- A bar extending from right to left, representing % overexposed pixels. Red if above threshold, max at middle of screen.
- A box or reticle around the meter area, controlled by Meter Area option below
Drawing takes adds a small amount to shooting time (e.g. 20 ms on d10.) The log contains this information and much more, but having it in the images makes it easier to see what the code is doing.
Controls whether the metering area is drawn, either just the corners or a box
Gauge Y offset %Edit
Controls the vertical location of the debug info, in % of full resolution jpeg area height. 0 uses a default offset of 80 pixels. Use to make the debug info visible on jpegs with a different aspect ration form the sensor.
If enabled, the log is written to A/rawopint.csv
- None = Do not record a log.
- Append = append to existing log. A new header row is written each time the script is started.
- Replace = Replace any existing log.
The log is described in more detail below.
Raw hook sleepEdit
Insert a script sleep just before the raw hook is released. This appears to avoid issues with D10 taking some shots with the shutter closed when debug drawing is not enabled.
Disable script yieldEdit
Disable automatic script yielding. May prevent D10 shutter issues mentioned above, and possibly reduce interval. May have other side effects.
PROBABLY BROKEN IN CURRENT SCRIPT VERSION. Read A/rawopsim.csv and feed the initial exposure values, scene brightness and over/under % to the metering algorithm without actually shooting. The CSV should be in the same format as the log file. Output will be written to A/rawopint.csv as usual, but some columns will not be filled in or will have values different form what the column label indicates.
Log file descriptionEdit
The log is a CSV with a bunch of columns. Some of the more relevant/useful ones are described below.
Values in the first line of data are from the pre-shoot before the first shot. The desc column in this line includes a summary of script settings, chdk version, and zoom + focus state.
sv, tv, avEdit
exposure values for the shot with the image number indicated in the exp column.
Scene brightness in APEX*96 units. For pre-shoot, this comes from the canon firmware. Otherwise, calculated from exposure params and meter value. The calculated value only roughly corresponds to the real definition of Bv.
Average value of the metered area: meter is the raw sensor value, *96 gives this as an APEX96 value where the target exposure is 0. In meter96, negative values indicate an underexposure, positive is overexposure.
Fraction of pixel in the over and under ranges, in percent.
meter_weight, over_weight, under_weightEdit
Weights controlling the relative contribution of the meter area, under and over exposure protection to the final exposure change.
The amount the target Ev was shifted to reflect scene brightness, in APEX*96 units.
exposure change requested for the next shot. Positive values mean more exposure (longer Tv, higher ISO)
various cases in the metering process trigger messages here, e.g if shutter limits hit etc. The first row contains initial settings.
How long the script slept between shots. If it's negative, it means the script wasn't able to maintain the requested frame rate.
The tick count (milliseconds since boot) of the start of the main loop. The values of shoot_ready, exp_start, raw_ready and raw_done are offsets form this value
Pressing menu ends the script. Pressing set turns on the display for 30 sec.