1Filter design 2============= 3 4This document explains guidelines that should be observed (or ignored with 5good reason) when writing filters for libavfilter. 6 7In this document, the word ���frame��� indicates either a video frame or a group 8of audio samples, as stored in an AVFilterBuffer structure. 9 10 11Format negotiation 12================== 13 14 The query_formats method should set, for each input and each output links, 15 the list of supported formats. 16 17 For video links, that means pixel format. For audio links, that means 18 channel layout, sample format (the sample packing is implied by the sample 19 format) and sample rate. 20 21 The lists are not just lists, they are references to shared objects. When 22 the negotiation mechanism computes the intersection of the formats 23 supported at each end of a link, all references to both lists are replaced 24 with a reference to the intersection. And when a single format is 25 eventually chosen for a link amongst the remaining list, again, all 26 references to the list are updated. 27 28 That means that if a filter requires that its input and output have the 29 same format amongst a supported list, all it has to do is use a reference 30 to the same list of formats. 31 32 query_formats can leave some formats unset and return AVERROR(EAGAIN) to 33 cause the negotiation mechanism to try again later. That can be used by 34 filters with complex requirements to use the format negotiated on one link 35 to set the formats supported on another. 36 37 38Buffer references ownership and permissions 39=========================================== 40 41 Principle 42 --------- 43 44 Audio and video data are voluminous; the buffer and buffer reference 45 mechanism is intended to avoid, as much as possible, expensive copies of 46 that data while still allowing the filters to produce correct results. 47 48 The data is stored in buffers represented by AVFilterBuffer structures. 49 They must not be accessed directly, but through references stored in 50 AVFilterBufferRef structures. Several references can point to the 51 same buffer; the buffer is automatically deallocated once all 52 corresponding references have been destroyed. 53 54 The characteristics of the data (resolution, sample rate, etc.) are 55 stored in the reference; different references for the same buffer can 56 show different characteristics. In particular, a video reference can 57 point to only a part of a video buffer. 58 59 A reference is usually obtained as input to the start_frame or 60 filter_frame method or requested using the ff_get_video_buffer or 61 ff_get_audio_buffer functions. A new reference on an existing buffer can 62 be created with the avfilter_ref_buffer. A reference is destroyed using 63 the avfilter_unref_bufferp function. 64 65 Reference ownership 66 ------------------- 67 68 At any time, a reference ���belongs��� to a particular piece of code, 69 usually a filter. With a few caveats that will be explained below, only 70 that piece of code is allowed to access it. It is also responsible for 71 destroying it, although this is sometimes done automatically (see the 72 section on link reference fields). 73 74 Here are the (fairly obvious) rules for reference ownership: 75 76 * A reference received by the filter_frame method (or its start_frame 77 deprecated version) belongs to the corresponding filter. 78 79 Special exception: for video references: the reference may be used 80 internally for automatic copying and must not be destroyed before 81 end_frame; it can be given away to ff_start_frame. 82 83 * A reference passed to ff_filter_frame (or the deprecated 84 ff_start_frame) is given away and must no longer be used. 85 86 * A reference created with avfilter_ref_buffer belongs to the code that 87 created it. 88 89 * A reference obtained with ff_get_video_buffer or ff_get_audio_buffer 90 belongs to the code that requested it. 91 92 * A reference given as return value by the get_video_buffer or 93 get_audio_buffer method is given away and must no longer be used. 94 95 Link reference fields 96 --------------------- 97 98 The AVFilterLink structure has a few AVFilterBufferRef fields. The 99 cur_buf and out_buf were used with the deprecated 100 start_frame/draw_slice/end_frame API and should no longer be used. 101 src_buf, cur_buf_copy and partial_buf are used by libavfilter internally 102 and must not be accessed by filters. 103 104 Reference permissions 105 --------------------- 106 107 The AVFilterBufferRef structure has a perms field that describes what 108 the code that owns the reference is allowed to do to the buffer data. 109 Different references for the same buffer can have different permissions. 110 111 For video filters that implement the deprecated 112 start_frame/draw_slice/end_frame API, the permissions only apply to the 113 parts of the buffer that have already been covered by the draw_slice 114 method. 115 116 The value is a binary OR of the following constants: 117 118 * AV_PERM_READ: the owner can read the buffer data; this is essentially 119 always true and is there for self-documentation. 120 121 * AV_PERM_WRITE: the owner can modify the buffer data. 122 123 * AV_PERM_PRESERVE: the owner can rely on the fact that the buffer data 124 will not be modified by previous filters. 125 126 * AV_PERM_REUSE: the owner can output the buffer several times, without 127 modifying the data in between. 128 129 * AV_PERM_REUSE2: the owner can output the buffer several times and 130 modify the data in between (useless without the WRITE permissions). 131 132 * AV_PERM_ALIGN: the owner can access the data using fast operations 133 that require data alignment. 134 135 The READ, WRITE and PRESERVE permissions are about sharing the same 136 buffer between several filters to avoid expensive copies without them 137 doing conflicting changes on the data. 138 139 The REUSE and REUSE2 permissions are about special memory for direct 140 rendering. For example a buffer directly allocated in video memory must 141 not modified once it is displayed on screen, or it will cause tearing; 142 it will therefore not have the REUSE2 permission. 143 144 The ALIGN permission is about extracting part of the buffer, for 145 copy-less padding or cropping for example. 146 147 148 References received on input pads are guaranteed to have all the 149 permissions stated in the min_perms field and none of the permissions 150 stated in the rej_perms. 151 152 References obtained by ff_get_video_buffer and ff_get_audio_buffer are 153 guaranteed to have at least all the permissions requested as argument. 154 155 References created by avfilter_ref_buffer have the same permissions as 156 the original reference minus the ones explicitly masked; the mask is 157 usually ~0 to keep the same permissions. 158 159 Filters should remove permissions on reference they give to output 160 whenever necessary. It can be automatically done by setting the 161 rej_perms field on the output pad. 162 163 Here are a few guidelines corresponding to common situations: 164 165 * Filters that modify and forward their frame (like drawtext) need the 166 WRITE permission. 167 168 * Filters that read their input to produce a new frame on output (like 169 scale) need the READ permission on input and must request a buffer 170 with the WRITE permission. 171 172 * Filters that intend to keep a reference after the filtering process 173 is finished (after filter_frame returns) must have the PRESERVE 174 permission on it and remove the WRITE permission if they create a new 175 reference to give it away. 176 177 * Filters that intend to modify a reference they have kept after the end 178 of the filtering process need the REUSE2 permission and must remove 179 the PRESERVE permission if they create a new reference to give it 180 away. 181 182 183Frame scheduling 184================ 185 186 The purpose of these rules is to ensure that frames flow in the filter 187 graph without getting stuck and accumulating somewhere. 188 189 Simple filters that output one frame for each input frame should not have 190 to worry about it. 191 192 filter_frame 193 ------------ 194 195 This method is called when a frame is pushed to the filter's input. It 196 can be called at any time except in a reentrant way. 197 198 If the input frame is enough to produce output, then the filter should 199 push the output frames on the output link immediately. 200 201 As an exception to the previous rule, if the input frame is enough to 202 produce several output frames, then the filter needs output only at 203 least one per link. The additional frames can be left buffered in the 204 filter; these buffered frames must be flushed immediately if a new input 205 produces new output. 206 207 (Example: frame rate-doubling filter: filter_frame must (1) flush the 208 second copy of the previous frame, if it is still there, (2) push the 209 first copy of the incoming frame, (3) keep the second copy for later.) 210 211 If the input frame is not enough to produce output, the filter must not 212 call request_frame to get more. It must just process the frame or queue 213 it. The task of requesting more frames is left to the filter's 214 request_frame method or the application. 215 216 If a filter has several inputs, the filter must be ready for frames 217 arriving randomly on any input. Therefore, any filter with several inputs 218 will most likely require some kind of queuing mechanism. It is perfectly 219 acceptable to have a limited queue and to drop frames when the inputs 220 are too unbalanced. 221 222 request_frame 223 ------------- 224 225 This method is called when a frame is wanted on an output. 226 227 For an input, it should directly call filter_frame on the corresponding 228 output. 229 230 For a filter, if there are queued frames already ready, one of these 231 frames should be pushed. If not, the filter should request a frame on 232 one of its inputs, repeatedly until at least one frame has been pushed. 233 234 Return values: 235 if request_frame could produce a frame, it should return 0; 236 if it could not for temporary reasons, it should return AVERROR(EAGAIN); 237 if it could not because there are no more frames, it should return 238 AVERROR_EOF. 239 240 The typical implementation of request_frame for a filter with several 241 inputs will look like that: 242 243 if (frames_queued) { 244 push_one_frame(); 245 return 0; 246 } 247 while (!frame_pushed) { 248 input = input_where_a_frame_is_most_needed(); 249 ret = ff_request_frame(input); 250 if (ret == AVERROR_EOF) { 251 process_eof_on_input(); 252 } else if (ret < 0) { 253 return ret; 254 } 255 } 256 return 0; 257 258 Note that, except for filters that can have queued frames, request_frame 259 does not push frames: it requests them to its input, and as a reaction, 260 the filter_frame method will be called and do the work. 261 262Legacy API 263========== 264 265 Until libavfilter 3.23, the filter_frame method was split: 266 267 - for video filters, it was made of start_frame, draw_slice (that could be 268 called several times on distinct parts of the frame) and end_frame; 269 270 - for audio filters, it was called filter_samples. 271