1This document is a tutorial/initiation for writing simple filters in 2libavfilter. 3 4Foreword: just like everything else in FFmpeg, libavfilter is monolithic, which 5means that it is highly recommended that you submit your filters to the FFmpeg 6development mailing-list and make sure it is applied. Otherwise, your filter is 7likely to have a very short lifetime due to more a less regular internal API 8changes, and a limited distribution, review, and testing. 9 10Bootstrap 11========= 12 13Let's say you want to write a new simple video filter called "foobar" which 14takes one frame in input, changes the pixels in whatever fashion you fancy, and 15outputs the modified frame. The most simple way of doing this is to take a 16similar filter. We'll pick edgedetect, but any other should do. You can look 17for others using the `./ffmpeg -v 0 -filters|grep ' V->V '` command. 18 19 - cp libavfilter/vf_{edgedetect,foobar}.c 20 - sed -i s/edgedetect/foobar/g -i libavfilter/vf_foobar.c 21 - sed -i s/EdgeDetect/Foobar/g -i libavfilter/vf_foobar.c 22 - edit libavfilter/Makefile, and add an entry for "foobar" following the 23 pattern of the other filters. 24 - edit libavfilter/allfilters.c, and add an entry for "foobar" following the 25 pattern of the other filters. 26 - ./configure ... 27 - make -j<whatever> ffmpeg 28 - ./ffmpeg -i tests/lena.pnm -vf foobar foobar.png 29 30If everything went right, you should get a foobar.png with Lena edge-detected. 31 32That's it, your new playground is ready. 33 34Some little details about what's going on: 35libavfilter/allfilters.c:avfilter_register_all() is called at runtime to create 36a list of the available filters, but it's important to know that this file is 37also parsed by the configure script, which in turn will define variables for 38the build system and the C: 39 40 --- after running configure --- 41 42 $ grep FOOBAR config.mak 43 CONFIG_FOOBAR_FILTER=yes 44 $ grep FOOBAR config.h 45 #define CONFIG_FOOBAR_FILTER 1 46 47CONFIG_FOOBAR_FILTER=yes from the config.mak is later used to enable the filter in 48libavfilter/Makefile and CONFIG_FOOBAR_FILTER=1 from the config.h will be used 49for registering the filter in libavfilter/allfilters.c. 50 51Filter code layout 52================== 53 54You now need some theory about the general code layout of a filter. Open your 55libavfilter/vf_foobar.c. This section will detail the important parts of the 56code you need to understand before messing with it. 57 58Copyright 59--------- 60 61First chunk is the copyright. Most filters are LGPL, and we are assuming 62vf_foobar is as well. We are also assuming vf_foobar is not an edge detector 63filter, so you can update the boilerplate with your credits. 64 65Doxy 66---- 67 68Next chunk is the Doxygen about the file. See http://ffmpeg.org/doxygen/trunk/. 69Detail here what the filter is, does, and add some references if you feel like 70it. 71 72Context 73------- 74 75Skip the headers and scroll down to the definition of FoobarContext. This is 76your local state context. It is already filled with 0 when you get it so do not 77worry about uninitialized read into this context. This is where you put every 78"global" information you need, typically the variable storing the user options. 79You'll notice the first field "const AVClass *class"; it's the only field you 80need to keep assuming you have a context. There are some magic you don't care 81about around this field, just let it be (in first position) for now. 82 83Options 84------- 85 86Then comes the options array. This is what will define the user accessible 87options. For example, -vf foobar=mode=colormix:high=0.4:low=0.1. Most options 88have the following pattern: 89 name, description, offset, type, default value, minimum value, maximum value, flags 90 91 - name is the option name, keep it simple, lowercase 92 - description are short, in lowercase, without period, and describe what they 93 do, for example "set the foo of the bar" 94 - offset is the offset of the field in your local context, see the OFFSET() 95 macro; the option parser will use that information to fill the fields 96 according to the user input 97 - type is any of AV_OPT_TYPE_* defined in libavutil/opt.h 98 - default value is an union where you pick the appropriate type; "{.dbl=0.3}", 99 "{.i64=0x234}", "{.str=NULL}", ... 100 - min and max values define the range of available values, inclusive 101 - flags are AVOption generic flags. See AV_OPT_FLAG_* definitions 102 103In doubt, just look at the other AVOption definitions all around the codebase, 104there are tons of examples. 105 106Class 107----- 108 109AVFILTER_DEFINE_CLASS(foobar) will define a unique foobar_class with some kind 110of signature referencing the options, etc. which will be referenced in the 111definition of the AVFilter. 112 113Filter definition 114----------------- 115 116At the end of the file, you will find foobar_inputs, foobar_outputs and 117the AVFilter ff_vf_foobar. Don't forget to update the AVFilter.description with 118a description of what the filter does, starting with a capitalized letter and 119ending with a period. You'd better drop the AVFilter.flags entry for now, and 120re-add them later depending on the capabilities of your filter. 121 122Callbacks 123--------- 124 125Let's now study the common callbacks. Before going into details, note that all 126these callbacks are explained in details in libavfilter/avfilter.h, so in 127doubt, refer to the doxy in that file. 128 129init() 130~~~~~~ 131 132First one to be called is init(). It's flagged as cold because not called 133often. Look for "cold" on 134http://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html for more 135information. 136 137As the name suggests, init() is where you eventually initialize and allocate 138your buffers, pre-compute your data, etc. Note that at this point, your local 139context already has the user options initialized, but you still haven't any 140clue about the kind of data input you will get, so this function is often 141mainly used to sanitize the user options. 142 143Some init()s will also define the number of inputs or outputs dynamically 144according to the user options. A good example of this is the split filter, but 145we won't cover this here since vf_foobar is just a simple 1:1 filter. 146 147uninit() 148~~~~~~~~ 149 150Similarly, there is the uninit() callback, doing what the name suggest. Free 151everything you allocated here. 152 153query_formats() 154~~~~~~~~~~~~~~~ 155 156This is following the init() and is used for the format negotiation, basically 157where you say what pixel format(s) (gray, rgb 32, yuv 4:2:0, ...) you accept 158for your inputs, and what you can output. All pixel formats are defined in 159libavutil/pixfmt.h. If you don't change the pixel format between the input and 160the output, you just have to define a pixel formats array and call 161ff_set_common_formats(). For more complex negotiation, you can refer to other 162filters such as vf_scale. 163 164config_props() 165~~~~~~~~~~~~~~ 166 167This callback is not necessary, but you will probably have one or more 168config_props() anyway. It's not a callback for the filter itself but for its 169inputs or outputs (they're called "pads" - AVFilterPad - in libavfilter's 170lexicon). 171 172Inside the input config_props(), you are at a point where you know which pixel 173format has been picked after query_formats(), and more information such as the 174video width and height (inlink->{w,h}). So if you need to update your internal 175context state depending on your input you can do it here. In edgedetect you can 176see that this callback is used to allocate buffers depending on these 177information. They will be destroyed in uninit(). 178 179Inside the output config_props(), you can define what you want to change in the 180output. Typically, if your filter is going to double the size of the video, you 181will update outlink->w and outlink->h. 182 183filter_frame() 184~~~~~~~~~~~~~~ 185 186This is the callback you are waiting from the beginning: it is where you 187process the received frames. Along with the frame, you get the input link from 188where the frame comes from. 189 190 static int filter_frame(AVFilterLink *inlink, AVFrame *in) { ... } 191 192You can get the filter context through that input link: 193 194 AVFilterContext *ctx = inlink->dst; 195 196Then access your internal state context: 197 198 FoobarContext *foobar = ctx->priv; 199 200And also the output link where you will send your frame when you are done: 201 202 AVFilterLink *outlink = ctx->outputs[0]; 203 204Here, we are picking the first output. You can have several, but in our case we 205only have one since we are in a 1:1 input-output situation. 206 207If you want to define a simple pass-through filter, you can just do: 208 209 return ff_filter_frame(outlink, in); 210 211But of course, you probably want to change the data of that frame. 212 213This can be done by accessing frame->data[] and frame->linesize[]. Important 214note here: the width does NOT match the linesize. The linesize is always 215greater or equal to the width. The padding created should not be changed or 216even read. Typically, keep in mind that a previous filter in your chain might 217have altered the frame dimension but not the linesize. Imagine a crop filter 218that halves the video size: the linesizes won't be changed, just the width. 219 220 <-------------- linesize ------------------------> 221 +-------------------------------+----------------+ ^ 222 | | | | 223 | | | | 224 | picture | padding | | height 225 | | | | 226 | | | | 227 +-------------------------------+----------------+ v 228 <----------- width -------------> 229 230Before modifying the "in" frame, you have to make sure it is writable, or get a 231new one. Multiple scenarios are possible here depending on the kind of 232processing you are doing. 233 234Let's say you want to change one pixel depending on multiple pixels (typically 235the surrounding ones) of the input. In that case, you can't do an in-place 236processing of the input so you will need to allocate a new frame, with the same 237properties as the input one, and send that new frame to the next filter: 238 239 AVFrame *out = ff_get_video_buffer(outlink, outlink->w, outlink->h); 240 if (!out) { 241 av_frame_free(&in); 242 return AVERROR(ENOMEM); 243 } 244 av_frame_copy_props(out, in); 245 246 // out->data[...] = foobar(in->data[...]) 247 248 av_frame_free(&in); 249 return ff_filter_frame(outlink, out); 250 251In-place processing 252~~~~~~~~~~~~~~~~~~~ 253 254If you can just alter the input frame, you probably just want to do that 255instead: 256 257 av_frame_make_writable(in); 258 // in->data[...] = foobar(in->data[...]) 259 return ff_filter_frame(outlink, in); 260 261You may wonder why a frame might not be writable. The answer is that for 262example a previous filter might still own the frame data: imagine a filter 263prior to yours in the filtergraph that needs to cache the frame. You must not 264alter that frame, otherwise it will make that previous filter buggy. This is 265where av_frame_make_writable() helps (it won't have any effect if the frame 266already is writable). 267 268The problem with using av_frame_make_writable() is that in the worst case it 269will copy the whole input frame before you change it all over again with your 270filter: if the frame is not writable, av_frame_make_writable() will allocate 271new buffers, and copy the input frame data. You don't want that, and you can 272avoid it by just allocating a new buffer if necessary, and process from in to 273out in your filter, saving the memcpy. Generally, this is done following this 274scheme: 275 276 int direct = 0; 277 AVFrame *out; 278 279 if (av_frame_is_writable(in)) { 280 direct = 1; 281 out = in; 282 } else { 283 out = ff_get_video_buffer(outlink, outlink->w, outlink->h); 284 if (!out) { 285 av_frame_free(&in); 286 return AVERROR(ENOMEM); 287 } 288 av_frame_copy_props(out, in); 289 } 290 291 // out->data[...] = foobar(in->data[...]) 292 293 if (!direct) 294 av_frame_free(&in); 295 return ff_filter_frame(outlink, out); 296 297Of course, this will only work if you can do in-place processing. To test if 298your filter handles well the permissions, you can use the perms filter. For 299example with: 300 301 -vf perms=random,foobar 302 303Make sure no automatic pixel conversion is inserted between perms and foobar, 304otherwise the frames permissions might change again and the test will be 305meaningless: add av_log(0,0,"direct=%d\n",direct) in your code to check that. 306You can avoid the issue with something like: 307 308 -vf format=rgb24,perms=random,foobar 309 310...assuming your filter accepts rgb24 of course. This will make sure the 311necessary conversion is inserted before the perms filter. 312 313Timeline 314~~~~~~~~ 315 316Adding timeline support 317(http://ffmpeg.org/ffmpeg-filters.html#Timeline-editing) is often an easy 318feature to add. In the most simple case, you just have to add 319AVFILTER_FLAG_SUPPORT_TIMELINE_GENERIC to the AVFilter.flags. You can typically 320do this when your filter does not need to save the previous context frames, or 321basically if your filter just alter whatever goes in and doesn't need 322previous/future information. See for instance commit 86cb986ce that adds 323timeline support to the fieldorder filter. 324 325In some cases, you might need to reset your context somehow. This is handled by 326the AVFILTER_FLAG_SUPPORT_TIMELINE_INTERNAL flag which is used if the filter 327must not process the frames but still wants to keep track of the frames going 328through (to keep them in cache for when it's enabled again). See for example 329commit 69d72140a that adds timeline support to the phase filter. 330 331Threading 332~~~~~~~~~ 333 334libavfilter does not yet support frame threading, but you can add slice 335threading to your filters. 336 337Let's say the foobar filter has the following frame processing function: 338 339 dst = out->data[0]; 340 src = in ->data[0]; 341 342 for (y = 0; y < inlink->h; y++) { 343 for (x = 0; x < inlink->w; x++) 344 dst[x] = foobar(src[x]); 345 dst += out->linesize[0]; 346 src += in ->linesize[0]; 347 } 348 349The first thing is to make this function work into slices. The new code will 350look like this: 351 352 for (y = slice_start; y < slice_end; y++) { 353 for (x = 0; x < inlink->w; x++) 354 dst[x] = foobar(src[x]); 355 dst += out->linesize[0]; 356 src += in ->linesize[0]; 357 } 358 359The source and destination pointers, and slice_start/slice_end will be defined 360according to the number of jobs. Generally, it looks like this: 361 362 const int slice_start = (in->height * jobnr ) / nb_jobs; 363 const int slice_end = (in->height * (jobnr+1)) / nb_jobs; 364 uint8_t *dst = out->data[0] + slice_start * out->linesize[0]; 365 const uint8_t *src = in->data[0] + slice_start * in->linesize[0]; 366 367This new code will be isolated in a new filter_slice(): 368 369 static int filter_slice(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) { ... } 370 371Note that we need our input and output frame to define slice_{start,end} and 372dst/src, which are not available in that callback. They will be transmitted 373through the opaque void *arg. You have to define a structure which contains 374everything you need: 375 376 typedef struct ThreadData { 377 AVFrame *in, *out; 378 } ThreadData; 379 380If you need some more information from your local context, put them here. 381 382In you filter_slice function, you access it like that: 383 384 const ThreadData *td = arg; 385 386Then in your filter_frame() callback, you need to call the threading 387distributor with something like this: 388 389 ThreadData td; 390 391 // ... 392 393 td.in = in; 394 td.out = out; 395 ctx->internal->execute(ctx, filter_slice, &td, NULL, FFMIN(outlink->h, ctx->graph->nb_threads)); 396 397 // ... 398 399 return ff_filter_frame(outlink, out); 400 401Last step is to add AVFILTER_FLAG_SLICE_THREADS flag to AVFilter.flags. 402 403For more example of slice threading additions, you can try to run git log -p 404--grep 'slice threading' libavfilter/ 405 406Finalization 407~~~~~~~~~~~~ 408 409When your awesome filter is finished, you have a few more steps before you're 410done: 411 412 - write its documentation in doc/filters.texi, and test the output with make 413 doc/ffmpeg-filters.html. 414 - add a FATE test, generally by adding an entry in 415 tests/fate/filter-video.mak, add running make fate-filter-foobar GEN=1 to 416 generate the data. 417 - add an entry in the Changelog 418 - edit libavfilter/version.h and increase LIBAVFILTER_VERSION_MINOR by one 419 (and reset LIBAVFILTER_VERSION_MICRO to 100) 420 - git add ... && git commit -m "avfilter: add foobar filter." && git format-patch -1 421 422When all of this is done, you can submit your patch to the ffmpeg-devel 423mailing-list for review. If you need any help, feel free to come on our IRC 424channel, #ffmpeg-devel on irc.freenode.net. 425