Why are we interested in them?

  1. Piecewise linear functions (#tex2html_wrap_inline3129#) are simple to evaluate.
  2. Smooth activation functions, such as #tex2html_wrap_inline3131# of the previous section, were originally introduced as smooth versions of certain piecewise linear activation functions (in this case, the clipping type).
  3. Clipping plays a quite useful role as outlined recently by Hertz et. al [#11##1#] (see the section on discretization and clipping).
  4. It is stated in Sontag's work[#15##1#] that the conventional squashing function (see previous section) as well as the #tex2html_wrap_inline3133# satisfy the criteria of continuity and differentiability almost everywhere (see conditions #tex2html_wrap_inline3135# and #tex2html_wrap_inline3137# of definition 3.2 in [#15##1#]).