- Piecewise linear functions (#tex2html_wrap_inline3129#) are simple to evaluate.
- Smooth activation functions, such as #tex2html_wrap_inline3131# of the previous
section, were originally introduced as smooth versions of certain
piecewise linear activation functions (in this case, the clipping
type).
- Clipping plays a quite useful role as outlined
recently by Hertz et. al [#11##1#] (see the section on
discretization and clipping).
- It is stated in Sontag's work[#15##1#] that the
conventional squashing function (see previous section) as well as the
#tex2html_wrap_inline3133# satisfy the criteria of continuity and differentiability
almost everywhere (see conditions #tex2html_wrap_inline3135# and #tex2html_wrap_inline3137# of definition 3.2 in
[#15##1#]).