Notation and Definitions

A feedforward network such as that of Figure-#fig1#295> consists of an input layer whose nodes are merely fan-outs (no processing takes place in these nodes), one or more hidden layer(s) of computational nodes, and an output layer, also with one or more computational nodes. For the rest of the discussion, we will assume that both the hidden and output layer nodes use the same activation function. It will be convenient to have a uniform notation for referring to the node input/output and weights in a feedforward network. Let #tex2html_wrap_inline2565# the output of the i#tex2html_wrap_inline2567# node in the #tex2html_wrap_inline2569# layer of the net, #tex2html_wrap_inline2571# is the connection weight between i#tex2html_wrap_inline2573# node in the #tex2html_wrap_inline2575# layer and j#tex2html_wrap_inline2577# node of the r#tex2html_wrap_inline2579# layer (the current layer), and #tex2html_wrap_inline2581# is the threshold of the j#tex2html_wrap_inline2583# node in the r#tex2html_wrap_inline2585# layer. Each computational node of layer r yields an output of #tex2html_wrap_inline2589#. The function #tex2html_wrap_inline2591# is any function satisfying the property that both #tex2html_wrap_inline2593# and #tex2html_wrap_inline2595# exist and are distinct. In this context, #tex2html_wrap_inline2597# is often called a squashing function [#11##1#]. #tex2html_wrap_inline2599# Typically, #tex2html_wrap_inline2601# is one of the following:
  1. For a #tex2html_wrap_inline2603# output range, the activation function is:

    #equation1566#

  2. For the output range of #tex2html_wrap_inline2605#, the activation function is:

    #equation1569#

#tex2html_wrap_inline2607# is equivalent to #tex2html_wrap_inline2609#, up to translations and change of coordinates. The relation between these two functions is:

#equation1574#

#tex2html_wrap_inline2611# We say that #tex2html_wrap_inline2613# is a <#325#> smooth activation function<#325#> if and only if

  • #tex2html_wrap_inline2615# is a squashing function,
  • #tex2html_wrap_inline2617# has a continuous derivative everywhere (hence, #tex2html_wrap_inline2619# is continuous),
  • #tex2html_wrap_inline2621# is odd (hence, #tex2html_wrap_inline2623#).
#tex2html_wrap_inline2625# #tex2html_wrap_inline2627# is a smooth activation function. #tex2html_wrap_inline2629#