Definitions and Observations

For an integer k ;SPM_gt; 0, and real value #tex2html_wrap_inline2849#, we define:

#equation1704#

The first #tex2html_wrap_inline2851# activation function used is:

#equation1708#

#tex2html_wrap_inline2853# The first derivative of the function in (#pwrform1#460>) is:

#equation1723#

and the second derivative is:

#equation1726#

Now the first derivative of #tex2html_wrap_inline2855# is:

#equation1729#

#tex2html_wrap_inline2857# From previous definitions and observations, we observe the following:

Also we observe some properties of #tex2html_wrap_inline2869# that are shared with #tex2html_wrap_inline2871#:
  • For all x, #tex2html_wrap_inline2875# .
  • #tex2html_wrap_inline2877# is odd.
  • #tex2html_wrap_inline2879# and #tex2html_wrap_inline2881# are continuous.
  • For all x, #tex2html_wrap_inline2885#.
  • #tex2html_wrap_inline2887# and #tex2html_wrap_inline2889#.
#tex2html_wrap_inline2891# The second #tex2html_wrap_inline2893# function, due to Elliott [#9##1#], is defined next.

#equation1748#

#tex2html_wrap_inline2895# The function stated in equation (#pwrtype21#493>) is differentiable everywhere to:

#equation1754#

and satisfies a ``general'' logistic differential equation as follows:

#equation1757#

#tex2html_wrap_inline2897# The steps leading to equation (#pwrtype22#501>) from function (#pwrtype21#502>) are as follows:

#equation1763#

differentiating to:

#equation1768#

and a second derivative of:

#equation1771#

Also,

#equation1774#

differentiating to:

#equation1779#

and a second derivative of:

#equation1782#

Now, from equations (#tex2html_wrap_inline2899#, #tex2html_wrap_inline2901#) we derive the following:

#equation1785#

#tex2html_wrap_inline2903# It is worth pointing out that #tex2html_wrap_inline2905# and #tex2html_wrap_inline2907# satisfy an ``autonomous'' differential equation, so that calculating #tex2html_wrap_inline2909# or #tex2html_wrap_inline2911# requires very little effort beyond the calculation of the function itself. The aforementioned ``autonomy'' concept can be illustrated by the following example: To evaluate functions #tex2html_wrap_inline2913# and #tex2html_wrap_inline2915#, the only values needed are #tex2html_wrap_inline2917# and #tex2html_wrap_inline2919#, respectively. Regarding the activation function #tex2html_wrap_inline2921#, it seems not to satisfy such a differential equation, but even so calculation of #tex2html_wrap_inline2923# requires negligible additional effort. Also, networks using these three types of activation functions, can be trained using backpropogation algorithm. #tex2html_wrap_inline2925# We observe some properties of #tex2html_wrap_inline2927# shared with #tex2html_wrap_inline2929#:

  • #tex2html_wrap_inline2931# and #tex2html_wrap_inline2933# are continuous.
  • For all x, #tex2html_wrap_inline2937#.
  • #tex2html_wrap_inline2939# and #tex2html_wrap_inline2941#.
  • #tex2html_wrap_inline2943# is odd.
  • For all x, #tex2html_wrap_inline2947#.
#tex2html_wrap_inline2949# %

#figure557#
Figure 2: f1() = #tex2html_wrap_inline2951#, f2() = #tex2html_wrap_inline2953#, and f3() = #tex2html_wrap_inline2955#.

% %