The Independence Property and Piecewise Rational Functions

Assume our network uses #tex2html_wrap_inline3083# functions (see definitions #pwrfunction#640> and #pwrfunctiontype2#641>) as the activations of the nodes. Now, we need to address the issue of whether these types of activation functions satisfy the IP. <#642#> First<#642#>: Assume the activation function is #tex2html_wrap_inline3087#. Choose an integer l ;SPM_gt; 0, non-zero real numbers #tex2html_wrap_inline3091# and real numbers #tex2html_wrap_inline3093#.

#equation1829#

otherwise

#equation1833#

Select a set of coefficients #tex2html_wrap_inline3095#. We need to show:

#equation1836#

By confining one's attention to x for which |x| is sufficiently large that #tex2html_wrap_inline3101#, for all i, one can see that it is sufficient to show that #tex2html_wrap_inline3105# satisfies IP. A sketch of the proof follows: <#1840#> Proof<#1840#>.\ <#651#>Note<#651#>: We show by contradiction that #tex2html_wrap_inline3107# satisfies the IP. The proof is a bit long and involved so we have decided not to include it here. Furthermore, it is quite trivial to show that #tex2html_wrap_inline3109# satisfies WIP. For more information on these concepts or the proof, contact the author. % <#652#> Second<#652#>: Let the activation function be #tex2html_wrap_inline3111#. In a personal communication, Sontag [#20##1#] said that #tex2html_wrap_inline3113# satisfies the independence property. Clearly, networks using #tex2html_wrap_inline3115#, and assuming the above conjecture, networks using #tex2html_wrap_inline3117#, satisfy the property that if #tex2html_wrap_inline3119# and #tex2html_wrap_inline3121# are irreducible in #tex2html_wrap_inline3123# and the #tex2html_wrap_inline3125# then #tex2html_wrap_inline3127#.