Patent No. 5995954 Method and apparatus for associative memory

 

Patent No. 5995954

Method and apparatus for associative memory (Loos, Nov 30, 1999)

Abstract

A method and apparatus for an electronic artificial neural network, which serves as an associative memory that has a complete set of N-dimensional Hadamard vectors as stored states, suitable for large N that are powers of 2. The neural net has nonlinear synapses, each of which processes signals from two neurons. These synapses can be implemented by simple passive circuits comprised of eight resistors and four diodes. The connections in the neural net are specified through a subset of a group that is defined over the integers from 1 to N. The subset is chosen such that the connections can be implemented in VLSI or wafer scale integration. An extension of the Hadamard memory causes the memory to provide new Hadamard vectors when these are needed for the purpose of Hebb learning.

Notes:  

Funded by DARPA. Filed March 1992, granted November 1999. This one is a computer program for a neural network memory. The rest, 5782874, 5800481, 5935054, 6017302, 6091994, 6238333, and 6506148 all deal with various forms of nervous system manipulation. Basically they deal with subliminal brainwave entrainment. The patents go into pulsative, thermal, electrical, magnetic, electromagnetic, and acoustical ways of doing this. The author refers to creating 'sensory resonances' by subliminally manipulating the various presentation methods described above and using them on other parts of the body away from the brain. This sends impulses to the brain and entrains it to the frequencies stated in the patents. The patents read like most other entrainment devices or methods. In the patents there are no mentions of any EEGs or other monitoring devices being used. Some of the patents state that they may have uses in the 'non-lethal' weapons area. All of the patents tend to stick to the same two frequencies for their effects (at ½ Hz a person would be almost literally brain dead and at 2.4 Hz a person would be little more than comatose). Only one of them mentions any malicious uses, 6506148. The ones that the MPRII guidelines would apply to, state that they can be used for manipulation of someone's nervous system and still comply with those guidelines. A couple of the patents mention the elapsed time before the mentioned effects occur, this happens with all entrainment devices. The patents are an example of causing some physiological effects through the subliminal entrainment. The frequencies as stated above should be taken into account when reading the list of effects caused by the various forms of the subliminal entrainment. Most of the effects sound like things that happen when someone is close to death.

SUMMARY

It is the object of the present invention to provide a method and means for constructing Hadamard memories of large dimension, in such a manner that 1) the connections can be implemented in VLSI or wafer-scale integration, 2) the synapses are such that they can be implemented by simple passive circuits, and 3) the Hadamard memory supplies heretofore unused Hadamard vectors for the purpose of Hebb learning.

Object 1) is met as follows.

In the course of computer experiments we discovered that, as the dimension N is increased, the fraction of connections that needs to be implemented in order to obtain good associative recall diminishes dramatically, to a degree that was completely unexpected. This sets the stage for massive pruning of the connections implied by (4) or (5), essentially without loss of performance.

The set of selected connections needs to be shift invariant, in order that the dendrites can be implemented in a compact fashion on a chip or wafer. There is also a slope limitation on the segments of the dendrites, which is chosen to avoid overcrowding of the dendrites.

Finally, the selected connections must be chosen from the components i, j, k, for which the connection tensor (4) or (5) is nonzero. For large N, the calculation of these index combinations (i,j,k), based on the Hadamard vectors, is very lengthy. We give a shortcut for this calculation, based on group properties, shift invariance, and the window property. From the resulting set of index combinations (i,j,k), a small subset of connections to be implemented is selected by using the slope limitation condition. An example result is shown in FIGS. 1 and 2, for dimension N=1024.

Object 2) is met as follows.

The quadratic dendrite current (1) requires use of product synapses, which, for signal inputs y.sub.j and y.sub.k, produce an output current proportional to y.sub.j y.sub.k. Although the product operation is simple in computer simulation, implementation in hardware is cumbersome, requiring at least 9 transistors. Considering the very large number of synapses that need to be placed on a chip or wafer for large dimension N, it is important to simplify the synapse circuitry, which means giving up the product function. We have found a synapse circuit that is comprised of 8 resistors and 4 diodes, and that provides, for fully developed neuron signals, the same output current as a product synapse; however, this circuit requires a doubling of the neuron signal lines, such that one line carries the signal voltage, and the other line carries the opposite voltage. For underdeveloped neuron signals, i.e., signals with magnitude below unity, the synapse output current deviates from the product function. However, computer simulations of associative recall for N=1024 and N=2048 have shown that these "RD" synapses are satisfactory, if the connections are chosen in the manner discussed.

Object 3) is met as follows.

Hebb learning amounts to adding to the synaptic matrix of the BLT a matrix

where h.sub..gamma. is the Hadamard vector that we want x to be labeled with. The Hebb addition (6) to the BLT matrix should be made only if the input vector x is much different from all the vectors that have been stored so far. That condition can be easily checked by computing the correlation between the input vector and output vector of the Hadamard memory. For large N, it is cumbersome to find the new Hadamard vector h.sub..gamma. that must be presented to the back of the BLT for the Hebb learning (6). We have found a method of acquiring h.sub..gamma., simply by letting the Hadamard memory determine this vector by associative recall, using as input vector a bipolar counter word of bit length m=log.sub.2 N, concatenated with zeros. Incrementing the counter, everytime that learning is required as evidenced by a "weak" correlation, and presenting the counter word to the Hadamard memory, then results in a new Hadamard vector for the Hebb learning. The method relies on the so-called window property of Hadamard vectors that have been constructed from maximal-length shift-register sequences [13].

The invention is not limited by the embodiments shown in the drawings and described in the description, which are given by way of example and not of limitation, but only in accordance with the scope of the appended claims.

REFERENCES

[1] H. G. Loos, "Reflexive Associative Memories", in NEURAL INFORMATION PROCESSING SYSTEMS, Ed. Dana Z. Anderson, American Institute of Physics, New York, 1988, p.495

[2] H. G. Loos, "Quadratic Hadamard Memories", Proceedings, International Joint Conference on Neural Networks, San Diego, Calif., Jun. 17-21, 1990, p. I-735

[3] H. G. Loos, "Continuum Model for Quadratic Memories", Proceedings, International Joint Conference on Neural Networks, Seattle, Wash., Jul. 8-12, 1991, p.11-297

[4] H. G. Loos, "Quadratic Hadamard Memories I", Tech. Rep. #1, ARPA Order No. 6429, Contract DAAH01-88-C-0887, 1989

[5] B. Kosko, Proceedings IEEE International Conference on Neural Networks (ICNN-87), p.II-261 (1987)

[6] B. Kosko, IEEE Trans. SMC-18, 49 (1988)

[7] B. Kosko, Appl. Optics 26, 4947 (1987)

[8] H. H. Chen, Y. C. Lee, G. Z. Sun, and H. Y. Lee, "High Order Correlation Model for Associative Memory", in NEURAL NETWORKS FOR COMPUTING, AIP Conference Proceedings #151, p.86 (1986)

[9] D. J. Volper and S. E. Hampson, Neural Networks 3, 93 (1990)

[10] J. J. Hopfield, "Neural networks and physical systems with emergent collective computational abilities", Proc. Natl. Acad. Sci. USA 79, 2554 (1982)

[11] C. Mead, ANALOG VLSI AND NEURAL SYSTEMS, Addison-Wesley, New York, 1989

[12] D. O. Hebb, ORGANIZATION OF BEHAVIOR; A NEUROPSYCHOLOGICAL THEORY, Wiley, New York, 1949

[13] M. Harwit and N. J. A. Sloane, HADAMARD TRANSFORM OPTICS, Academic Press, New York, 1979

[14] H. G. Loos, Quadratic Hadamard Memories II", Tech. Rep. #2, ARPA Order No. 6429, Contract DAAH01-88-C-0887, 1990

[15] H. G. Loos, "Group Theory of Memories with Quadratic Activation", Tech. Rep. #3, ARPA Order No. 6429, Contract DAAH01-88-C-0887, 1991

[16] H. G. Loos, "Adaptive Stochastic Content-Addressable Memory", Final Report, ARPA Order No. 6429, Contract DAAH01-88-C-0887, 1991