Issues of implementing neural network algorithms on memristor crossbars

The property of natural parallelization of matrix-vector operations inherent in memristor crossbars creates opportunities for their effective use in neural network computing. Analog calculations are orders of magnitude faster in comparison to calculations on the central processor and on graphics accelerators. Besides, mathematical operations energy costs are significantly lower. The essential feature of analog computing is its low accuracy. In this regard, studying the dependence of neural network quality on the accuracy of setting its weights is relevant. The paper considers two convolutional neural networks trained on the MNIST (handwritten digits) and CIFAR_10 (airplanes, boats, cars, etc.) data sets. The first convolutional neural network consists of two convolutional layers, one subsample layer and two fully connected layers. The second one consists of four convolutional layers, two subsample layers and two fully connected layers. Calculations in convolutional and fully connected layers are performed through matrix-vector operations that are implemented on memristor crossbars. Sub-sampling layers imply the operation of finding the maximum value from several values. This operation can be implemented at the analog level. The process of training a neural network runs separately from data analysis. As a rule, gradient optimization methods are used at the training stage. It is advisable to perform calculations using these methods on CPU. When setting the weights, 3—4 precision bits are required to obtain an acceptable recognition quality in the case the network is trained on MNIST. 6-10 precision bits are required if the network is trained on CIFAR_10.

memristor, crossbar, accuracy, neural network, convolution, MNIST, CIFAR_10

1. Wong H.-S. P., Lee H.-Y., Yu S., Chen Y.-S., Wu Y., Chen P.-S., Lee B., Chen F. T., Tsai M.-J. Metal-oxide RRAM. Proceedings of the IEEE, 2012, vol. 100, no. 6, pp. 1951—1970. DOI: 10.1109/JPROC.2012.2190369

2. Yang J. J., Strukov D. B., Stewart D. R., Memristive devices for computing. Nature Nanotechnology, 2013, vol. 8, no. 1, pp. 13—24. DOI: 10.1038/nnano.2012.240

3. Li C., Hu M., Li Y., Jiang H., Ge N., Montgomery E., Zhang J., Song W., Dávila N., Graves C. E., Li Z., Strachan J. P., Lin P., Wang Z., Barnell M., Wu Q., Williams R. S., Yang J. J., Xia Q. Analogue signal and image processing with large memristor crossbars. Nature Electronics, 2018, vol. 1, no. 1, pp. 52—59. DOI: 10.1038/s41928-017-0002-z

4. Hu M, Graves C. E., Li C., Li Y., Ge N., Montgomery E., Dávila N., Jiang H., Williams R. S., Yang J. J., Xia O., Strachan J. P. Memristor-based analog computation and neural network classification with a dot product engine. Advanced Materials, 2018, vol. 30, no. 9, p. 1705914. DOI: 10.1002/adma.201705914

5. Tarkov M. S. Implementation of a neural WTA-network on the memristor crossbar. Prikl. Diskr. Mat. Suppl., 2015, no. 8, pp. 151—154. (In. Russ.). DOI: 10.17223/2226308X/8/59

6. Diehl P. U., Cook M. Unsupervised learning of digit recognition using spike-timing-dependent plasticity. Front. Comput. Neurosci., 2015, vol. 9, art. 99, p. 9. DOI: 10.3389/fncom.2015.00099

7. Ambrogio S., Balatti S., Milo V., Carboni R., Wang Z.-Q., Calderoni A., Ramaswamy N., Ielmini D. Neuromorphic learning and recognition with one-transistor-one-resistor synapses and bistable metal oxide RRAM. IEEE Transactions on Electron Devices, 2016, vol. 63, no. 4, pp. 1508—1515. DOI: 10.1109/TED.2016.2526647

8. Guo Y., Wu H., Gao B., Qian H. Unsupervised learning on resistive memory array based spiking neural networks. Front. Neurosci., 2019, vol. 13, art. 812. DOI: 10.3389/fnins.2019.00812

9. Milo V., Pedretti G., Laudato M., Bricalli A., Ambrosi E., Bianchi S., Chicca E., Ielmini D. Resistive switching synapses for unsupervised learning in feed-forward and recurrent neural networks. In: IEEE International Symposium on Circuits and Systems (ISCAS). Florence (Italy): IEEE, 2018, pp. 1—5. DOI: 10.1109/ISCAS.2018.8351824

10. Pedretti G., Bianchi S., Milo V., Calderoni A., Ramaswamy N., Ielmini D. Modeling-based design of brain-inspired spiking neural networks with RRAM learning synapses. In: IEEE International Electron Devices Meeting (IEDM). San Francisco (CA, USA): IEEE, 2017, pp. 28.1.1—28.1.4. DOI: 10.1109/IEDM.2017.8268467

11. Milo V., Ielmini D., Chicca E. Attractor networks and associative memories with STDP learning in RRAM synapses. In: IEEE International Electron Devices Meeting (IEDM). San Francisco (CA, USA): IEEE, 2017, pp. 11.2.1—11.2.4. DOI: 10.1109/IEDM.2017.8268369

12. Li B., Shan Y., Hu M., Wang Y., Chen Y., Yang H., Memristor-based approximated computation. In: International Symposium on Low Power Electronics and Design (ISLPED). Beijing (China): IEEE, 2013, pp. 242—247. DOI: 10.1109/ISLPED.2013.6629302

13. Teplov G. S., Gornev E. S. Multilevel bipolar memristor model considering deviations of switching parameters in the Verilog-A language. Russ. Microelectron., 2019, vol. 48, no. 3, pp. 131—142. DOI: 10.1134/S1063739719030107

14. Morozov A. Yu., Reviznikov D. L. Adaptive interpolation algorithm based on a kd-tree for numerical integration of systems of ordinary differential equations with interval initial conditions. Differential Equations., 2018, vol. 54, no. 7, pp. 945—956. DOI: 10.1134/S0012266118070121

15. Morozov A. Yu., Reviznikov D. L., Gidaspov V. Yu. Adaptive interpolation algorithm based on a kd-tree for the problems of chemical kinetics with interval parameters. Mathematical Models and Computer Simulations, 2019, vol. 11, no. 4, pp. 622—633. DOI: 10.1134/S2070048219040100

16. Morozov A. Yu., Reviznikov D. L. Modelling of dynamic systems with interval parameters on graphic processors. Programmnaya Ingeneria = Software Engineering, 2019, vol. 10, no. 2, pp. 69—76. DOI: 10.17587/prin.10.69-76

17. Morozov A. Yu., Reviznikov D. L. Metody komp’yuternogo modelirovaniya dinamicheskikh sistem s interval’nymi parametrami [Methods for computer modeling of dynamic systems with interval parameters ]. Moscow: Izd-vo Moskovskogo aviatsionnogo instituta, 2019, 160 p. (In Russ.)

18. Gulli A., Pal S. Deep learning with Keras: implement neural networks with Keras on Theano and TensorFlow. Birmingham; Mumbai: Packt Publishing Ltd., 2017, 490 p.

19. MNIST CNN. URL: https://keras.io/examples/mnist_cnn/ (accessed: 01.10.2019).

20. Train a simple deep CNN on the CIFAR10 small images dataset. URL: https://keras.io/examples/cifar10_cnn/ (accessed: 01.10.2019).