In 1949, Donald Hebb simple proposal that "neurons that fire together wire together" inspired research into neural networks. His idea was that neurons that fire at the same time should strengthen their connection weight. In this assignment, you will explore Hebbian learning and see how it can support aspects of human categorization.
1. You are going to train a hebbian neural network to learn the concepts bird and mammal. The model will be trained on specific examples of birds and mammals. Each example will be represented as a list in python consisting of four numbers. Each number in the list will be -1.0 or 1.0. The first member of the list will indicate whether the example is a bird (-1) or mammal (1). The second number indicates whether the example is small (-1) or large (1). The third number indicates whether the example lives on the ground (-1) or in a tree (1). The fourth number indicates whether the example doesn't sing (-1) or does (1).
Come up with three real world examples of birds and three examples of mammals that have two of the three characteristic or prototypical features. Give the list representation for each item.
2. Using these six items, train up a hebbian auto-associative network with four inputs and four outputs. There will 4X4=16 weights. Use the hebbian learning rule .1*INPUT*OUTPUT to update each weight on each learning trial. Auto-associate each item with itself using a random ordering of items (use random.shuffle to accomplish this). Do this ten times (60 learning trials) and save and examine the weights. What did the model learn?
3. Using the weights from the previous question, you will use your hebbian model to auto-complete missing information. What happens when you input a trained item minus the category label? What happen when you show the idealized prototype (i.e., an item that has all the characteristic features of the category, plus or minus 1)? Do old items or the idealized prototype create a more definitive response? Why is this?
4. What happens if you show the category label and all the features but one? How does the model fill in the missing feature? Does it "remember" the item it saw during training correctly?
5. What happens if you just show the category label?
6. Change the learning rule to be .1*INPUT*OUTPUT*(1-|weight|). How does this change affect learning? Vary the number of training blocks (before you always trained for 10 blocks). What do the weights converge to? If you now include both prototypes during training, how does this affect what the weights converge to?
7. Make up two new categories that the model cannot learn. You might want look back at your prototype assignment if stuck.
8. Come up with some exploration on your own. If you are stuck, one suggestion is to look at recency effects (i.e., how the last item experienced changes the response of model to the next item). If you explore this idea, change the learning rate to see how it affects the size of the recency effect.