Degenerate matter is matter whose properties are determined by quantum mechanics. This matter can behave very differently to the matter that we are familiar with; you can call degenerate matter the counterpart of matter that respects the ideal gas law. Matter becomes degenerate under the enormous pressures of very dense stellar remnants.
If the core of a star is compressed, the atoms and the electrons in the core get closer and closer to each other. Normally such a compression happens if the temperature in the core decreases and as a consequence the thermal pressure that counteracts the force of gravity decreases too. But the electrons cannot get closer to each other than the laws of quantum mechanics allow. According to these laws it is not possible that more than two electrons in the same atom can have the same quantum state. This means that if matter is compressed to a certain degree, the Pauli Exclusion Principle generates a counter pressure called degeneracy pressure which prevents further contraction of the matter. Due to this degeneracy pressure the matter can support a much higher pressure than it would be able to support without degeneracy. One important consequence is that the pressure of degenerate matter does not depend on temperature any more as is the case for normal matter when increasing its temperature causes a gas to also increase its pressure (see ideal gas law). Amongst others this fact plays a crucial role with supernova Type Ia. Now, by further increasing the pressure on degenerate matter there is still no further compression due to the Pauli Exclusion Principle, but instead the speed with which the electrons move increases. This way the pressure on degenerate matter can still be increased without collapse of the matter. Anyhow, if pressure is increased to a certain point (the Chandrasekhar limit) the electrons approach the speed of light. There is nothing in this universe that can reach or exceed the speed of light. At this point degeneracy pressure is no longer able to support the pressure of the matter and the matter suddenly collapses. In astronomy this happens with core collapse supernovae (Type II, Type Ib and Type Ic supernova).
The next "stop" of the matter is a neutron star. Most of the electrons and protons of the progenitor star have been literally pressed into each other by the enormous pressure, forming neutrons. You can imagine the neutron star as a gigantic nucleus consisting of almost nothing but neutrons. It is extremely compact. To understand this better, imagine the size of a normal atom to be as big as a football stadium. In such an atom the nucleus would be the size of a grain of rice at its centre. All the other space would be empty with just the even smaller electrons flying around in it. A neutron star is so compact because the former atom (or football stadium) is now squeezed into the size of just the nucleus (or rice grain) itself. In effect, the neutron star still consists of single atomic nuclei, but packed together almost seamlessly.
Now, the Pauli Exclusion Principle is not only valid for electrons but for any particles called fermions (which are all half integer spin-particles like protons, neutrons, muons and others). So, instead of electron degeneracy the neutron star is held up against collapse from neutron degeneracy with the main difference that the neutron degeneracy pressure is much higher. The same Pauli Exclusion Principle applies; a neutron must occupy its own quantum state (or space) and cannot be compressed further. If we apply even higher pressure again by adding more matter to the neutron star, this leads to higher speeds of the vibrating neutrons. If the mass of a neutron star reaches 3 solar masses the speed of the neutrons again reaches the speed of light and the neutron star cannot support its own weight anymore. This limit is called the Tolman-Oppenheimer-Volkhof limit; it's analogous to the Chandrasekhar limit of electron degenerate matter. The neutron star collapses to a theoretical object called a quark star. These quark stars have not been discovered and are only a hypothesis at this time.
You can read more about the formation of neutron stars and black holes in our supernova article; more information about a neutron star can be found in the corresponding neutron star article on Sun.org.