CAT5 Cable for AES3 (balanced) Digital Audio
While the AES standard for balanced digital audio (AES3-2003) specifies balanced and twisted shielded cable with particular characteristics, it acknowledges, in Appendix D, that unshielded twisted pair (UTP) CAT5 cabling can also be successfully deployed. It also provides the following guidelines.
When choosing a cable, make sure there's enough twist in the twisted pair to reject magnetic or electrical interference.
Don't mix cable types. If you choose CAT5 UTP cable for AES3, then use this unshielded cable throughout the system. On the other hand, if you choose CAT5 shielded twisted pair, then use that consistently.
On RJ45 connectors, the preferred pair for AES3 is pins 4 and 5, with pins 3 and 6 being the second choice. If you're wiring a newly built system and you don't need to conform to previous wiring, then use pins 4 and 5 consistently.
If you need to make RJ45 to XLR adapter cables, wire pin 2 of the XLR to pin 5 (or other odd-numbered pin) of the RJ45, and pin 3 of the XLR to pin 4 (or other even-numbered pin) of the RJ45.
With quality cable properly terminated and installed, you should be able to use CAT5 cable up to 1300 feet or so, unequalized, or double that equalized.