You can't put RTSP inside of the setup(), block, since neither Flash or HTML5 supports it.
HLS Live on Android
For live streams, there's not yet a good solution for playback on Android or other non-iOS mobile platforms. One workaround is to offer a native app for Android, in which HLS streams can be supported. This is beyond the scope of JW Player though. Another option, if you use the Wowza Media Server, is to offer a fallback RTSP stream for devices that don't support HLS. For example:
watch this stream over RTSP
jwplayer("myElement").setup({
file: "http://example.com:1935/vod/mp4:myVideo.mp4/playlist.m3u8",
image: "/assets/myPoster.jpg",
fallback: false
});