Large scale concert production TCP/IP networking

[Mike Brown]

As the industry moves into more and more networked and smart devices I wonder what you guys are doing as far as network configurations. Almost seems like I am wiring more ethernet than XLR these days!

Lets say we have on a show:

A FOH amplifier monitoring/dsp system like IRIS net or LA Network manager or similar.
A MON amplifier monitoring/dsp system like IRIS net or LA Network manager or similar.
A FOH console with tablet/ipad mixing capabilities like VENUE etc
A MON console with tablet/ipad mixing capabilities (may or may not be same brand)
Wireless microphone monitoring via network
Wireless internet (have to surf the forums to complain about rogue BE engineers ruining your stuff)
And maybe some other crap.

For some damn reason everyone wants all of these features all the time, before and during the show.

Plus maybe some non-audio (but who really cares) networks like:

A lighting control console with wireless DMX
Video camera PTZ control/whatever via network etc etc etc

I really see two ways to go about doing this:

1. Create an "Audio" network for all audio related tasks, a "Lighting" and "Video" networks for their tasks as well.

Pros: FOH or MON or system tech could check in and control any service without having to swap networks. Simple computer setup.
Cons: Reliability? Bandwidth? Is anyone running into problems here? Annoying wiring/router setup

2. Crate a separate network for each service, "FOH AMPS" "MON AMPS" "FOH CONSOLE" "MON CONSOLE" etc etc etc

Pros: Reliability? Bandwidth? Fairly easy wiring router setup
Cons: System tech/whoever having to continuously change networks to control different devices, annoying usability?


Anyone starting to hit the limits of bandwidth of 802.11x? Reliability problems with many different clients and apps? Slow response on VNC stuff etc?


I would imagine most production houses going route 2 so that they can build it into their cases and have very modular shows. I imagine most tours going with 1 for ease of setup, tuning, fast control, etc.


So how are you guys attacking this to maximize setup/teardown speed, ease of use, and, of course, reliability.

 

[Rasmus Rosenberg]

Re: Large scale concert production TCP/IP networking

As the industry moves into more and more networked and smart devices I wonder what you guys are doing as far as network configurations. Almost seems like I am wiring more ethernet than XLR these days!

Lets say we have on a show:

A FOH amplifier monitoring/dsp system like IRIS net or LA Network manager or similar.
A MON amplifier monitoring/dsp system like IRIS net or LA Network manager or similar.
A FOH console with tablet/ipad mixing capabilities like VENUE etc
A MON console with tablet/ipad mixing capabilities (may or may not be same brand)
Wireless microphone monitoring via network
Wireless internet (have to surf the forums to complain about rogue BE engineers ruining your stuff)
And maybe some other crap.

For some damn reason everyone wants all of these features all the time, before and during the show.

Plus maybe some non-audio (but who really cares) networks like:

A lighting control console with wireless DMX
Video camera PTZ control/whatever via network etc etc etc

I really see two ways to go about doing this:

1. Create an "Audio" network for all audio related tasks, a "Lighting" and "Video" networks for their tasks as well.

Pros: FOH or MON or system tech could check in and control any service without having to swap networks. Simple computer setup.
Cons: Reliability? Bandwidth? Is anyone running into problems here? Annoying wiring/router setup

2. Crate a separate network for each service, "FOH AMPS" "MON AMPS" "FOH CONSOLE" "MON CONSOLE" etc etc etc

Pros: Reliability? Bandwidth? Fairly easy wiring router setup
Cons: System tech/whoever having to continuously change networks to control different devices, annoying usability?


Anyone starting to hit the limits of bandwidth of 802.11x? Reliability problems with many different clients and apps? Slow response on VNC stuff etc?


I would imagine most production houses going route 2 so that they can build it into their cases and have very modular shows. I imagine most tours going with 1 for ease of setup, tuning, fast control, etc.


So how are you guys attacking this to maximize setup/teardown speed, ease of use, and, of course, reliability.

What a good question, can't wait to hear other answers. The way we tackle it is as 2 create a separate network for each service. That serves as a good building block for the rental company. You get a service and a computer to control that service. That said most jobs I do you spend a little time at the warehouse to group things, so how ever i responsible gets control. An example one network for the sys tech with FOH amps, measurement setup. The FOH guy get his console and Internet. The Mon guy got his Mon amps and the IEM and his console. A separate computer for the WL mic's etc etc. So I guess I link the networks to what can fail together. You don't want the mon eng crashing the PA network og the FOH guy to take out the network for the WL mic's by accident.

Also a lot of times its a question of making sense, just because you can get all on network with direct access, all them time, don't mean its better, easier to use or safer, in alot of instance i would say its not. I Think a lot over complicate things just because they can, but hell we do that with most things in audio

 

[Cameron Stuckey]

Re: Large scale concert production TCP/IP networking

I've discovered that 802.11a/b/g is totally unreliable for any show. During set-up and tech there will be a dozen devices in peoples' pockets that will be in the 2.4GHz environment, which any upper level consumer or professional access point can easily handle while providing full bandwidth. But open the doors and in walks thousands of devices all requesting security information from the AP, and you effectively DDoS your wireless AP slowing output to a trickel if at all*. Right now 5GHz is viable because zero cell phones have a chipset that operates up there. But that spectrum will become consumer territory soon, it's just too good to pass up. Because of that unreliability it's gotta be CAT5 for any show critical operations.

*A great example was the WWDC from 2010 where an iPhone demo failed because there was over 500 WiFi devices in one room all competing for the same spectrum, not even the same AP. Wireless Woes Rain 'Fail' on Steve Jobs' Keynote

 

[Tim Duffin]

Re: Large scale concert production TCP/IP networking

As the industry moves into more and more networked and smart devices I wonder what you guys are doing as far as network configurations. Almost seems like I am wiring more ethernet than XLR these days!

Lets say we have on a show:

A FOH amplifier monitoring/dsp system like IRIS net or LA Network manager or similar.
A MON amplifier monitoring/dsp system like IRIS net or LA Network manager or similar.
A FOH console with tablet/ipad mixing capabilities like VENUE etc
A MON console with tablet/ipad mixing capabilities (may or may not be same brand)
Wireless microphone monitoring via network
Wireless internet (have to surf the forums to complain about rogue BE engineers ruining your stuff)
And maybe some other crap.

For some damn reason everyone wants all of these features all the time, before and during the show.

Plus maybe some non-audio (but who really cares) networks like:

A lighting control console with wireless DMX
Video camera PTZ control/whatever via network etc etc etc

I really see two ways to go about doing this:

1. Create an "Audio" network for all audio related tasks, a "Lighting" and "Video" networks for their tasks as well.

Pros: FOH or MON or system tech could check in and control any service without having to swap networks. Simple computer setup.
Cons: Reliability? Bandwidth? Is anyone running into problems here? Annoying wiring/router setup

2. Crate a separate network for each service, "FOH AMPS" "MON AMPS" "FOH CONSOLE" "MON CONSOLE" etc etc etc

Pros: Reliability? Bandwidth? Fairly easy wiring router setup
Cons: System tech/whoever having to continuously change networks to control different devices, annoying usability?


Anyone starting to hit the limits of bandwidth of 802.11x? Reliability problems with many different clients and apps? Slow response on VNC stuff etc?


I would imagine most production houses going route 2 so that they can build it into their cases and have very modular shows. I imagine most tours going with 1 for ease of setup, tuning, fast control, etc.


So how are you guys attacking this to maximize setup/teardown speed, ease of use, and, of course, reliability.

Its not really a problem. You just have to statically address your devices and turn all of the consumer functions of your routers off. This means no UDP broadcast, no wep encryption, block ICMP ping, all of it has to be turned off--which also mean that you can't download itunes on your phone while you are attempting to comm with a DSP. For the purposes of this forum, turning off wep actually reduces the number of rejections per second of failed login attempts, lowering total overhead in the application layer on some chipsets... Also, ditch your 49.99 router you got from bestbuy and get a managed gigabit switch with high powered WAP. Enterprise level IT engineers can help you choose an appropriate device configuration.

 

[Jens Bacher]

Re: Large scale concert production TCP/IP networking

I've just finished a system for a musical with 8 amps on LA Network, a Yamaha DME Matrix, a Digico SD10, and 45 Sennheiser devices monitored on WSM. We used two 16 port gigibit switches at Amp-world and wireless-world, connected a Linksys router to it for wireless access during setup, and internet access through the WAN port. The FOH and WSM computers are connected with cable. The trick to making this work the first time is to make sure the area where the router hands out adresses via DHCP is not overlapping with the static IP's for the amps and DME.

 

[Jeff Stevens]

Re: Large scale concert production TCP/IP networking

Its not really a problem. You just have to statically address your devices and turn all of the consumer functions of your routers off. This means no UDP broadcast, no wep encryption, block ICMP ping, all of it has to be turned off--which also mean that you can't download itunes on your phone while you are attempting to comm with a DSP. For the purposes of this forum, turning off wep actually reduces the number of rejections per second of failed login attempts, lowering total overhead in the application layer on some chipsets... Also, ditch your 49.99 router you got from bestbuy and get a managed gigabit switch with high powered WAP. Enterprise level IT engineers can help you choose an appropriate device configuration.

I second everything Tim said except turning off the encryption. There is a huge risk in doing that for a small performance gain. Just because your SSID is not being broadcast does not mean that the network cannot be found. There are scores of utilities that can sniff out non-broadcasting networks and the traffic going across them. Given just a few packets, they can determine your IP address scheme and how to spoof any network application, subjecting you to a possible Man-in-the-Middle attack. While I believe that most are just innocents looking for the quickest way to Facebook, it is never a good idea to leave the front-door unlocked.

The way I tend to do these things is every rack gets a network switch. All static devices get an IP address statically assigned (i.e., if it is mounted in the rack, then it is static to that rack) with the rack number (e.g. 192.168.[Rack Number].[Device Number]). Then, the master rack that holds the intercom, also gets a managed gigabit switch and an enterprise router. Every rack then connects to that switch in its own VLAN using the gigabit uplink ports and routing between networks is handled by the router, also through a gigabit port. This way, I can talk to any device and if any of the links goes down, the rack will still function individually. Plus, using the router, I can block certain traffic from talking to each other to reduce overhead (to make device discovery easy, things like wireless use a lot of broadcast traffic to identify themselves, but this goes to all devices in the LAN, wasting their resources) and enable Quality of Service so audio traffic is prioritized over data.

I only use wireless for "client" machines like laptops and tablets. I prefer keeping the number of access points to a minimum since most consumer access points do not have the ability to adjust their frequency to avoid conflicts with adjacent access points. I secure the networks by disabling SSID broadcast, disallowing users to connect to the access points' admin functions through the wireless itself, and using WPA 2 encryption. If you know all the devices that are going to be connecting to any given WAP, you can also use the MAC address filter. Generally, it is a PITA to administer if you want to change to a new device, but it will prevent the WAP from responding to any requests from devices it does not recognize (it is very easy to spoof a MAC address, though, so you still need the encryption to prevent that data from from being stolen).

~Jeff Stevens

 

[Klaus Zimmermann]

Re: Large scale concert production TCP/IP networking

If you know all the devices that are going to be connecting to any given WAP, you can also use the MAC address filter. Generally, it is a PITA to administer if you want to change to a new device, but it will prevent the WAP from responding to any requests from devices it does not recognize (it is very easy to spoof a MAC address, though, so you still need the encryption to prevent that data from from being stolen).

from MY experience, I use a combination of MAC addresses - I call it WHITE list - invisible SSID and NO encryption, as well NO INTERNET ACCESS with big success! and if you need to add an additional device, whatever it may be, to the network, just add it's Mac address (this will be done within 5 min MAX!).

also, start with a ifferent number than 192.168.x.x, because about 5 billion people on this planet are using this sceme :lol:
... but your [rack].[device] is cool - just need to embedd in my next plannings8)~~
TX

...and I'M no network manager/admin or so...

 

[Chad Young]

Re: Large scale concert production TCP/IP networking

from MY experience, I use a combination of MAC addresses - I call it WHITE list - invisible SSID and NO encryption, as well NO INTERNET ACCESS with big success! and if you need to add an additional device, whatever it may be, to the network, just add it's Mac address (this will be done within 5 min MAX!).

also, start with a ifferent number than 192.168.x.x, because about 5 billion people on this planet are using this sceme :lol:
... but your [rack].[device] is cool - just need to embedd in my next plannings8)~~
TX

...and I'M no network manager/admin or so...

Oddly I *AM* a network manager/admin. My day job is IT security.

For the record, MAC addresses can be easily spoofed and viewed freely, even on an encrypted/hidden network. Also, 'invisible' SSIDs aren't, at least not someone who knows how to 'see' invisible SSIDs. Making it invisible just keeps it from being casually seen by yoru average Joe with a smart phone or laptop.

I suggest the network be isolated (no internet access), enable WPA2-PSK security, and your call on whether to broadcast the SSID or not - it makes no difference to anyone who is serious about compromising your network. Make sure you change the default password and IP ranges on the router. Makre sure no one writes the PSK on the router or where people can casually see it.

 

[Scott Helmke]

Re: Large scale concert production TCP/IP networking

Hey Chad (and others who may know), are you familiar with the old Cisco Aironet 1200 configuration? We use one for big shows, because of the antenna options. But every time I try to do anything with the security I end up needing to do a factory reset - it's a whole different animal from the little Netgear boxes I use for small gigs.

 

[David Jameson]

Re: Large scale concert production TCP/IP networking

Although I haven't used that exact model, I believe you will need to connect to it through console and use IOS commands to change the configurations (like a Cisco Switch). I've used the 1131s and now the 1231s and that is how they have to be configured.

Good luck.

 

[Scott Helmke]

Re: Large scale concert production TCP/IP networking

Although I haven't used that exact model, I believe you will need to connect to it through console and use IOS commands to change the configurations (like a Cisco Switch). I've used the 1131s and now the 1231s and that is how they have to be configured.

Thanks - I know how to connect through the console port and do basic config stuff. It's just that the security on the 1200 is somewhat of an older generation of wifi, and also has many more options. I couldn't figure out how to just give it a simple security setup with a regular password instead of a hexadecimal string password.

 

[Mike Brown]

Re: Large scale concert production TCP/IP networking

Thanks guys for the posts, this conversation went in a different direction than I first envisioned, but very interesting stuff none the less!

Oddly I *AM* a network manager/admin. My day job is IT security.

For the record, MAC addresses can be easily spoofed and viewed freely, even on an encrypted/hidden network. Also, 'invisible' SSIDs aren't, at least not someone who knows how to 'see' invisible SSIDs. Making it invisible just keeps it from being casually seen by yoru average Joe with a smart phone or laptop.

I suggest the network be isolated (no internet access), enable WPA2-PSK security, and your call on whether to broadcast the SSID or not - it makes no difference to anyone who is serious about compromising your network. Make sure you change the default password and IP ranges on the router. Makre sure no one writes the PSK on the router or where people can casually see it.

My question for you would be: While concealing the SSID does not stop anyone who wants to find it, does concealing the SSID prevent the effect of almost DDoSing your (possibly consumer) access point(s) in an arena like situation?

 

[Jeff Stevens]

Re: Large scale concert production TCP/IP networking

Thanks guys for the posts, this conversation went in a different direction than I first envisioned, but very interesting stuff none the less!




My question for you would be: While concealing the SSID does not stop anyone who wants to find it, does concealing the SSID prevent the effect of almost DDoSing your (possibly consumer) access point(s) in an arena like situation?

Possibly. Chad mentioned that it will stop the average Joe from seeing and attempting to connect to your network, reducing that overhead. It is important to remember, though, that we should not be using consumer WAPs in a pro setting. Would you use a Behringer or Mackie at FOH and expect it to get the job done? Probably not. The same goes for network hardware. In truth, the amount of additional overhead encryption places on the computer and WAP is minimal and provides a great deal of security. MACs can always be spoofed, because they are never encrypted. Logically, if the MAC or IP were encrypted, there would be no way for the routers and switches to understand where the message is going. It would be like the Post Office trying to deliver a letter addressed in a secret code. The contents may be encrypted, but the address may not be.

In general, I have found internet access to be extremely valuable when I have needed a manual, tech support, or even the most recent FCC database for WWB. I put the wireless in a different subnet and then use the router to restrict access for all the audio devices and allow it only for the client computers.

Scott, I have never configured a 1200 series, but I have configured my fair share of other Cisco devices, including more recent WAPs (I too am Cisco Certified). If you share the exact model number and software version, perhaps in a new thread or PM with the current configuration, I'm sure there are people who can help.


~Jeff Stevens

 

[Jelmer de Jong]

Re: Large scale concert production TCP/IP networking

.

The way I tend to do these things is every rack gets a network switch. All static devices get an IP address statically assigned (i.e., if it is mounted in the rack, then it is static to that rack) with the rack number (e.g. 192.168.[Rack Number].[Device Number]). Then, the master rack that holds the intercom, also gets a managed gigabit switch and an enterprise router. Every rack then connects to that switch in its own VLAN using the gigabit uplink ports and routing between networks is handled by the router, also through a gigabit port. This way, I can talk to any device and if any of the links goes down, the rack will still function individually. Plus, using the router, I can block certain traffic from talking to each other to reduce overhead (to make device discovery easy, things like wireless use a lot of broadcast traffic to identify themselves, but this goes to all devices in the LAN, wasting their resources) and enable Quality of Service so audio traffic is prioritized over data.


~Jeff Stevens

If you give ip-adresses based on device type you can reduce broadcasttraffic with subnets. 192.168.1.x for wireless mics, 192.168.2.x for IEMs, etc.

 

[David Jameson]

Re: Large scale concert production TCP/IP networking

Thanks - I know how to connect through the console port and do basic config stuff. It's just that the security on the 1200 is somewhat of an older generation of wifi, and also has many more options. I couldn't figure out how to just give it a simple security setup with a regular password instead of a hexadecimal string password.

Scott....I believe secret is enabled. I don't have a switch in front of me to try but I remember having to enable secret in order to get the password in hex once before. Try using the command no secret and then reset your privilege level password you are using. Try both at the main prompt and if doesn't work....give config t a try. This may put it in text. I can give it a try Monday back at work if you like. We use IOS version 12.2.

Good luck.

 

[Scott Helmke]

Re: Large scale concert production TCP/IP networking

Scott....I believe secret is enabled. I don't have a switch in front of me to try but I remember having to enable secret in order to get the password in hex once before. Try using the command no secret and then reset your privilege level password you are using. Try both at the main prompt and if doesn't work....give config t a try. This may put it in text. I can give it a try Monday back at work if you like. We use IOS version 12.2.

Good luck.

Thanks! Hopefully I'll have time to try that out this week. For winter there's still a lot to get done around the shop right now.

 

[Chad Young]

Re: Large scale concert production TCP/IP networking

Hey Chad (and others who may know), are you familiar with the old Cisco Aironet 1200 configuration? We use one for big shows, because of the antenna options. But every time I try to do anything with the security I end up needing to do a factory reset - it's a whole different animal from the little Netgear boxes I use for small gigs.

I am familiar with those APs and they can be fussy. If it is running the old VxWorks OS, get the tool to upgrade it to Cisco IOS. If you want some more detailed help, IM me. The good news is the 1200 series is a pretty good unit in the more modern g/n iterations.

My question for you would be: While concealing the SSID does not stop anyone who wants to find it, does concealing the SSID prevent the effect of almost DDoSing your (possibly consumer) access point(s) in an arena like situation?

Short answer: Sort of. Long answer: hiding your SSID keeps the casual Joe with a phone or tablet from trying to connect, and, yes, 1k folks on iPhones trying to connect to your network in the arena could degrade performance or make it impossible for legit users to connect - a spontaneous DDOS. A determined miscreant with the tools, skills, and intent to interfere with your wireless network will not be deterred.

I would be loathe to be 100% reliant on a 802.11 wireless signal for any mission-critical service. It is just too darn easy to render the spectra in question quite useless with very little effort. A $20 investment at Rat Shack and a little knowledge can make the 2.5Ghz and 5Ghz bands quite unusable.

In general, I have found internet access to be extremely valuable when I have needed a manual, tech support, or even the most recent FCC database for WWB. I put the wireless in a different subnet and then use the router to restrict access for all the audio devices and allow it only for the client computers.

A laptop with a 3G/4G wireless card is a Godsend for this sort of thing. I use a sprint 3G adapter and get decent speed. The newer 3g/4g MiFi are super handy. Just do not connect your mission-critical network to the internet.

If you give ip-adresses based on device type you can reduce broadcasttraffic with subnets. 192.168.1.x for wireless mics, 192.168.2.x for IEMs, etc.

This is not easy for the average sound guy to do and may require more advanced knowledge of routing and switching. Simple is best unless you want to add an IT guy to the sound crew.

 

[Helge A. Bentsen]

Re: Large scale concert production TCP/IP networking

I ran into a issue with poor network performance during show time, so I tried hiding the SSID. Network performance is slightly better, but my Android phone can't connect to the network, I guess it's because it can't see it. So when I need to use my phone as a remote, I have to unhide my SSID. Bummer.

Is there any better way to do this?

 

[Chad Young]

Re: Large scale concert production TCP/IP networking

Yes, there is. You have to manually add the WiFi network. Settings -> Wireless and Network -> wiFi Settings -> Add WiFi network. You will need to type in the SSID and key *exactly* as you have it on the WAP - it is case sensitive.

 

[Tim Duffin]

Re: Large scale concert production TCP/IP networking

If you give ip-adresses based on device type you can reduce broadcasttraffic with subnets. 192.168.1.x for wireless mics, 192.168.2.x for IEMs, etc.

There is no reason to use a 192.x class C network in a live situation. The 192.x is only used because it is not resolvable from an exterior domain. The reason to not use WEP specifically, is because the number of scripting repeated attempts at breaking into your network rely on the router to reject the login attempts quickly. Wired Equivalent Privacy does not have any way of limiting the number of connection attempts per second-- this causes a reduction in available slots for incoming valid connections. Only WPA and WPA2 can do this type of filtering (with WPA now disallowed in critical path networks), thus they are used for high sensitivity networks. The idea being that by the time the code is broken, the user has lost interest in acquiring the data. Hacking of the actual login passkey can only occur if there are multiple devices logging into and out of the router during the time that the hacker has access to listen to the incoming and outgoing data packets and use packet injection. For now,WPA2 passkeys are unbreakable if used correctly. As an additional level of security, it is a good idea to also name your access point with a random sequence of letters and numbers to stop android phones with "rainbow tables" from guessing your SSID.