5-minute troubleshooting. Can you hear me?

Here I want to start new article series called “5-minute troubleshooting”. In these articles I’m going to describe short simple cases which we solved very quickly using protocol analysis and which could take much more time if we use another approach. So, let’s go.

One day we received an IP camera from our customer with the next complaint: “No image, no ping, the camera just disappeared from a network”.
First, we checked whether the camera was receiving PoE (yes it was), and booted up normally (according to camera LEDs no problems were here also).
Next step? Let’s try to ping it using IP-address our customer provided to us. No reply (and no surprise actually).
At this point it would be nice to check port statistics and status to eliminate L1 failure:

1_switch

Camera is on port 2. The port is in “Up” state, autonegotiation is completed successfully. There is some traffic passing in and out (RX/TX counters are both non-zero). No bad packets, no collisions, no ideas what could be wrong instead of we’ve been provided the wrong IP-address information.
So, it’s time to look what is happening on the wire.
We arranged simple capturing scheme using SPAN:

2ip_cam

And captured some traffic:

3ip_cam

What do we see here? The camera has an IP of 192.168.88.203 like it should have.

Right after boot process the camera tries to resolve some standard host name using MDNS (packets 1, 2). This is built-in mechanism and it’s described in documentation. When it sees no answer (also possible situation), it tries to resolve an IP 192.168.88.202 using ARP (packets 3, 4 are ARP requests from camera).

Why does it resolve exactly this address? I’ll tell you. Our customer said that he has NAS device with that IP, and therefore camera has been configured to use it. Knowing that, we intentionally assigned 192.168.88.202 to our test client machine. Also that means, that camera config is still inside it and is usable.

So, our client replies to camera’s ARP request and all seems to be fine, but… after 6 seconds the camera asks the same question again. There is no other traffic between these requests, PCAP file is not filtered. Later in the trace client tries to resolve camera’s MAC using ARP (packets 9-14) but gets no reply at all. Strange situation.
Question: what do you think about trace file above? Hint: look at timing between first two MDNS packets and further ARP requests. Is it normal?
Let’s try to boot the camera in DHCP mode. We can do that by pressing one of the buttons located on camera’s case during the boot process:

4ip_cam

So, we see the next:
“Is there any DHCP server here? – Yes, I’m here to offer you an IP address! – Looong quiet time… – Is there any DHCP server here?”
The only reason is left. We see every packeton the wire, but they get lost downstream. Downstream? Wait, we’re mirroring exact link that is connected directly to the camera! But it does not hear incoming packets despite they’re actually present on the wire. There is something wrong with NIC driver, or with some electrical circuit inside the NIC. This fault doesn’t affect Layer 1 (physical), doesn’t affect sending part of it but also it does not allow the data to be received and (or?) processed by camera further at some stage. As this is just an IP-camera, not a server, that means we can’t check drivers, log into console, change NIC or do any further diagnosing.
End of the story: we sent the camera to manufacturer with the above description and eventually it’s been replaced because of “faulty NIC” problem.

Train your brain. Answer the following question.
1. Probably you’ve guessed that first trace file contains duplicate broadcast packets, right? What do you think will happen if you deduplicate this trace file using editcap utility with default settings? What settings would be better in this case?

Поделиться
  •  
  •  
  •  
  •  
  •  

Comments:

2 thoughts on “5-minute troubleshooting. Can you hear me?

  1. Nice article, Vladimir – though this kind of “one port SPAN” setup should not lead to duplicates – or did you mirror both camera and client PC port? If not, the switch is probably treating the monitor port (with the Wireshark PC) as a normal device port and sends the multicast/broadcasts there together with the mirrored packets. Normally, it shouldn’t do that. Or do you have any other explanation why the duplicates appeared?

    1. Hi Jasper,
      Only one port connected to the camera was SPAN source. You’re right, switch used “Wireshark PC” port as normal access port and SPAN destimation port at the same time.
      Therefore broadcast/multicast frames appeared there two times.

      Cheers,
      Vladimir

Leave a Reply

Your email address will not be published. Required fields are marked *