A client recently had an interesting issue. They upgraded their Primary Site and was in the process of updating their Secondary Sites (I know, I know, long story). While upgrading their Secondary's they noticed that they would remain in a "Link has Failed" state until about 8-8:30 PM. Very bizarre.
So, the first place we checked in troubleshooting this was the "Sender.log" to see what traffic was being sent to the sites. In the log we were seeing the following entries "Send request .... is skipped for site " and "All the existing addresses to destination site ... are closed".
So that's no good but its helping us narrow it down a bit. Its telling us that it wants to send data to the site but the links are closed. Well, after seeing those log entries combined with the consistency of a 8-8:30 PM kickoff for traffic to start flowing it all pointed to one place.......Scheduling.
After some questioning it seems a fellow co-worker made the changes to help reduce WAN traffic but forgot to inform his other co-workers of his changes. These settings can be found in the SCCM Console by going to "Administration - Hierarchy Configuration - File Replication" and then going to Properties of a Site.
So now we had an answer. We modified the Schedule to allow "Medium to High Priority" traffic during those hours, and wide open during the hours of their upgrade so they don't have to wait until the next day to ensure the link status changes to Active.
Now this was only one site we modified but the client had close to 100 sites and the technician couldn't remember which sites he made the changes on. To save us having to check every single site we used the table "dbo.SC_Address" in SQL to view the sites with modified Schedules and Rates.
As you can see in the above screenshot from SQL Management Studio, we can see the Destination Site Code, the Schedule, and even the Rates if those are limited also. The Schedule is sorted Sunday through Saturday by 48 individual numbers for each half hour increment, and each day is separated by a comma and then another 48 individual numbers representing each half hour increment. In the screenshot we can only see Sunday but its all "1's" telling us that each of the half hour increments are 100% open, if there were any "2's" then that would tell us that its limited to "Medium and High Priority" traffic only. Also, in this screenshot we can see the rate limits were also set for a percentage of the available bandwidth at certain hours of the day. Similar to the Schedule, this has a number for each hour (24 in total) separated by a comma. In the screenshot we can see that site SE1 has its rate limit to 65% of bandwidth till 0800 hours and then its 100% (unlimited) for hours after 0800.
We took this view and exported it to excel so we can better see the data. What was nice about this is that it not only showed the limits under "File Replication" but also the limits that were set on individual "Distribution Points". Yup, the technician not only set up limits under File Replication but he also configured limits on individual Distribution Points which we were not aware of. The view from table "dbo.SC_Address" helped us quickly identify and document what limits were configured. In the console you can find the Distribution Point limits under "Administration - Site Configuration - Servers and Site System Roles" and going to the properties of the Distribution Point Role.
For some additional information you can view the following tables:
dbo.SC_Address_PropertyLimit
dbo.SC_Address_Property
*WARNING* It is not recommended OR supported by Microsoft to edit the SQL Tables. DO NOT EDIT THEM. Go into the SCCM Console and make your edits. This was only shown so you can grab a quick excel spreadsheet to help identify. You will break your site if you edit this table directly so DON'T !!!!
No comments:
Post a Comment