Out of hours callout compensation approaches (UK only)

[u/simbastears]

Hi Peers

I have been tasked with reviewing my organisation's technical out of hours response capability. I have experience of establishing and operating this function in other organisations but wanted to seek some wider perspectives - ideally from other UK peers.

I'm interested in

1) Do you operate a formal rota, if not, how do you provide an assured response to major incidents - e.g. an active cyber security incident.

2) Do you do anything different over the bank holidays/Christmas periods?

3) If you do operate a formal function, how do you compensate your team? For example, do you provide an annual salary supplement based on their participation on the rota, or pay per coverage slot, or is it just an expected element of the role with no additional compensation?

4) If an engineer is called, is compensation for the hours worked included in the rota participation compensation, or paid as additional overtime?

5) Does your rota include management or escalation coverage?

6) How do you balance skills coverage and fair division of standby available across your teams.

Thank you in advance for any replies. If you would rather privately message, I would be fine with that, any information is treated with confidence.

[u/poizone68]

I haven't worked in the UK, but I think these questions are quite general. I'll provide my unsolicited response in any case :)

1. Yes, formal rota is definitely the way to go. By all means let people swap slots or cover for each other, but it needs to be communicated in advance. This is especially important if your company has a NOC or Server Ops 24x7 team.
2. The main thing is that a strict calendar approach may make the rota quite unfair in some years, for example Christmas and New Year's could fall within the same on-call period. Split those weeks so that people either cover Christmas or New Year. Making someone lose out on both is not family friendly. Do the same with other holidays too, so that nobody loses out on both Easter and Christmas the same year.

3&4. I've worked with various models. In one job it was straight overtime for each hour plus fixed yearly add-on to salary. In another it was a daily rate for each day on-call, first hour of callout was covered by the daily rate, with the second hour onwards being overtime. However, managers who were on-call got a fixed salary addition with no overtime.

1. Managers should have an on-call rota separate from the engineers. Their role was mainly comms to leadership in the companies I worked in, or helped calling additional resources while the engineer on call continued working on the incident.

2. If the organisation is large enough to have several team competencies, ideally each team should have their own rota. Having e.g a Windows sysadmin on call when the active issue is on a Linux system may not make much sense, or letting the Windows sysadmin having to call the Linux sysadmin to get them to look at it is just wasting a lot of time.

[u/Dangerous_Plankton54]

Not UK but Ireland is similar work styles.

Always had a rota with engineers on for a week at a time. On call managers on for a month at a time.

Generally set day rate to be on call with time and a half pay for hours worked.

Main issues faced was that the on call engineers were out L2 engineers and experience varies widely. Which meant when I moved to management I was constantly called into out of hours whether on call or not. So making sure the engineers on all cam actually resolve the most typical calls you get would be a great achievement but having a senior engineer who doesn't mind being pulled in and also gets their time and a half is invaluable.

[u/HotTakeThenGo]

Scope the requirements more. What’s an “assured response” that’s acceptable?

A solid third party SOC/SIEM will stop most threats. Next, you need to “fix” things. Can “fix” simply be a backup restore? If so, you can tier this down to any tech and a proven SOP (which should include disabling the account/device entry points).

With just that, you have a 4 hour Incident Response and Recovery plan - in theory. But let’s go further…

Do you need all options all the time? Probably not. Most of my customers just need email and everything else needs to be ready by 8am Monday.

Let’s move email off to the side. How fast can one of your top 3 engineers recover a client from backups and do minor security repairs? I’m guessing that’s 4 hours or so.

Ignoring email, most clients can get away with one of my techs starting their Monday shift at 4am instead based on the above.

We’re optimizing but we can go further still. I don’t want to waste my tech in the name of “just in case.” Give them things to do. I’m leaving this part vague as there is no shortage of things you likely need that would benefit from the users not being there.

I have a tech I shifted entirely from 3am to noon. He’s almost entirely doing project work. I’ve covered the clients needs without raising costs or complexity. He loooooves this shift.

On emails, make BEC hard and recover easy. Use phish resistant MFA with device context, dial in Conditional Access policies, use LAPs and JIT, no standing admin access except a breakglass account that is heavily boxed, build honeypots and monitor them, automate what occurs when a risk ever happens (e.g. force reset the password) and dial in SSPR.

Your company might need more. My clients always think they do. There are 4 more shifts required to truly accomplish 24/7 and if they need to be experts too the cost will become astronomical.

I have extended hour contracts with some clients. We need to be profitable to staff for those and that’s accounted for. But I advise that same “profitability” standard for Internal IT teams too. We all need to justify our cost to the business with the same metric - money.

This is a cost/benefit analysis. I’m delivering the same benefit at a reduced cost. There’s no value/benefit for operations to function when it’s not required but the cost of buying a tech’s time is high.

Further - I am offering work life separation. The other model does the exact opposite. My techs and staff don’t work outside their routine hours. I can attract better talent because I offer something my competitors don’t - freedom of their own time. What’s the cost for that? I don’t know - but it seems to be at least a $10k year benefit in the USA.

You folks across the pond get much better time off in general and your culture allows for more downtime in the USA. So results may vary but I’d still recommend working the problem focusing on the true value of off-schedule operations (near $0), the cost of additional technician capacity (more than $0), and the price for compromising a tech’s personal time (bigger than you think).

[u/dariusbiggs]

Yes, formal.

Yes, rotation, depending on your staff turnover, ensure that the same people are not on call over multiple holidays, and across years. Don't let Dave work three Christmas and Easter periods for 3 years in a row.

Compensation for being on call, whatever is reasonable and industry standard in shift/on call work.

Compensation for the actual effort of callouts as overtime, with a minimum of X hours, even if it took 5 minutes.

Compensation for any travel required (they should have everything they need to do their work from anywhere on the planet, so a company cell phone (with a VPN if needed), with sufficient data, and a laptop with a VPN.

Release the schedule at least 1 but preferably 3 months in advance, be flexible and allow people to swap. Within reason.

Ensure you have a drug and alcohol policy in place for being on call. (ie. stay legally sober and competent to do their job).

Have an escalation policy and process in place in case they don't answer (shit happens, people get hit by a bus all the time).

Have a handoff policy and process in place if the on call people can communicate they got hit by a bus and are unable to remain being on call.

Have an escalation policy in place to get more senior engineers or other subject matter experts IF required (they will be need to be compensated too).

Management must always be included in the rotation and they get the worst schedules, and they're part of the escalation path.

Plan for the worst case scenarios, document the risks, create the policies and procedures, make them simple to follow. Minimize paperwork and bureaucracy.

Train staff on the processes and policies.

Post mortem the incidents the next available work day, after action reports help here as well as audit trails, logs, etc. Plan for fixes to prevent the issue, or automate the repair.

If it can wait until the next business day, it can wait. Not everything is that urgent. If it doesn't need to wake up an engineer for immediate actioning, then it can be an email notification and actioned during the day.

Audit your on call system, callouts, and who actually did the work. Make sure the schedule and compensation are reviewed and stay fair.

[u/Si0_x]

Most teams I've seen handle this with a flat standby allowance plus actual call-out hours paid as overtime or TOIL, because lumping it all into salary tends to burn people out fast once the incident volume creeps up. Bank holiday and Christmas coverage usually gets a premium rate or rotated so nobody draws the short straw two years running. The split between rota participation pay and per-call overtime matters more than people expect when you're trying to keep the rota voluntary rather than mandatory.