r/BestofRedditorUpdates • u/JoySubtraction I will never jeopardize the beans. • 2d ago
CONCLUDED The Server Upgrade
I am not the OOP, OOP is u/jbanelaw
Originally posted to r/talesfromtechsupport
"How Did You Get This Number??!!!" - "You Realize You Are 'On Call' Right?" - posted 2021-11-13
Despite almost doubling my rates for new clients (and that rate was not bargain basement either to begin with) the work keeps on coming in. I know this wave isn't going to last forever, so am taking full advantage of the shortage in the labor market while I can do so.
This means an average work week for me in around 70-80 hours, but again, I'm getting paid by the hour so I don't really care it is pretty much sleep, eat, work, repeat. I'm starting to feel like Scrooge McDuck and his money bin with the good times rolling. I just need to make sure to put enough away so when the bottom falls out of the this labor market and the leads dry up, I've got enough to go lean on for awhile.
Anyway, here is a quick one that I just experienced. Enjoy.
The Scene
"Hey IT Guy! When can you get to that new server we were talking about???" the manager screamed over at me while I was tending to a network issue across the cubicle farm.
"Yeah, it is going to be sometime here..." I say looking at him annoyed.
"Well, I need it up by the weekend if possible. Feel free to do any of the work at night if that helps."
"Actually it does....I might have some time this weekend...Who is my technical contact after hours"
"Great. We really need that up by Monday. If you have any hangups just ping the on-call tech. They are usually around on the weekends anyhow..."
Saturday Night
I remember when I used to have a social life and hated it when a job took me for weekend work. But, that seems like ancient history here approaching year two of a pandemic. Now a Saturday night is just like any other evening. This one though I was about four cups of coffee in and knew sleep was not in the cards. So over to my client's office I go to finally tackle that server I had been promising him.
Should be a 4-5 hour job to get it into production. Nothing special here. Just decommission the old one. Restore a backup. Optimize some stuff. And boom I should be done. Home by midnight. Client will be happy and I'll get a few more hours to bill this week.
First things first, I show up and go to the location where the server was supposed to be. It is just gone. I look around in all the usual places - nothing. Check the not so typical places where some of the in house guys will put stuff - nothing. Hmm....stumped but not wanting to go home, I remember there is an on call tech. I look through my email and see the announcement about who it is this weekend and give his number a call.
Rings through to voicemail. Sounds like a personal phone number, so I leave a message. Call back time says one hour or less in the protocol so I hunker down doing some odd tasks waiting for the return call.
An hour later I don't have a call, so I ring the number again. Once more through to voicemail. I leave another message asking if this is not the right person to please let me know, but I have a job that needs to be finished tonight so give me a call back ASAP.
"How Did You Get This Number!!!!!"
I went to the bathroom and left my cell phone on a desk. When I come back there are 5 missed calls. Thinking something is wrong I page through the call logs to see it is the same number 5 times in a row in about 7 minutes. Hmmm....so I give it a call back thinking it might be the on call tech.
Firs thing out the mouth of the guy who answers - "How Did You Get This Number!!!!!"
"Ummm....." taken aback.... "You called me five times in a row just now...."
"No I didn't....." "Put me on your do not call list!"
"Hey look you called me figured it might be a return call for XXXXX company, I'm a contractor doing some work tonight...."
"You just can't go calling people in the middle of the night because you want to work!"
"Ummm according to the office email you are the on-call tech...."
"I told my boss I don't do call anymore!"
"Well, tell that to the office admin who put down your name....."
The guy is clearly really annoyed and this is a new client. I don't want to get a bad reputation, so I just try to make nice, but he would have none of it.
"You contractors are all the same...." blah blah blah as he dumped on contractors for five minutes.
"Look guy I just need to know where the server might be. Someone moved it today and I would really like to just get it up tonight...."
"This isn't why we have an on call person on the weekends. It is for emergencies only and you wanting to work ungodly hours on a Saturday night is not my emergency!"
At this point in time, I just want an exit strategy.
"OK got it. I'll just let my contact know that we couldn't locate the machine and see what he wants to do on Monday..."
"Got that right bud. Don't call me again unless there is an actual emergency!" Click.
Emailing the Boss
As a consultant I know it is best to keep EVERYTHING in writing if at all possible So I start an email to my contact who is the site manager:
"Hey Manager Guy - I was at XXXXXX company just now to get that server deployed and the physical machine is missing. I contacted the on call tech (if you remember you told me to reach out to them with any problems) to see if they had any insight into its location. Unfortunately, the interaction I had with him was less than positive and I don't think I'll be able to locate the server to get it up by this weekend. Let's talk on Monday. Sincerely - IT Guy."
Figured that was about a diplomatic as I could put it even though I was mildly annoyed to have wasted at least 2-3 hours that night with the whole drama.
I didn't hear back for the rest of the weekend, but figured he might just be waiting until he saw me on Monday morning.
Monday Bloody Monday
Turns out the site manager didn't email me because he was livid at the on call tech. He and that guy had a history of duking it out over on call assignments and other office related politics. So, the manager was taking full advantage of this situation and set up the tech to be fired Monday morning.
Apparently it was "fireworks" galore, but unfortunately I got there about 15 minutes after a few tables got flipped and the police had to be called. And also apparently everyone hated this guy for a long time (I wouldn't know why.....). Word got out that the tech was sacked mostly because I "complained" about him. Now I have people coming up to me thanking me profusely for getting rid of the guy like I actually fired him.
It was an odd way to curry favor with a new client, but hey if it gets me more work I'll take it. Also the random people treating me like some sort of hero, seeking me out to say "Thank you IT Guy" is icing on the cake.
----
The Case of the Missing Server - posted 2021-11-18
You all thought I was going to leave you hanging there, didn't you? Don't worry The IT Guy would never do that to a bunch of strangers on the internet.
So, just to review, the guy who was supposed to be on call got fired Monday morning for failing, officially, performance issues, but unofficial, because he was a jerk too many times. I show up to a round of "high fives" because people think I advocated for getting the guy fired, when it was instead the site manager who had a beef with him. Doesn't really matter now. It looks like actually no one who currently works there actually likes the guy and it is like I just landed a big giant house on the big bad Witch of the East. Whatever, I'll take it.
But, that leaves me, your friendly IT services consultant, still with a job to do....
"When can you deploy that server," the Site Manager throws at me in the late afternoon.
"As soon as you can find it...."
"Yeah, no one seems to know where it might have gone. Can you check the (fancy hardware location application - going to be vague with this one for corporate reasons) logs and see if it has any history?"
"Ummm, sure wasn't aware you had that kind of visibility on the equipment. Where can I get access?"
"I emailed corporate security to have them create you a login and you should be able to find the machine profile by doing a search for the asset tag."
"OK...I've never used (or really even heard of this system) but I'm sure I can figure it out...."
"Yeah IT Guy I'm sure you can...."
Location, Location, Location
OK I've got to be a little vague here because what I will call an asset tracking system is FANCY and I don't want to even hint at anything that might touch a non-disclosure. I've never seen anything like it. Looks custom to a capital C and like someone high up at one point in time had some OCD about hardware, corporate espionage, or security, or something then threw a ton of money at a solution. Anyway, trust me. This thing is NICE.
So, just as the site manager said I get my login credentials, pop the asset tag into a query, and poof the whole profile is right there. Has a history of checking in on the internet and hey look there even GPS coordinates. This ought to be easy as long as the GPS is accurate...
"Hmmm....those coordinates aren't anywhere close to any of our offices...." I think to myself.
Looking at Google Maps, it sends me to a residential section of town about 30 minutes away.
"That can't be right, I can see the delivery confirmation here in the profile. This isn't on the back of some truck still out for delivery....." I continue thinking as I mull over the possibility.
The GPS location is solidly in a house, so I do a quick property tax record search to see who might be the owner of that address and poof it all makes sense.
"I'll be damned if this is right...."
The address comes back to one owner - Mr. I Got Fired This Morning.
Quickly I think my options are:
- Call the Site Manager and tell him I think the server has been stolen
- Call Corporate Security and make a report
- Hey I have the guy's number from the weekend, I can just give him a call and see if he will give me the server.
Yeah I know I should have done option #1 or #2, but I'm feeling frisky for some reason. Figure might as well text him and give him a chance to give up the goods. Guy has already had a bad day, last thing he needs to do is catch a theft charge on top of it all. Maybe he had a legit reason why he took it home....
"Hey This is the IT Guy..."
So I start a text.
IT Guy: Hey, this is XXXXXX. You yelled at me this weekend when I called asking if you could help me locate a server. Security software is showing it at your house. If you have it for whatever reason just drop it back off at the office. All I want to do is get this thing in production tonight.
Jerk Guy: Fuck you. I don't have shit.
IT Guy: Fine by me. I'm going to report it to corporate security and the police.
Jerk Guy: Wait, wait.....I might have something but I didn't steal it. Had it to do some work and needed....(long, desperate explanation here)
IT Guy: Look, I don't care why you might have had it. I just need to have it and tonight. If it appears and everything looks good, I don't care how it shows up.
Jerk Guy: OK, so if I drop it off you are fine with this just going away.
IT Guy: Look I'm not going to lie if asked but in my mind all that people need to know is that the server has been properly installed.
Jerk Guy: I'll drop it off (location at side of building) after hours tonight.
(OK before you security guys jump all over me. The thing was going to get completely wiped anyhow and I made sure there was no tampering. This guy was not some kind of criminal mastermind looking to defraud the company. I think he just had an expensive piece of hardware and after getting axed probably was planning on keeping it.)
About two hours later.
Jerk Guy: Server is by the side door. I kept my part of the bargain, please keep yours.
Sweet Irony
In the end, the on call guy still found me the server and ended up personally delivering it. All that it cost him was his job (which was probably in doubt anyhow since he was a jerk). There was something wholly ironic (or maybe that isn't actually irony - Alanis Morissette anyone?) about him ultimately finding it for me. But, I digress....
Server was still in the box, sealed. I spent the next few hours getting it up and running. Emailed the site manager that night to tell him the job was done and was just going to leave it at that unless asked for further details.
The next morning:
"That was some quick work IT Guy, where was the server?"
"I found it by the side door."
"Huh....that is strange. Maybe one of the guys moved it thinking it was something else."
"Yeah....who knows. Anyway job is done. What is in store for the rest of the week?"
"Thanks IT Guy. I've got a few more projects for you to tackle over here...."
Reminder: I am not OOP.
388
u/SmartQuokka We have generational trauma for breakfast 2d ago
Why did OOP do him such a favour, OOP put their own ass on the line for someone who would probably throw him under the bus in a heartbeat.
284
u/Sad-Stop-2401 2d ago
Time. Plus as a contractor you don't want to be the guy who always finds shit. Not to mention, you want to paid. If he just bills 2 hours or 3 hours he is not going make that much money. However if he finishes and bills entire hours, he will make money.
202
u/DMercenary 2d ago
Yup. A difference between "Fuck now I have to contact Security and the rest of IT and the manger and this is going to take forever." vs "Give me the equipment and we never cross paths ever again."
37
u/SmartQuokka We have generational trauma for breakfast 2d ago
And if the truth comes out OOP is in the firing line.
94
u/Sad-Stop-2401 2d ago
Actually its worse. He may come in line for a lawsuite. He is a contractor. So firing is not that big of an issue but reputational damage is.
That being said, if the turth comes out, lots of heads will roll. Who allowed the server to go missing? Why after having CIA class asset tracking system, no one noticed that server was missing? What else is missing? So for a time he would be fine and hopefully by then moved elsewhere.
1.2k
u/desert_jim 2d ago
Wow. I would have not gotten involved with douche canoe after the firing the way OOP did. I would have just notified the client that the hardware wasn't on site and where it was. No way I'm covering for douchebag. I wouldn't be putting my job on the line for them.
434
u/MichaSound 2d ago
Yeah, OOP is an idiot - if that guy stole one piece of hardware, he probably stole a bunch of other stuff as well.
Years ago I worked with a guy who was fired because he was ordering AV equipment to our department (not unusual), and selling it on eBay. They also discovered he’d clogged our on-site host server with over a terabyte of porn. I suspect he was running an adult site, using our storage (part of our business was hosting websites for clients).
144
u/KaziArmada He's effectively already dead, and I dont do necromancy 1d ago
Yeah, but OOP is a contractor. If that guy stole other shit, that's the clients problem, not 'Mine'. I'm here to do one thing and GTFO, and if you can't track your own shit? Too bad, I don't care unless I'm getting paid.
57
u/GetOffMyLawn_ You underestimate my ability to do no work and too much Reddit 1d ago
Oh have I got a story...
I worked for a company that had multiple sites around the country. Our division got taken over by a different division and their IT department thought we were a bunch of know nothing dorks and were very hostile to us. Especially the one asshole and his manager thought this guy walk on water and the sun shone out of his ass. Yeah no, the guy was a bullshit artist. So the manager thought we were dorks too. The manager knew nothing about the type of servers that were being managed so he was easy to bullshit.
Anyway, bullshit artist was also a shit human being to the other guys he worked with at his site and they hated him too. He thought they were also know nothing dorks. Yeah, never make enemies and never underestimate them.
So bullshit boy bought himself a big fancy workstation, back when they cost $40K, and this was in 1990s dollars, plus numerous hard drives for it, also expensive. All under the pretense of using it as a system management server for all the other servers on the site. Of course manager didn't understand any of it because he was blinded by the sun shining out of this bullshit boy's ass.
Bullshit boy also set up a bunch of dial up phone lines for dialing into and out of the site. Because he needed to be able to dial in from home for emergencies or whatever. Also not cheap in 90s dollars.
One evening one of the abused underlings is walking past bullshit boy's server and sees porn on the screen. Now this division was in the midwest and they were very very uptight. They used to freak out because everybody in the NJ division swore like sailors and wore shirts that weren't white. Straitlaced and uptight would be an understatement.
So underling realizes this is his big chance. He finds out that the workstation has been configured as a dial up BBS for porn. That's why it needed all the big disk drives and the phone bank. So underling carefully documents all of this, even goes to the phone department to get the bills and call logs. Backs up the whole system to tape and makes a lot of printouts. Puts together a 3 ring binder full of evidence and marches it down to HR. HR didn't even know how to spell computer but they knew what one was by the end of the meeting.
So big meeting was called with bullshit boy and his manager. His manager was a major prude and the word is the manager's face was absolutely purple with rage in the meeting. HR pushed the binder across the table to bullshit boy and said, what is this? Bullshit boy said, oh hey well I'll reimburse the company for costs. That was the extent of his defense. He was fired on the spot. Walked out to his car by security.
9
u/Fredredphooey 12h ago
I worked IT at a large and well known ad agency. Some moron thought that having p*do pr0n on his work computer was a good idea. His wife couldn't find it that way, right? Smh. He was walked off-site immediately.
65
u/QuesInTheBoos **jazz hands** you have POWWWEERRRSSS 2d ago
Not to mention they could have kept denying it and sold it to someone before the "deadline"
18
u/FireStorm005 1d ago
You must have missed the line where OOP said he was calling corporate security and the police. OOP was perfectly willing to let authorities handle it if the guy didn't return it, the guy just realized it wasn't worth getting charges.
18
u/HaggisLad Drinks and drunken friends are bad counsellors 1d ago
This is what occurred to me, just disable the tracking and move it then you might be able to keep it
1
u/Outrageous_Guard_674 15h ago
Except the tracking logs would still show where it last was. OP said the system was fancy.
5
u/maxdragonxiii 11h ago
yeah. maybe its because im a woman im like whoa no contact the company tell them hey located your server in the guy who got fired, get it back thank you.
2
68
u/Phanimazed 2d ago
IT stories on here are always so stressful to me. I could not live like this.
28
u/anubis_cheerleader I can FEEL you dancing 2d ago
Agreed. Thank goodness they can live like that, because otherwise think of all the online banking, medical records, and many, many other things we would not have without their hard work!
49
u/KiloJools It's just pharmaceutical-grade banana bread 2d ago
It's so much manufactured urgency and anxiety for ultimately nothing. And of course, if for job is fixing things, everyone you deal with is in a bad mood because things are broken. So yeah, it's stressful as hell and usually totally inappropriately so.
22
u/Jhamin1 The murder hobo is not the issue here 2d ago
It's manufactured urgency until your bank account isn't accessible, or your sibling is going into surgery and the doctor can't get to their medical records.
I've been on call for both.
A lot of stuff is made more of than it's worth, until it isn't.
7
8
u/drunkenvalley I beg your finest fucking pardon. 1d ago
Sure, but it's still often manufactured urgency - far too many of these issues are created by manufactured timelines that don't make sense, and the solution is crammed in with too little QA, etc, all because But I Want To Show It At The Conference urgency.
2
u/GetOffMyLawn_ You underestimate my ability to do no work and too much Reddit 1d ago
But most of the time it's not mission critical. It's just annoying stuff.
1
29
u/OpheliaRainGalaxy 2d ago
I've got a buddy who works in IT security and is a near total recluse. We meet up once a year for Pride, and it's always super unfortunate when his on-call rotation happens to land on that day. Can promise that dude is not sweating an extra couple of buckets just to look important hauling that laptop bag around in the summer heat.
22
u/shmoo92 cat whisperer 2d ago
At this one pokemon go! event, I wound up hanging out with a friend who wasn’t exaaaactly the on call tech guy but was in a role that more or less amounted to the same thing.
He had to plunk right down on the side of the circuit with his laptop and two cellphones three times that afternoon. the breaks worked out great for me too—I was getting over an awful cold and the chance to sit guilt free for 15-20min was lovely!
8
u/altaccount_28 2d ago
God it sucks to have to remember to take the damn laptop everywhere. Kids soccer game, having cigars and drinks with friends that laptop better be in your car.
8
u/hey_nonny_mooses 👁👄👁🍿 2d ago
Just finished an IT go live and have been rethinking my life’s choices after all this stress.
386
u/mrdaimler retaining my butt virginity 2d ago
Though the writing style was a little cringey, the whole tale was well told. Only worked on one side of IT (mostly ticketing and easy "please restart your device" projects) but hearing tales from the technical side is always a great read.
173
u/YeaRight228 the Iranian yogurt is not the issue here 2d ago
Last week, IT at my work wanted to add 5 IP addresses for server upgrades or whatever idk.
So they called Verizon and asked for 5 new IP addresses. They said "sure" and promptly replaced the 5 IP addresses with 5 new ones.
The entire network instantly crashed and required an overnight contractor to get everything fixed.
113
u/OpheliaRainGalaxy 2d ago
Huh. Ya know, for some reason it never occurred to me that business-to-business services would be just as crusted in enshittification as what we-the-customers deal with as random nobodies.
35
u/nsmon 2d ago
It's mostly held together with duct tape
15
15
u/altaccount_28 2d ago
Whats fun is finding out the 2 different major communications companies you ordered circuits from contracted to the same last mile provider who buried the fiber lines in the same conduit.
Care to guess how we found that out.
11
u/undertow25 1d ago
Ah yeah, "carrier diversity". Or as we call it on the European side of the industry, American diversity :-) It's cheaper for a reason.
57
u/LittleMsSavoirFaire I’m turning into an unskippable cutscene in therapy 2d ago
We recently moved a whole warehouse and I told everyone "you think the issue is going to be lost inventory, lost shipments or getting the deliveries to the right place, but I assure you, it's going to be the ISP trying to figure out where we are and how to wire an 80yo warehouse."
I personally spent at least five hours on the phone and the techs sat for at least two hours looking at the router waiting for it to show signs of life.
21
u/IntelligentLake 2d ago
At least now they'll get charged for 10 addresses (or 50) despite only having 5, but that's Verizon math.
8
u/YeaRight228 the Iranian yogurt is not the issue here 2d ago
We were supposed to have 10 now, I think. Operations was on the phone with Verizon support for like 3 hours.
I'm in AP/AR so I don't really know all the details
11
u/INeedANappel 2d ago
This is a case of language clarity. They should have asked for 5 additional addresses. It would have removed any chance of ambiguity.
It's like when you're in a car giving directions to a driver and tell them to turn left at the next traffic light. They get to the light and ask, "Turn here?"
You say, "Right."
So they turn right.
You should have said "Correct" or "Yes."
12
u/nicktheone 2d ago
This happens because everything in a company becomes a game of telephone.
7
u/YeaRight228 the Iranian yogurt is not the issue here 1d ago
Especially at a telephone company
7
u/MrHappyHam Hyuck at him, see if he gets a boner 1d ago
Guess you can't blame them for acting in their nature
3
76
u/DMercenary 2d ago
Things like above dont happen often but they do happen. My work in the past had to fire a guy and sue him because he was taking home old devices.
And then selling them.
24
u/djseifer Last good thing my mom made was breast milk -Sent from my iPad 2d ago
I'm pretty sure I got a GeForce 3 Ti500 from the IT guy at one of my old jobs that way.
13
u/Warm_Score1176 1d ago
When we got bought out by a big company we got a lot of new devices to put us in line with their other offices. We were told some of the devices would be wiped and we could take them home (things like old phones, 1st gen iPads).
Junior IT guy goes around with a box to collect them to wipe. We never see them again and think the promise of taking items home was a corporate lie to make us new employees feel good.
Yep he took them and put them on ebay.
7
u/Wokuling Editor's note- it is not the final update 2d ago
A "friend" of mine was caught doing this. He's exactly the type to do it.
2
184
u/LittleMsSavoirFaire I’m turning into an unskippable cutscene in therapy 2d ago
All the IT story subs are like this. I'm just glad they have mostly dropped the $$ANGRY_GUY naming convention
84
u/JustAnotherLurkAcct 2d ago
But how else will they let us know their scripting creds if they don't name characters like variables?
26
u/miserablenovel Please kindly speak to the void. I'm too busy. 2d ago
Yeah that's old old. Blame BOFH for that one
17
u/TyrconnellFL I’m actually a far pettier, deranged woman 2d ago
That’s ancient to the ancients. The PFY is now a middle-aged father of two with a sore back. His oldest is probably in college by now!
10
6
u/SnowingDandruff crow whisperer 2d ago
I've noticed that too and I've always wondered why. Could it be that sometimes the people who work IT are also going to be gamers, tabletop players, and/or anime (or other such media) enjoyers?
23
u/weaver_of_cloth Tree Law Connoisseur 2d ago
It's more like a small-ish percent of us old-timers AREN'T into one or more of those things. The younger ones aren't quite as nerdy.
7
u/KaziArmada He's effectively already dead, and I dont do necromancy 1d ago
The younger ones, or the super old ones for the most part. Even then the older guys usually got some flavor of nerd they're into.
...Unless they really like Jesus. I had to throw a tech out of my imaging lab once because he decided he wanted to have a theological debate with me and wasn't accepting 'no, I don't talk about Religion or Politics at work'. I just wanted to image my god damn eight billion laptops.
9
u/gdidontwantthis 1d ago
My IT coworker who really really liked Jesus got fired inatantly after threatening folks from CorpIT for touching "his" server room. For a couple of years after I watched the news for his name because Jesus was the only thing keeping him from being a serial killer.
3
u/SnowingDandruff crow whisperer 2d ago
What? Really? I live under a rock (despite being on the Internet so much) so I had no idea. Thanks.
58
u/Phanimazed 2d ago
Credit where due, this guy kept things to relevant details a lot better than some past IT stories. I don't feel like you would need to shave off paragraphs to make it coherent, etc.
15
u/FlyingAce7 2d ago
As a non-IT gal, I found it easy to understand what was going on.
14
u/Machine-Dove surrender to the gaycation or be destroyed 1d ago
As an IT gal, I guarantee at least half of this story is complete nonsense.
2
u/FlowerFelines I am old. Rawr. 🦖 1d ago
What about it is unbelievable?
18
u/HelenGonne 1d ago
I think he's lying about the server being untouched and sealed inside its box and never opened. And I think the reason he's lying is because he was backpedaling after people started calling him out for not doing right by his client company.
If they are that serious about their Asset Management system, then they are also far past the point of being serious about security to want to know if that machine has ever been outside the expected chain of custody. It's not for him to make the call on what should be done with the machine if that is the case. They have a whole process that deals with that issue, and it is not for him to take it upon himself to decide.
Obviously it was powered up at the guy's house in order for the asset management system to be able to track its location. He knew that when he decided to tell that part of the story. If it were for some reason shipped with a powered up location tracker in the device or package, he would have said so when he first started telling that part of the story.
So he only appended the bit where it was sealed and never opened after people started calling him out, by my reading.
10
u/Machine-Dove surrender to the gaycation or be destroyed 1d ago
Yea, exactly. If you have a tracking system that sophisticated, you also have several layers of controls in place way before you ever get to that step to keep equipment from leaving the site, or to at the very least send up a few alerts if something moves to an unexpected place. If it's a network-based tracking system rather than something physical, there are technical reasons why it wouldn't work the way it's being described. Mainly that networks aren't magic and usually there's a wee bit of configuration to do before something is going to actually connect on a new (fired guy's home) network. Even a VPN would require initial connection to the local network.
I work in a place with serious asset tracking, and nobody is getting out of the building with so much as a defunct monitor, much less an entire server. That doesn't stop things from getting shoved into closets or stuck in weird places and generally driving the people who do inventory completely batty, but there's absolutely no doubt that the devices are still in the building. Somewhere.
I can definitely see the on-call guy losing his shit, maybe getting fired over it, but the rest is exaggeration at a minimum, outright fabrication at worst.
(I once spent three days hunting down a server that wasn't where it was supposed to be, wasn't on the network as anything remotely recognizable, and generally seemed to have been received, logged, verified, and then dropped off the face of the earth. Ended up finding it still boxed up, under some guy's desk. He was using a $150k specialized server as a footrest.)
9
u/artipants 1d ago
Obviously it was powered up at the guy's house in order for the asset management system to be able to track its location. He knew that when he decided to tell that part of the story. If it were for some reason shipped with a powered up location tracker in the device or package, he would have said so when he first started telling that part of the story.
Yep, this was the point at which I called bullshit. The rest was bizarre but theoretically plausible.. I've worked in a place where the IT department was essentially the wild west and previous workers and managers had some really fancy solutions to issues that no one else cared about. An over the top asset management solution that doesn't actually lead to action would've fit right in.
But I'd really like an explanation of how the sealed, never powered on server could register a location and "Has a history of checking in on the internet."
3
u/FlyingAce7 1d ago
... that makes sense. I had commented that I found it easy to understand the story because many IT stories overdo it with the tech lingo and I end up missing half the plot; however, I'm now seeing the inconsistencies you mention.
3
u/FlowerFelines I am old. Rawr. 🦖 1d ago
Ah, I hadn't clocked that inconsistency. Everything about the behavior of the guy who got fired reads like "Yeah, I know IT people who've dealt with this kind of thing" but that bit is pretty telling that something is up somewhere.
8
u/Dramatic_Buddy4732 It's just pharmaceutical-grade banana bread 2d ago
Have you tried turning it off and on again?
21
u/BringMeInfo Not a cult leader, a cosmic architect 2d ago
A little? I stopped reading halfway through to see if the style bothered anyone else as much as it bothered me.
7
u/Quasirandom1234 it’s solving a puzzle only crabs understand 2d ago
It’s the standard style of all the IT story subs. I’ve learned to read past it.
16
140
u/Cloudy_Retina It's just pharmaceutical-grade banana bread 1d ago
Server was delivered to the side door still in the box, sealed.
But same server was regularly online to ping this super duper asset management software with its address.
28
132
u/Kindly_Zucchini7405 2d ago
I can't imagine why the on-call guy was so unpopular. He has such a professional attitude.../s
2
92
u/panderp It's just pharmaceutical-grade banana bread 1d ago
So the server was still sealed in box and had GPS, which was on, despite the thing never have been used and still in box...
And this is at a time when the AirTag had only just come out.
I may have been out of the game for a while but this sounds a little.. hmmm.
23
u/R_V_Z 1d ago
Plus from a security standpoint this is no different than plugging in a USB drive you found in the parking lot into your work computer but saying "it's OK, it was still in the packaging." A company asset left custody and was in the hands of a hostile ex-employee. That's not IT's issue to solve; that's your security department contacting the authorities territory.
3
5
u/WarlockSoL being delulu is not the solulu 11h ago edited 11h ago
My "kind" interpretation (that is, giving them all benefits of every doubt) is that the GPS tracking wasn't live but maybe a log of where the package was delivered (assuming it wasn't "stolen" but actually delivered to that guy's house through their inventory chain for some reason). But I dunno, because you're right that if it's an actual air tag that kind of makes it harder to believe.
72
u/the_pressman 2d ago
How would they have GPS tracking on a sealed new product?
99
u/BraxtonFullerton 2d ago
When you have a 5-figure machine in a box it's not that uncommon that they'll toss a tracker in from the manufacturer, that way if the delivery company "loses" it, they're not liable and are able to resolve a non-delivery dispute fairly quickly.
50
u/theWindowsWillyWonka 2d ago
And it said the asset tracker showed a history of checking in on the Internet. This sealed box is networked?
41
19
7
16
u/altaccount_28 2d ago
I can answer that for you. If you are big enough client for a Dell or HP they can preload your image onto devices before they are shipped to you. Dell calls it their Dell Imaging Services. So you would see your image boot up check in with the mothership at the office.
I suspect the box was not as sealed as it might first appear and the IT guy plugged it in at his house and the computer used IP geolocation to report its general location.
8
u/Machine-Dove surrender to the gaycation or be destroyed 1d ago
That would only work if the device was preconfigured to use Fired Guy's network gateway etc. An imaged machine would boot, but it it's not connected to the corp network it isn't pinging shit.
2
12
u/_DoogieLion 2d ago
They wouldn’t. Bullshit story.
6
u/Machine-Dove surrender to the gaycation or be destroyed 1d ago
Yea, this ranks high on the "things that didn't happen" scale.
14
u/DamnitGravity 2d ago
Dude repackaged and resealed it. Not very difficult to do.
2
u/Mayonaigg 18h ago
The guy who was stealing it opened it to install a gps tracker for the company asset management and then resealed it? Do you read what you type?
8
u/robertbieber 2d ago
Putting GPS trackers on everything ain't cheap, either. It's really hard to imagine someone shelling out that kind of money on hardware and monitoring subscriptions for equipment that's meant to be installed in a rack in a secure building and then sit there for years
11
6
33
u/Logvin 2d ago
Why in the world would they give a contractor access to their hardware management system for the sole purpose of looking up the location of a single asset?
9
u/bubbleteabob 1d ago
At a guess, manager was sick of all the drama going on, didn’t want to have to justify losing track of an expensive piece bit of machinery on top of that, so instead of asking anyone who already had access he just ordered that the IT get access?
OR he suspected the fired employee had it and wanted someone outside the company reporting structure to make the issue disappear as quietly as possible?
31
u/captain_borgue I'm sorry to report I will not be taking the high road 2d ago
IT Guy being all "I found it by the side door" was some peak r/technically_correct.
Also right kind of him to not hang Jerkface out to dry. Dude could've easily landed himself in prison if IT Guy hadn't been so magnanimous.
18
u/Boeing367-80 2d ago
In this case (might not always be true) magnanimity got him what he wanted - the server lickety-split. Had he gone the police/security route, the server might well have been tied up for some time.
13
u/FloralStormVale 2d ago
That’s the kind of technicality that actually helps someone instead of just winning an argument. He stayed truthful while giving the guy a chance to avoid a potentially life changing mistake.
51
u/cdrfuzz 1d ago
Amazing that OOP really has no idea how fucking boring he is.
17
u/letters_numbers_and- 1d ago
The moment I saw it was written in "I'm being clever" tones, I knew it was going to be a rough read.
3
u/AffordableGrousing 18h ago
Yeah, this was a lot of words for not a lot happening. The missing hardware was at the disgruntled IT employee's house -- shocking!
Also, while the fired "on-call" guy was definitely a jerk about it, I wouldn't want to be contacted on Saturday night for a non-emergency either. It seemed like doing the exact same work on Monday morning would have made very little difference. The manager should have verified if or when Mr. Sort-of-on-call would be available over the weekend before telling OOP to move forward.
27
u/skoltroll I can't believe she fucking buttered Jorts 2d ago
A solid that didn't need to be a solid. Good guy, OOP.
30
u/WelshWickedWitch 1d ago
I don't understand. Surely the server would have to be "on" in order for the GPS to work?!! Therefore it would have to be unboxed and not brand new, sealed in its box?!!
17
u/Cloudy_Retina It's just pharmaceutical-grade banana bread 1d ago
And then the entire company gave me high fives and carried me on their shoulders to an all you can eat pizza party!
24
u/Matt_NZ 2d ago
I'm a bit dubious on the GPS tracking. Maybe their workflow is to add something like a AirTag to every shipment that comes in, but why would he ponder that it might still be out for delivery by the courier tho? I'm not aware of any OEM adding GPS tracking hardware into shipments either.
26
u/tdeasyweb 2d ago
No don't worry, the name of this magical third party software that has the ability to activate GPS on a factory sealed OEM item that the previous IT guy installed and activated in their own home, is the only piece of information that OOP does not want to reveal for NDA reasons.
I'm surprise anybody who has worked in sysadmin or really anything IT related bought this story.
5
u/kenyafeelme 1d ago
I already started skimming because the writing style seemed like a person who enjoys telling stories for attention. You’ve convinced me to never go back and read the rest.
11
u/SirLoremIpsum 2d ago
I'm surprise anybody who has worked in sysadmin or really anything IT related bought this story.
On call guy was right.
This was not an emergency requiring action at stupid o clock on the weekend.
13
u/tdeasyweb 2d ago
The last half was so stupid, I completely forgot how annoying that first part was. The guy who is forced to be on call is the bad guy for not being happy at receiving a call at midnight on Saturday to give information on work OOP has decided is critical, even though it obviously isn't.
2
2
u/AffordableGrousing 18h ago
Yeah, that was a management failure. The manager offered an employee's weekend time (for no extra compensation) for a non-emergency without clearing it with them first. Server guy obviously could have been more polite, but OOP is so oddly flabbergasted that other people have social lives or just want to relax on a Saturday night, pandemic or no.
6
u/yavanna12 the laundry wouldn’t be dirty if you hadn’t fucked my BF on it 1d ago
I work in the medical field. Our $$$$ implants arrive with trackers. So if this server was expensive likely they did the same.
7
u/Matt_NZ 1d ago
I work in the IT industry and while servers are expensive (especially today), I've never heard of GPS trackers being an available option in them. The kind of company that is buying super expensive servers usually isn't using an MSP with someone like OOP either - this sounds like your bog standard "everyday" server hardware
14
u/Anla-Shok-Na 1d ago
So if the server is brand new in a sealed box, that means it's off. How is it pinging the internet and reporting it's location? It's a pretty fancy piece of equipment if it has some kind of independently powered asset tracker.
8
12
3
u/KiloJools It's just pharmaceutical-grade banana bread 2d ago
For a minute there, I was SURE that Manager Guy had moved the server just to set up On-Call Guy for failure.
5
u/waterdevil19144 It's just pharmaceutical-grade banana bread 2d ago
Was this posted here instead of [r/bestofreddit](r/bestofredditsagas)[orsagas](r/bestofredditsagas) by mistake? It’s not like OOP took advice from anyone that I could see.
16
2
2
u/yavanna12 the laundry wouldn’t be dirty if you hadn’t fucked my BF on it 1d ago
I love stories from tales from tech support but I don’t think these classify as updates as the posts as a whole are just one big story.
8
u/SirLoremIpsum 2d ago
On call guy was right
This was not an emergency requiring on call escalation after hours on the weekend.
6
10
u/NYCinPGH 2d ago
Except, of course, for the fact that he stole a high-enough-end server that they had a GPS locator on it. The “I had it to do some work” was obviously a lie, since it was in the original box.
1
u/errant_night 1d ago
I was very worried this would end with the guy getting swatted because there as abhorrent material on that server
1
u/TerminalJammer 1d ago
Resealing a box isn't that hard, but I have my doubts about whether he had been doing legitimate things with it.
1
u/AffordableGrousing 18h ago
Yeah, but OOP didn't know that at the time. In any case it should be pretty clear why someone wouldn't want to be contacted on a Saturday night for a non-urgent project they weren't aware of and never agreed to.
1
1
u/cleric3648 Editor's note- it is not the final update 1d ago
I’ve worked places like this before. Company IT has equipment drop shipped to employees but it’s usually laptops and monitors. Occasionally a server gets shipped this way if the person ordering it is the assigned tech and they don’t select an office.
1
u/WarlockSoL being delulu is not the solulu 11h ago
Ok that was actually great. As someone who has worked on-call support before, yeah, sorry, that's your job. You don't get to blow it off because you don't feel like doing it.
I genuinely wonder if the dude purposefully stole the server or if he just happened to have it for some work-related reason and "conveniently forgot to return it" when he got fired. That was honestly insanely kind of OOP not to just straight up report him because the guy would have been in serious trouble if he had. I'm not sure he'd get jail time (considering he returned it to OOP no problem at least - I'm sure he would have handled it over if the police came knocking too), but I gotta think at least some kind of fine/criminal record which doesn't help when you've just lost your job.
1
u/Fun-War6684 I will never jeopardize the beans. 19h ago
I’m with the angry guy because the on call number is actually for emergencies like a whole rack went down or something. Not for guy doing after hours work
6
u/Outrageous_Guard_674 15h ago
Even if that was true, which it really isn't necessarily, the problem only existed because he took the server home with him in the first place.
0
u/Fun-War6684 I will never jeopardize the beans. 13h ago
Gonna be honest wit you. I didn’t even finish the post It’s written for children who still think IT work is the like Hollywood says it is
1
u/boppitywop 1d ago
How was the server still in the box sealed, if they could find it on GPS because it had pinged the internet?
-23
u/Sad-Stop-2401 2d ago
People still fukken install servers on-site? I thought it was all cloud!!!!
LOL!
But WTF man!
23
u/bentnotbroken96 2d ago
"The Cloud" is just someone else's computer.
There's still a computer.
2
-3
u/Sad-Stop-2401 2d ago
Yeah, but not of this kind. These are handpicked servers. Clouds are farms of servers in a DC, installed en mass with a supply of spare kept to replace them. They are also not hand configured like this. They are purpose built for mass deployment using extreme degree of automation.
This is very 90s style server management.
39
u/tyda1957 2d ago
You do realise cloud servers also need physical installment. Right?
5
-3
u/Sad-Stop-2401 2d ago
Yeah, but not of this kind. These are handpicked servers. Clouds are farms of servers in a DC, installed en mass with a supply of spare kept to replace them. They are also not hand configured like this. They are purpose built for mass deployment using extreme degree of automation.
This is very 90s style server management.
17
u/Imoa 2d ago
People still do it - just depends on where you’re at.
-4
u/Sad-Stop-2401 2d ago
Yeah they do it. Most of the time for not the right reasons. People live off grid too. A lot of them are luddites or weirdos.
15
16
u/Nadamir 2d ago
Tell me you’ve never worked in a regulated industry without telling me.
1
u/Sad-Stop-2401 2d ago
Yeah, thats why I gave example of JP Morgan Chase in one of my comments. However I have seen this BS to be beyond regulatory requirements. A lot of time, just very luddite cynicism.
15
u/archangelzeriel sometimes i envy the illiterate 2d ago
I used to have a sticker on my laptop that said "The cloud is just someone else's computer".
Except I crossed out "someone else's" and wrote "my" in sharpie, because I was a cloud hardware administrator.
2
u/Sad-Stop-2401 2d ago
Difference is that this server has a very 90s feel. Hand installed and managed. A cloud worth its salt will be automated in configuration and installation. Otherwise it can not expand.
10
u/archangelzeriel sometimes i envy the illiterate 2d ago edited 2d ago
So at least when I was doing this (and by "this" I mean "VP of Operations" at a midsize multisite datacenter-and-bespoke-private-cloud company) in 2021, the FIRST server at a site needed some handholding. VMWare VCF, you'd instantiate the first server manually and then you could upload a CSV to the cloudbuilder VM and it'd handle the rest of it.
If you have hardware domain controllers, etc, that's another situation where you see hand-tuned servers.
Or, frankly, if your company/site is small enough to not have a proper configure-on-boot service on a given network segment, which is still VASTLY more common than not. Again, to be perfectly frank, "just plug in a new server and it configures automatically" is still mostly the domain of 1k+-employee companies with dedicated internal cloud or hardware management teams, and as OOP is a consultant (and some random on-call employee has the ability to grab a server and take it to his house without setting off 12010 asset management alerts) I suspect this is a substantially smaller workplace.
10
u/N0stradama5 2d ago
Lots of reasons to have on prem.
1
u/Sad-Stop-2401 2d ago
Most of the time it is because "boss said it so". Real advantages do come but unless you are JP Morgan Chase, it is not there. Even for JP Morgan Chase, a lot of their stuff can happily live on AWS DynamoDB/S3.
6
u/Ontological_Gap 2d ago
If you want 10x the bill, sure
5
u/Sad-Stop-2401 2d ago
Its a question of whose time is more expensive. A server in cloud or a tech on prem..
9
4
u/KiloJools It's just pharmaceutical-grade banana bread 2d ago
There are A TON of reasons for "a server" to be installed on-site. Cloud doesn't do everything.
3
u/Deepspacedreams 2d ago
It’s fairly common especially for VMs you have giant clusters for Vsphere. I haven’t work at 100% cloud company yet, most are hybrid.
6
u/eshtonrob 2d ago
Uhh you know the cloud is still somebody's physical servers right? Onsite could just mean at the data centers.
2
u/Sad-Stop-2401 2d ago
This has 90s server management and hand raised servers written all over it.
5
u/eshtonrob 2d ago
90s server management is quite alive and well. Lots of older multinationals don't want to migrate to the cloud. As late as 2019, I was seeing IT administering small fleets in my day job.
3
2
2
u/yogos15 cat whisperer 2d ago
Some people “don’t trust the cloud”. I have to deal with this bullshit at my work.
3
u/Sad-Stop-2401 2d ago
I can sympathize. People think their data is worth more than reputation damage to AWS/GCP.
Its a weird thinking that does not go away. I mean, you are Asthetica Civil Works, not Skunkworks. Your cookie cutter home floor plans are not worth for AWS S3 team to take a repetitional damage on.
1
•
u/AutoModerator 2d ago
Do not comment on the original posts
Please read our sub rules. Rule-breaking may result in a ban without notice.
If there is an issue with this post (flair, formatting, quality), reply to this comment or your comment may be removed in general discussion.
CHECK FLAIR For concluded-only updates, use the CONCLUDED flair.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.